Overview

This document is a collection of best practices for Wrangler CSV parsing and cleansing of CSV files.

General Tips

• Parse CSV Avoid using automatic header detection with parse-as-csv should avoid using automatic header determination directive(parse-as-csv :col ‘\t’ false). On large files that are distributed across multiple partitions and header is no available to be set in different partitions.

• Sometimes a file looks fine, It could contain non-printable ASCII characters that usually don’t belong in CSV files. It can be hard to track these down. Use find-and-replace directive. `find-and-replace :col 's/\000-\007\013-\037//g'.

• To remove CTRL-M from the end of each line again use find-and-replace directive. find-and-replace :col 's/\r$//g' if you want to remove CTRL-M at end of the line. Make sure you apply this directive before applying parse-as-csv, the header line which is the first line of CSV is not present. This will either result in failure or records will be lost.

• If you have to use parse-as-csv directive, then make sure the files are smaller than 128 MB (lowest data block).