How csvkit Handles CSV Format Detection and Type Inference

csvkit detects the CSV dialect (delimiter, quoting, etc.) by sniffing the first 1024 bytes of input, and it infers column types by converting text that looks like numbers, dates, or booleans into those types. Both behaviors are on by default and work well in most cases, but they can produce occasional errors. If you don't want them, disable sniffing with --snifflimit 0 (or -y 0) and type inference with --no-inference (or -I).

Format detection: how the sniffing works

When csvkit reads a file, it doesn't assume commas. It samples the first 1024 bytes and deduces the dialect — for example, whether fields are separated by commas, tabs, or spaces. This means a tab-separated or space-separated file can often be read without you specifying anything.

The trade-off is that the sample is limited. If the first 1024 bytes don't represent the rest of the file well — say, a header block with unusual spacing before the real table begins — the deduced dialect can be wrong, and columns may split or merge incorrectly.

Controlling the sniff limit

Set --snifflimit 0 (-y 0) to turn sniffing off entirely. When you do this, csvkit falls back to its default dialect assumptions rather than guessing from the data. Use it when:

• You already know the file's format and want deterministic parsing.
• A file's opening bytes are misleading and cause wrong delimiter detection.

Type inference: what gets converted

By default csvkit performs type inference: it converts text to numbers, dates, booleans, and similar types. This is what lets tools like csvstat compute summary statistics without you writing any code — a column of digit strings is treated as numeric rather than as text.

Inference is convenient but not infallible. Values that resemble a type without being one (for example, an ID like 007 or a date-shaped string that isn't a date) can be converted in ways you didn't intend.

Disabling inference

Pass --no-inference (-I) to keep everything as text. This is the safer choice when:

• You're passing data through to another tool and want byte-for-byte fidelity.
• Columns contain codes, IDs, or leading zeros that must not be reinterpreted.
• You've seen unexpected conversions in your output.

Choosing between the defaults and the flags

The two flags are independent: you can disable sniffing while keeping inference, or the reverse. The documentation's own guidance is that these features "are useful and work well in most cases, but occasional errors occur" — so reach for the flags when you hit one of those errors, not as a default.

A related note on output formatting

Sniffing and inference affect how csvkit reads input. To change how output is formatted — the field separator, line terminator, and so on — you use csvformat, not the input flags above. Keep the two concerns separate: input parsing is controlled by --snifflimit and --no-inference; output formatting is controlled by csvformat.

When csvkit itself is the wrong tool

If you need more complex analysis than csvkit handles, the documentation points to agate. If you need more speed or larger files, you may be hitting csvkit's limits — consider loading the data into SQL, or using qsv or xsv instead.