miller

name	miller
description	Drive Miller (mlr) to process CSV/TSV/JSON/etc. data. Use when constructing mlr command lines: discover capabilities from the catalog rather than guessing, learn the data's shape before writing expressions, validate DSL before running, and recover from failures via structured errors.

Miller agent playbook

Miller (mlr) is a command-line data processor for CSV, TSV, JSON, JSON Lines, and other tabular/record formats, with SQL-like verbs (cut, sort, join, stats1, ...) and an awk-like DSL (put, filter).

Work this loop. Each step exists to prevent a specific, common failure.

1. Discover — never invent names

Everything valid is in the catalog; anything not in the catalog does not exist. Hallucinated flag/function names are the top failure mode.

• Route an intent: which with e.g. "join two files on a key" → ranked candidates. confident: true means a name matched; trust the top hit.
• Browse cheaply: list_capabilities with index: true → every verb/function/flag/keyword with one-line summaries.
• Drill in: list_capabilities with kind: "verb", names: ["join"] → the full entry. Prefer the structured options list (flag, arg, type, enum values) when present; usage_text is the prose fallback.
• The whole catalog is cacheable against (mlr_version, catalog_schema_version) — re-fetch only when either changes.

2. Constrain — learn the data before touching it

Call describe_data on the input first. It returns, per field: name, types seen with counts, occurrence count, null count, cardinality, min/max, and — for low-cardinality fields — every distinct value.

• Copy field names exactly from describe_data; never guess casing or spelling.
• For flags like -g (group-by) and DSL comparisons, use values from the values array, not values you expect to exist.
• Fields whose count is less than other fields' are absent in some records: guard DSL with is_present($field).

3. Validate — check DSL before spending a run

Before any run that includes put or filter, call validate_dsl with the expression. Cost: parse-only, no data read. On valid: false, the error document has kind, hint, and did_you_mean — apply the hint, don't re-guess syntax.

4. Run — and read errors structurally

Call run with argv as a list, one element per shell word (no shell quoting):

{"args": ["--icsv", "--ojson", "cat", "data.csv"]}

Command-line shape rules that prevent most argv errors:

• Main flags (I/O formats etc.) come before the verb: mlr --icsv sort -f name f.csv.
• Format shorthands: --icsv --ojson (separate in/out), --csv/--c2j etc. (combined).
• Chain verbs with then: ["--icsv", "sort", "-f", "k", "then", "head", "-n", "3", "f.csv"].
• If a field value being compared in filter might collide with a verb flag, end verb flags with -- before filenames.
• Inline data goes in stdin_text; files go at the end of args.

On failure, exit_code is nonzero and error (when present) carries kind, hint, and did_you_mean — hint is often a corrected command line; prefer executing it over reasoning from the message. stdout_truncated: true means the output exceeded the server's cap: narrow the query (e.g. head, cut) rather than re-running the same command.

Notes

• run cannot execute external commands (DSL system/exec, piped redirects, --prepipe) unless the server was started with --allow-shell; such calls fail cleanly. It can write files via tee, split, and DSL output redirects — treat it as a write-capable tool.
• Long inputs: prefer describe_data + targeted verbs over dumping whole files through run.
• One record format in, another out: Miller is format-to-format; there is no separate conversion step.