| name | data-convert |
| description | Re-express structured data to hit a DIFFERENT target — map a source (system A's export) onto another system's import CONTRACT, and/or RESHAPE its structure (wide↔long, nested JSON↔flat table, split one file into many, union many into one). Use when the user wants to "convert this to X's format", "map these columns to the import template", "get this into the format System B needs", "reshape / pivot / unpivot this", "flatten this JSON", "split this file by column", "combine these files", or "prepare this for upload/import". Works INTENT-FIRST: proposes the target shape from the contract + purpose, you confirm, it applies deterministically and writes a reusable **conversion card** (Markdown + an embedded JSON spec) a future agent re-runs after sense-checking the source. Delegates cleaning to data-tidy; never invents values; live inputs (FX) are pinned/user-approved, not fetched. NOT data cleaning (data-tidy), NOT matching two record sets (data-reconcile), NOT raw file-format-only conversion. |
Data Convert
Take a clean-enough source and re-express it in the structure or contract a downstream
system requires. The interoperability counterpart to data-tidy: tidy = quality (messy →
clean, same-ish shape); convert = interoperability (clean → a different structure/contract).
If the source is messy, convert delegates the cleaning to data-tidy first, then converts the
clean result.
Two jobs, often combined:
- Contract mapping — map a source onto another system's import contract (columns, order,
types, required fields). Unmapped sources are reported; required-but-missing rows follow
rules.on_missing_required (flag / exclude / error) — never invent values.
- Structural reshape —
unpivot (wide→long), pivot (long→wide), flatten/nest
(JSON↔table), split (one file→many by a key), union (many→one with column alignment).
Workflow
import sys; sys.path.insert(0, "scripts")
from convert import (load_spec, render_card, convert_file, convert_rows,
sense_check, render_report)
Run from the skill directory (skills/data-convert/). The scripts path resolves
to this skill's scripts/ subdirectory where convert.py lives.
1. Intent first — what's the target, and why?
Establish the source, the target contract (another system's import spec — its columns,
order, required fields, formats) and the purpose. Once you understand the purpose you can
propose the target shape and the mapping; the user confirms. If you don't have the target
contract, ask for a sample of the destination format or the field list.
2. Card-first — reuse before you rebuild
Always check whether a conversion card already exists for this source→target (e.g. a
convert_[name].md in the working folder). If one does, do not blind-run it:
- Load it (
load_spec) and read its Mapping + Expected source.
- Sense-check today's source against it (
sense_check(spec, header, rows) /
convert.py <src> --card card.md --check-only).
- Flag any discrepancy to the user — a mapped column missing/renamed, a new column, a
required field now empty, or a stale pinned rate — and let them decide: proceed / adjust /
regenerate. Never apply over drift silently.
3. Apply (deterministic)
spec = load_spec("convert_gl_to_journal.md")
report = convert_file(spec, "gl_export.xlsx", "journal_import.csv")
print(render_report(report, name=spec["name"]))
Or one-shot from the CLI:
python scripts/convert.py gl_export.xlsx --card convert_gl_to_journal.md -o journal_import.csv
The engine reads the source (any format via the shared ingest), runs any reshape ops, maps to
the target contract, and writes the target format (csv / json / xlsx). It reports rows
in/out, the sense-check, unmapped source columns and required-but-missing target fields.
4. Save the reusable card
Put the user's standing confirm-first rules on the spec as standing_rules (verbatim) before
rendering — later sessions must not rediscover them from chat.
spec["standing_rules"] = [
"Exclude any row whose required target fields are blank from the import file.",
]
open("convert_gl_to_journal.md", "w", encoding="utf-8").write(render_card(spec))
The card is the reusable artefact — a Markdown doc a person can read (purpose, standing rules, target
contract, a Source→Target mapping table, the Expected source to verify) with an embedded
```convert-spec JSON block that is the machine source of truth. No per-conversion .py
runner: next month an agent reads the card, sense-checks the new export, and re-runs this engine.
Live inputs (FX and the like) — pinned, never fetched
The engine never reaches the network — reproducibility and the data-handling rule both depend
on that. When a conversion needs a live value (e.g. a USD→GBP rate):
- the agent may fetch it (web search, on the user's instruction) and show it; the user
decides whether to apply it (confirm-first);
- the value is pinned into the card (
fx: {pair, rate, as_of, source}) and applied by the
engine as a recorded constant, so a re-run is deterministic and auditable;
- refreshing is a deliberate act — the sense-check flags a pinned rate that's old for the run.
Fetching a public rate is a public lookup, not egress of the user's data — the dataset never
leaves. Never send the actual rows to an external service to "convert" them.
Which skill? (avoid overlap)
- Messy source (junk rows, mixed dates, dedupe, flagging) → data-tidy first, then convert.
- Matching two record sets to find breaks → data-reconcile, not convert.
- Insights / metrics → data-analyse. Convert reshapes; it does not aggregate for insight
(a bounded rollup is a future addition, deliberately out of v1).
The spec / card format
A declarative spec (source, target contract, map, optional filter, reshape, fx,
rules, expected_columns). Beyond the core mapping + reshape it also supports: a row
filter, a lookup compute (enrich against a reference table / inline map), output
validation on target columns (allowed / pattern / max_len / check: iban|bic),
date-keyed FX (per-row rate by transaction date), and fixed-width / template output.
Full schema, every reshape op and compute function: see references/conversion-spec.md.
Ready-made example contracts: references/contracts/journal-import.md,
references/contracts/payments-upload.md.
Safety
- Deterministic & human-in-the-loop — the engine applies a confirmed spec; it never invents a
value. Output is a draft for review, not a posting or an upload — a person submits it.
- Never blind-applies — a drifted source is flagged, not force-converted.
Principles
Behavioural charter: ../../PRINCIPLES.md — drafts not advice, never invent, honesty and
calibration, plain speech, action boundary.
Data handling
The engine runs on your machine and makes no network calls; cleaning delegates to data-tidy
locally. As with any AI-assisted work, the agent driving the skill sends whatever it reads into
its context to your AI provider — see ../../PRINCIPLES.md#data-handling--pii-policy. Keep sources and outputs on your
synced or shared file store; a target file carrying personal or confidential data is gated — only
share it with the entitled recipient.
Feedback
Have an improvement or found a bug? Capture it with the toolkit's shared feedback format
(../../CONTRIBUTING.md#skill-feedback-format) and hand it to the user to file; fix in scope if asked.
Requirements & mode
Pre-screen: ../../README.md#mode--environment-compatibility + python ../../scripts/envcheck.py. Python + openpyxl
(for .xlsx in/out); reads CSV/PDF/.docx/.msg via the shared engine (same optional deps as
data-tidy). Portable — the code needs no network, MS Office, credentials or connector.
