process-data

name	process-data
description	Process raw battery cycling data into the ionworks standardized format (parquet + JSON) or BDF. Use when the user wants to convert data from cyclers (Arbin, Maccor, Basytec, Neware, etc.), MATLAB files, CSVs, JSON, or any other raw format; adds a new open-source dataset; or mentions ionworksdata transforms, step summaries, the time_series/steps parquet format, or the BDF (.bdf) format.

Battery Data Processing

This skill guides you through converting raw battery cycling data into the ionworks standardized format. Two output shapes are supported:

• Parquet + JSON — the canonical dataset format for the Ionworks platform (time series + step summaries as parquet, metadata as JSON).
• BDF (.bdf) — a single-file portable format (Battery Data Format) suitable for sharing, archiving, or uploading to the platform when a full JSON metadata bundle isn't needed.

Both use the same ionworksdata column conventions; choose based on what the user is trying to do.

When to use this

• The user has raw battery data (MATLAB, CSV, JSON, HDF5, cycler exports, etc.) and wants to process it
• The user wants to add a new open-source dataset to the repository
• The user is debugging or modifying an existing processing script
• The user mentions ionworksdata, parquet output, BDF, or the standard battery data format

High-level workflow

1. Understand the raw data — inspect the files, figure out the formats and cohorts
2. Audit headers across every file and confirm a column mapping with the user (mandatory — see below)
3. Write a processing script that loads the raw data into a Polars DataFrame with the required columns
4. Use ionworksdata to compute derived quantities (cycle/step counts, capacity, energy, step summaries)
5. Diff reader output against raw headers and recover any auxiliary columns the reader dropped
6. Write the output files — parquet + JSON for the full dataset format, or .bdf for a single-file export

The hardest parts are steps 1–2 and step 5 — every dataset is different, and silent column drops between cycler families are the most common cause of bad data landing on the platform. Steps 3, 4, 6 follow a consistent pattern.

Header audit (mandatory)

Before writing any standardized parquet, you must audit every raw file's headers and confirm with the user what each column maps to. Datasets that look uniform at the top level routinely contain cohorts that came from different cyclers or different export profiles, and the canonical readers in ionworksdata.read.* silently drop columns they don't recognize. One missed auxiliary thermocouple channel means weeks of downstream analysis with empty plots — catch it here.

The eight rules

1. Walk every raw file — not a representative sample. For xlsx, list every sheet and the header row of each. For csv, read the header. For zips/archives, do not skip files. For mixed-cycler datasets, treat each sub-folder / cohort as its own audit unit and explicitly surface differences between cohorts.

2. Build a header inventory grouped by (cohort × column-set) and show it to the user. Every distinct column-set must be visible — not just "Arbin format detected". Example:

   ncell / Cap_rated/40oC
     3 files → Aux_Temperature_1(C), Cycle_Index, Step_Index, …
     6 files → Temp 1, Temp 2, Temp 3, Temp 4, Cycle P, Cycle C, …
   

3. Classify every raw column into one of three buckets and confirm with the user:

   • Standard — maps to a canonical ionworksdata name (Voltage [V], Temperature [degC], …)
   • Auxiliary — kept under a sensible name but outside the canonical set (Temperature 2 [degC], Aux_Voltage_2(V), …)
   • Drop — explicitly chosen to discard (timestamps the platform reconstructs, internal cycler flags, …)

   Default behaviour: any column you cannot immediately classify is Auxiliary — preserve it under a clean name. Dropping requires explicit user agreement. Never silently drop a column.

4. Diff reader output against raw headers. Whether you use ionworksdata.read.X() or a hand-written parser, after the read step compare the resulting frame's columns against the source headers and flag any columns the reader stripped. Auxiliary channels (Maccor Temp 2..4, Arbin aux probes beyond #1, Basytec custom columns) routinely fall through this gap.

5. Plumb auxiliary columns through with extra_column_mappings. ionworksdata.read.* (and read.time_series) accept an extra_column_mappings={raw_name: clean_name} dict that tells the reader to preserve additional raw columns under your chosen names. Use this instead of re-reading the file and merging — the reader applies the mapping inside the same pass and you get a single, consistent frame out.

6. Keep multi-channel signals separate. If a file has multiple thermocouples (Temp 1..4), expose each as its own Temperature N [degC] column. Do not average — the channels may correspond to different physical sensors (cell can, tab, chamber air). Averaging discards information.

7. Confirm units and sign conventions per cohort. Pull a small sample from each cohort and ask the user to confirm before applying any transforms:

   • Time in seconds vs. minutes
   • Current sign (positive = charge vs. positive = discharge)
   • Capacity in Ah vs. mAh
   • Temperature in °C vs. °F

8. Surface "this cohort has no temperature" as a real finding. Missing temperature is not a silent column-absent state — explicitly tell the user "cohort X has no temperature signal — confirm this is expected before we ship downstream."

Required audit report

Before any write happens, produce a report of this shape and wait for user confirmation:

=== Raw column audit ===

ncell / Cap_rated/40oC  (9 files, 2 distinct column sets)
  set A (3 files, Arbin):  Aux_Temperature_1(C), Test_Time(s), …
  set B (6 files, Maccor): Temp 1, Temp 2, Temp 3, Temp 4, Test Time, …

Proposed mapping for set B:
  Test Time              → Time [s]            (parse "Nd HH:MM:SS")
  Current (A)            → Current [A]
  Voltage (V)            → Voltage [V]
  Capacity (AHr)         → (recomputed via set_capacity)
  Temp 1 → Temperature [degC]     STD (reader's canonical mapping — do not re-map)
  Temp 2 → Temperature 2 [degC]   AUX, kept via extra_column_mappings
  Temp 3 → Temperature 3 [degC]   AUX, kept via extra_column_mappings
  Temp 4 → Temperature 4 [degC]   AUX, kept via extra_column_mappings
  Rec, Cycle P, MD, ES, DPT Time  → DROP (internal cycler bookkeeping)

⚠  Reader pipeline note: ionworksdata.read.maccor() emits Temperature [degC]
    from Temp 1 natively but drops Temp 2..4. Pass them through
    extra_column_mappings={"Temp 2": "Temperature 2 [degC]", ...} to keep them.

Proceed?  (y/N)

Only write standardized parquet after the user confirms.

Auxiliary columns and reader gaps

ionworksdata.read.* parsers return a canonical subset of columns by default — they are not faithful round-trips of the source file. The canonical set covers the columns required by downstream transforms (Time [s], Voltage [V], Current [A], cycler step/cycle ids, sometimes a single temperature channel), and everything else is dropped silently unless you opt in.

The supported way to preserve additional columns is the extra_column_mappings parameter on read.time_series / read.time_series_and_steps (and the format-specific readers). Pass a {raw_name: clean_name} dict and those columns flow through the reader untouched:

extra_column_mappings = {
    "Temp 2": "Temperature 2 [degC]",
    "Temp 3": "Temperature 3 [degC]",
    "Temp 4": "Temperature 4 [degC]",
}
# Use time_series_and_steps when you want both frames in one call;
# time_series returns only the time-series frame. Same extra_column_mappings
# applies to both.
ts, steps = iwdata.read.time_series_and_steps(
    path, extra_column_mappings=extra_column_mappings
)

Note: don't include the reader's canonical mappings (e.g. Temp 1 → Temperature [degC] for Maccor) in extra_column_mappings — let the reader produce the canonical Temperature [degC] column natively, and use extra_column_mappings only for the channels the reader would otherwise drop.

After the read, still diff the frame's columns against the raw headers to confirm everything you intended to keep is present — typos in extra_column_mappings keys fail silently (the column just won't appear in the output).

Use extra_column_mappings instead of re-reading the file and merging with join_asof. The reader handles the mapping in a single pass, avoids alignment edge cases (resampling, duplicate-timestamp nudging at step boundaries), and keeps the processing script linear.

Output structure (parquet + JSON)

The hierarchy mirrors the platform data model: cell spec → cell instance → measurement. One directory per level, each containing its own JSON metadata file. Parquet data lives under the measurement that produced it.

{cell_spec_name}/
├── spec.json
└── {cell_instance_name}/
    ├── instance.json
    └── {measurement_name}/
        ├── measurement.json
        ├── time_series.parquet
        └── steps.parquet

A cell spec directory can hold many instance subdirectories (cells built to the same design), and each instance can hold multiple measurement subdirectories (e.g. separate cycling + calendar aging + RPT periods). Names should be filesystem-safe identifiers.

Time series requirements

The time_series.parquet must have these columns:

Column	Type	Notes
Time [s]	float	Cumulative across all cycles — never resets to 0
Voltage [V]	float	Cell voltage
Current [A]	float	Positive = discharge, negative = charge (platform convention)
Cycle count	int	Cumulative cycle number, 0-indexed
Step count	int	Cumulative step number across all cycles

Optional columns: Temperature [degC], Cycle from cycler, Step from cycler, Discharge capacity [A.h], Charge capacity [A.h], Discharge energy [W.h], Charge energy [W.h].

Current sign convention

Raw cycler files export positive = charge (standard electrochemical convention). The Ionworks platform uses the opposite: positive = discharge. A sign flip is therefore usually required.

Call iwdata.transform.set_positive_current_for_discharge(df) on essentially all raw data. It detects the source orientation (via voltage response, or a mode column if present) and outputs the platform convention: positive = discharge, negative = charge.

The platform validator enforces this — if positive current correlates with rising voltage, it rejects the data with a Current sign convention error pointing you to set_positive_current_for_discharge. That error means the flip hasn't been applied yet.

Do not apply a manual df.with_columns(-pl.col("Current [A]")) negation in addition to the transform — the transform already produces the correct orientation, and double-flipping puts you back to positive = charge.

Some synthetic or pre-converted datasets already store positive = discharge, so set_positive_current_for_discharge is a no-op on them. Don't infer from such a dataset that flipping is unnecessary — real cycler exports almost always need it.

The most common pitfalls

Current sign — raw is charge-positive, platform is discharge-positive, so flip. Cyclers follow the textbook (positive = charge); Ionworks uses the opposite. Run set_positive_current_for_discharge() to convert. The trap is not applying it (validator rejects the data) or applying it on top of a manual negation (double-flip puts you back where you started). See the Current sign convention section above.

Silent column drops between cycler families. The most expensive failure mode. ionworksdata.read.* returns a canonical subset and drops auxiliary channels (Maccor Temp 2..4, Arbin aux probes beyond #1, Basytec custom columns) with no warning. Verify every raw header column ends up either in the output, in an explicit auxiliary column, or in a documented drop list — see the Header audit and Auxiliary columns and reader gaps sections above.

Time must be cumulative. Raw data often resets time to 0 each cycle. Track an offset:

cumulative_time_offset = 0.0
for cycle in cycles:
    time_sec = raw_time + cumulative_time_offset
    cumulative_time_offset = time_sec[-1]

Step count must be cumulative. Same idea — track the offset across cycles so step numbers never repeat.

Using ionworksdata

The ionworksdata library handles the tedious derived quantities. Use it — don't reimplement these calculations by hand.

import ionworksdata as iwdata

Setting cycle and step counts

If the raw data has a step column, always prefer set_step_count with that column. It detects a new step on any change in the step id — increase, decrease, or wrap-around — because internally it keys off np.sign(np.diff(...)) rather than assuming monotonic ids. So protocols that reuse step ids within a cycle (e.g. GITT alternating pulse/rest as ... 9 10 9 10 9 10 ..., or RPTs with repeated substeps) work out of the box. Reaching for current sign is not a workaround for repeating step ids — it's a fallback for when no usable step column exists at all.

# If the data has a "Cycle from cycler" column:
df = iwdata.transform.set_cycle_count(df)

# Preferred: use the cycler's step column. Handles repeating/decreasing ids
# (GITT, RPT substeps, etc.) correctly — every transition starts a new step.
df = iwdata.transform.set_step_count(df, options={"step column": "Step from cycler"})

# Fallback only when no step column exists at all:
df = iwdata.transform.set_cumulative_step_number(df, options={"method": "current sign"})
df = df.with_columns(pl.col("Step number").alias("Step count"))

Caveat — duplicate timestamps at step transitions. Some cyclers emit two rows with the same Time [s] at the boundary between steps: the last sample of step N (at the old current) and the first sample of step N+1 (at the new current). This is not a bug — both rows are real measurements that straddle the current switch and together encode the instantaneous IR drop (ΔV across the step change). Dropping one of them silently destroys information that pulse fits, GITT analysis, and ECM R₀ extraction depend on.

Downstream transforms do require Time [s] to be monotonically non-decreasing, so the fix is to nudge, not dedupe: add a small epsilon (e.g. half the smallest positive diff(Time [s]) in the file, or 1 ms if no finer resolution exists) to the second row of each duplicate pair. That preserves both samples and the voltage jump between them.

# Preserve IR-drop pairs at step boundaries by nudging duplicate timestamps.
# For each group of rows sharing the same Time [s], add k * epsilon to the k-th
# row (k = 0, 1, 2, ...) so the first row is unchanged and later rows are
# shifted just enough to make time strictly increasing.
df = df.sort("Time [s]")
positive_dt = df["Time [s]"].diff().drop_nulls().filter(pl.col("Time [s]") > 0)
epsilon = (positive_dt.min() or 1e-3) / 2
df = df.with_columns(
    (pl.col("Time [s]") + pl.int_range(pl.len()).over("Time [s]") * epsilon).alias("Time [s]")
)

Only fall back to dropping exact duplicates if you've confirmed the rows are genuine duplicates (identical voltage and current), not an IR-drop pair.

Setting capacity and energy

Call these after cycle/step counts are set:

df = iwdata.transform.set_capacity(df)   # Creates Discharge/Charge capacity [A.h]
df = iwdata.transform.set_energy(df)     # Creates Discharge/Charge energy [W.h]

These integrate current (and power) over time per step. They need Time [s], Current [A], and Voltage [V] to exist first.

When the source file already ships capacity and energy columns, validate them against the integral of current and power before keeping them — and use iwdata.transform.fix_swapped_charge_discharge_columns to recover from cyclers that mislabel charge vs discharge (common with half-cell exports). See the validate-data skill (references/strict_validation.md) for the full pattern.

Creating step summaries

steps_df = iwdata.steps.summarize(time_series_df)

This produces a DataFrame with per-step aggregates: voltage/current/power statistics, inferred step types (Rest, CC charge, CC discharge, CV, etc.), durations, and accumulated capacity/energy. Write this to steps.parquet.

Reading common raw formats

ionworksdata.read provides parsers for common cycler exports — use these before writing a custom parser:

from ionworksdata import read

df = read.basytec("cell_001.txt")    # Basytec exports
df = read.bdf("cell_001.bdf")        # round-trip from BDF
# plus .biologic, .maccor, .neware, etc. — check `dir(read)` for current list

read.detect(path) will pick the right parser when the format is ambiguous.

What each parser drops (incomplete — always diff against raw headers per the audit step):

• read.maccor — keeps a single temperature channel at most; drops Temp 2, Temp 3, Temp 4 and other auxiliary signals.
• read.arbin — keeps Aux_Temperature_1 and main signals; drops aux voltage/temperature probes beyond #1 and most DAQ_* channels.
• read.basytec — drops user-defined custom columns that aren't in the canonical mapping.
• read.neware, read.biologic — similarly canonical-subset only.

Treat the above as a reminder, not a contract. The reader code evolves; the diff against raw headers is the only reliable check.

Prefer the one-call pipeline

The format-specific parsers above return a raw canonical-subset frame — no step count, capacity, energy, or step summary. Prefer read.time_series / read.time_series_and_steps, which run the standard pipeline in one call and take the same reader/extra_column_mappings/options args:

• read.time_series runs parse → step/cycle count → capacity → energy and returns the time-series frame.
• read.time_series_and_steps does all of that, then adds the per-step summary and a validation-gated current-sign auto-fix: with the default options={"validate": True} it validates the frame and, only if validation reports a reversed/indeterminate current sign, flips the sign and recomputes capacity/energy/steps.

ts, steps = read.time_series_and_steps(
    path, reader="neware", extra_column_mappings={...}, options={...}
)

If the source has no step column, time_series derives one from current-sign transitions automatically — no manual step-index pass needed.

Because time_series_and_steps already applies the sign fix when validation flags it, do not also call iwdata.transform.set_positive_current_for_discharge on its output — that would double-flip back to positive = charge (see Current sign convention above). If you take the lower-level read.<format> / read.time_series path instead, the auto-fix does not run, so apply set_positive_current_for_discharge yourself.

Reader gotchas

• Multi-sheet Excel (Neware BTSDA). A BTSDA .xlsx splits into unit/test/cycle/step/record sheets; the time series is on record, but the reader defaults to sheet 0 (metadata) and fails with a missing-Timestamp error. Select the sheet: options={"sheets": {"type": "name", "value": "record"}}. The record sheet often has no step/cycle column — current-sign step derivation (above) covers it.
• Current units (milliamp vs amp headers). The Neware reader maps amp-headed columns (Current (A), Current(A)) to Current [A] and milliamp-headed columns (Current (mA), Cur(mA)) to Current [mA]. Trust the values, not the header text: sanity-check the post-read current range against the cell's C-rate. If a header lies about its unit (e.g. an amp-headed column actually carrying milliamps), override its target with extra_column_mappings={"<raw header>": "Current [mA]"} — extra_column_mappings replaces the reader's built-in mapping for that key rather than adding to it.
• Vendor / MES exports (Voltaiq, etc.). When a platform re-exports a cycler file (extra derived columns plus a timezone-aware Timestamp), the native reader can choke on the timezone (a polars datetime parse error). Fall back to reader="csv" with an explicit extra_column_mappings — the generic CSV reader skips the datetime magic and just applies your mapping. Map the already-elapsed time column (e.g. Test Time (s) → Time [s]) so no timestamp parsing is needed.

Writing the BDF format

When the user wants a single-file portable export rather than the full parquet/JSON bundle, use the BDF writer:

from ionworksdata import write

write.bdf(df, "cell_001.bdf")

Inputs: a Polars DataFrame with at least Time [s], Voltage [V], Current [A]. Other ionworksdata columns (Cycle count, Step count, temperature, capacity, energy) are preserved. Columns not recognised by the BDF mapping are passed through under their ionworksdata names so read.bdf round-trips cleanly.

BDF uses a machine-readable vs. labeled column convention internally. Pass use_machine_readable_names=True if the consumer expects the compact form.

JSON metadata files (for the parquet bundle)

Three JSON files, one per level of the hierarchy:

• spec.json — the cell design. One per cell spec directory.
• instance.json — one specific physical cell.
• measurement.json — one test run on that cell.

Parent/child relationships are implicit in the directory tree — the JSON files do not carry explicit cross-reference IDs.

Getting the authoritative schema

Invoke the discover-api skill to fetch the live schemas via client.schema("cell_specification"), client.schema("cell_instance"), and client.schema("cell_measurement"). The Pydantic models served by /discovery are the single source of truth — field names, required/optional, allowed enum values, and inline guidance (on units, sign conventions, when to use which measurement_type, etc.) all come from there. Don't guess from memory; the platform evolves.

Minimal examples

These sketches are just to show the shape — always cross-check field lists against discover-api output before writing a real file.

// spec.json
{
  "name": "Example_18650",
  "form_factor": "18650",
  "manufacturer": "Example Manufacturer",
  "ratings": {
    "capacity": {"value": 1.1, "unit": "A.h"},
    "voltage_min": {"value": 2.0, "unit": "V"},
    "voltage_max": {"value": 3.6, "unit": "V"}
  },
  "source": {
    "doi": "https://doi.org/10.xxxx/xxxxx",
    "citation": "Author et al. Title. Journal vol, pages (year).",
    "publication_date": "YYYY-MM-DD"
  }
}

// instance.json
{
  "name": "dataset_cell_id",
  "batch": "2023-05-12",
  "date_manufactured": null,
  "measured_properties": {"cell": {}}
}

// measurement.json
{
  "name": "dataset_cell_id_cycling",
  "start_time": "2023-01-01T00:00:00+00:00",
  "measurement_type": "time_series",
  "protocol": {
    "name": "Fast charging cycling",
    "ambient_temperature_degc": 30.0,
    "initial_soc": 1.0,
    "c_rate": 1.0
  },
  "test_setup": {"cycler": "Arbin LBT", "operator": "John", "lab": "Lab A", "channel_number": 73},
  "step_labels_validated": false
}

protocol — what was done to the cell; anything that affects the measurement outcome. Both dicts are free-form — use whatever keys are present in the source data. Common protocol keys:

• name — human-readable protocol name
• ambient_temperature_degc — chamber/ambient temperature during the test
• initial_soc, c_rate, dod, pressure_mpa, and any other test condition
• definition — only include when you have an actual UCP protocol dict or string to attach; omit otherwise

test_setup — physical logistics of how the test was run (does not affect electrochemical outcome). Common keys:

• cycler — cycler model (e.g. "Arbin LBT", "Maccor 4000")
• operator, lab, channel_number, and any other logistical metadata

test_setup lives only on measurements, not on cell instances.

All numeric values with physical units use the Quantity format: {"value": 1.1, "unit": "A.h"}. Units use PyBaMM notation — .-separated atoms with signed integer exponents for compound units (A.h, mg.cm-2, W.h.kg-1). Pint-style strings (A*h, mg/cm**2) are also accepted and normalized to PyBaMM notation. Common: V, A, A.h, W.h, degC, s, ohm, mg.cm-2.

Script organization

Place processing scripts under {dataset}/scripts/process/ and analysis scripts under {dataset}/scripts/analyze/. The script should be runnable standalone and process all cells in the dataset.

Existing examples

When writing a new processing script, look at the existing ones for patterns. They cover a range of raw formats:

• MATLAB structs loaded via h5py (nested references, time in minutes)
• CSV files with varying column names and units
• JSON columnar data requiring parsing and column renaming
• Native cycler exports handled via ionworksdata.read.*

Read the most similar existing script to the new dataset's format, but adapt it — don't copy blindly. Each dataset has its own quirks in column naming, time units, and data structure.