validate-data

name

validate-data

description

Validate battery measurement data against the Ionworks platform's authoritative schemas and strict measurement checks — schema/metadata validation via the discovery API, column/dtype/monotonicity checks on parquet files, and the platform's strict `validate_measurement_data` validators on time series + steps. Use whenever the user wants to check data before uploading, audit data already on the platform, debug an upload that failed validation, or verify that processed parquet/JSON files conform to the canonical format. Triggers on "validate data", "check before upload", "validate parquet", "validate metadata", "schema check", "header audit", "strict validation", "validation errors", "MeasurementValidationError", or whenever a user is about to run `cell_measurement.create_or_get` and wants to be sure it'll succeed.

Validate Data

Validate battery cycling data against the Ionworks platform's authoritative format. Catches problems before upload so you never publish broken records, and audits data already on the platform so you can find issues post-hoc.

The four layers of validation

The platform format has four independent layers; each needs its own check.

Layer	What it covers	Tool
1. Header audit	Are raw input columns being silently dropped by the parser?	manual diff before processing
2. Schema (metadata JSON)	Do spec.json / instance.json / measurement.json conform to the create-schema?	jsonschema against client.schema("data")
3. Parquet columns + dtypes	Do time_series.parquet / steps.parquet have the required columns with correct dtypes and monotonic time?	column/dtype check against the discovery schema
4. Strict measurement	Do the time series and steps pass the platform's domain validators (sign convention, current consistency, capacity sanity, time gaps)?	ionworks.validators.validate_measurement_data(..., strict=True)

Run all four before upload. Schema-only validation is not enough — a perfectly schema-conformant parquet can still fail the strict checks (e.g., a charge step labelled with negative current). The strict validators are what the platform runs at create_or_get(..., validate_strict=True); running them locally first avoids round-tripping a broken file.

Workflow

1. Header audit — for every raw file (not a representative sample), compare the columns in the raw file against the columns that landed in the parquet. Cycler readers in ionworksdata return only canonical columns and silently drop auxiliary channels (Maccor Temp 1..4, Arbin aux probes, custom Basytec columns). If a column is in the raw file but not in your parquet and not in an explicit "drop" list, it's a bug. The canonical procedure (eight rules, audit-report template, extra_column_mappings plumbing) lives in the process-data skill — run that audit before validating, not a parallel one.

2. Fetch the platform schemas via discovery. Never hardcode column lists. The schemas come from client.schema("data") and define the authoritative create-schema for each entity plus the time-series and steps column manifests.

   from ionworks import Ionworks
   client = Ionworks()
   data_schema = client.schema("data")
   # data_schema keys: cell_specification, cell_instance, cell_measurement, time_series, steps
   

   See the discover-api skill for the broader discovery pattern.

3. Run schema validation on every JSON file in your processed directory. Use jsonschema.Draft202012Validator with the create_schema from the discovery output. $ref resolution needs a small fix — see references/schema_validation.md.

4. Run parquet validation on every time_series.parquet and steps.parquet. Check required columns are present, dtypes match the schema, Time [s] is monotonically non-decreasing, Step count is monotonically non-decreasing, Current [A] isn't all-null.

5. Run strict validators on every measurement. ionworks.validators.validate_measurement_data(df, strict=True, steps_df=..., rated_capacity=..., skip_checks=...) runs the same checks the platform runs at upload time. Raises MeasurementValidationError with a list of structured errors (each has a .code and a .message).

6. Report failures, exit non-zero if any. Pre-upload validation is a gate — if it fails, you don't upload until you've fixed the source.

For data already on the platform, the layers are slightly different — you can't header-audit retroactively, but you can re-fetch and re-validate the time series and steps. See references/platform_validation.md.

Pre-upload validation script (skeleton)

A single pre-upload entrypoint that runs all four layers. Copy this into scripts/process/validate.py:

"""Validate every processed dataset against the platform schemas and strict
measurement checks. Exits non-zero if any file fails. Run before upload.
"""
from pathlib import Path
import sys
from ionworks import Ionworks
from ionworks.validators import validate_measurement_data, MeasurementValidationError
import polars as pl
from jsonschema import Draft202012Validator

# ... see references/schema_validation.md for build_validators(), validate_json(),
# validate_time_series(), validate_steps()
# ... see references/strict_validation.md for the strict pass

def main():
    client = Ionworks()
    schemas = client.schema("data")
    # ... validate every JSON, every parquet, every measurement ...
    # ... print a summary, return 0 on success, 1 on any failure ...

The full implementations are in the reference files; this skill's reference layout puts the recipes in one place so the entrypoint stays small.

Parallel execution for large datasets

When the processed directory has hundreds of measurements, the strict validator pass benefits from ProcessPoolExecutor. One process per worker, 2 polars threads per worker so they don't fight over the CPU:

import os
os.environ.setdefault("POLARS_MAX_THREADS", "2")

from concurrent.futures import ProcessPoolExecutor, as_completed

workers = max(1, min(8, (os.cpu_count() or 4) // 2))
with ProcessPoolExecutor(max_workers=workers) as ex:
    futs = [ex.submit(_validate_one, str(p)) for p in measurement_dirs]
    for fut in as_completed(futs):
        meas_dir, errors = fut.result()
        if errors:
            failures.append((meas_dir, errors))

Schema validation is fast enough to run inline; only the strict pass needs parallelism.

Common validation failures and what they mean

• time_gaps — large gaps in Time [s]. Legitimate for combined multi-day measurements (cell rests between RPTs). Skip via skip_checks={"time_gaps"} for those files.
• current_sign — discharge steps with negative mean current or charge steps with positive mean current. The Ionworks sign convention is positive = discharge, negative = charge. Fix at processing time (e.g. iwdata.transform.set_positive_current_for_discharge); don't hand-flip in the analysis layer.
• step_capacity_mismatch — Charge capacity [A.h] and Discharge capacity [A.h] columns don't match the integrated current. Usually means single-row steps slipped through; remove via the prep_upload step.
• missing required column — schema validation flags this with the column name. Almost always means the raw reader dropped it. Go back to the header audit.
• Time [s] is not monotonically non-decreasing — most often happens when concatenating files without resetting time. The processor should reset Time [s] to start at zero per measurement.

Discovery API as the source of truth

Don't hardcode the column list. The schemas evolve. The discovery API returns the current authoritative definition:

ts_schema = data_schema["time_series"]
ts_schema["standard_columns"]   # {col_name: {"type": "float", "required": True, ...}}
steps_schema = data_schema["steps"]
steps_schema["columns"]         # same shape

This means a new optional column added platform-side automatically shows up in validation without a code change. Hardcoded lists drift quietly; the discovery output doesn't.

When to validate

• Before every upload — validate.py is part of the upload gate, not optional.
• After every processing change — touched the parser? Re-validate the whole tree.
• When investigating a report anomaly — sparse data in a section often traces back to a measurement that quietly failed validation but landed on the platform anyway because validation was skipped. Validate the source.
• Periodically over the entire platform — drift catches up. A schema-validation pass over every measurement on prod once a quarter is good hygiene.

See also

• references/schema_validation.md — fetching schemas via the discovery API, $ref normalization, JSON and parquet validators
• references/strict_validation.md — validate_measurement_data usage, error handling, when to skip checks
• Header audit (pre-parse) — see the process-data skill, "Header audit (mandatory)" section
• references/platform_validation.md — validating data already uploaded (re-fetch, sample, audit)
• process-data skill — populating measurement.protocol.* correctly at processing time (validation catches missing fields; the processing skill explains how to populate them)
• discover-api skill — broader discovery pattern, all /discovery endpoints