differential-test

name	differential-test
description	Compare a parser or scientific tool against an independent reference implementation over a real corpus. Trigger on 'differential test', 'compare against ASE/cclib/OpenBabel', 'validate parser output', or 'does this agree with a reference?'. Produces a persistent harness with agree, disagree, and tool-refused outcomes.

Differential Test

Tests that assert only that something parsed cannot catch a parser returning plausible-looking wrong science. Differential testing compares the quantities that matter against an independent implementation over the same corpus. The deliverable is a harness in the project, not an LLM findings report.

This is strongest when the domain already has mature independent readers: structure formats, QC output, numerical file formats, or standards-backed serializers. It is not useful when both sides share the same parser/library or when there is no agreed semantic quantity to compare.

Boundaries

• vs bug-scan: that skill finds suspicious code and reports it. This one executes independent implementations and leaves a reproducible harness.
• vs unit tests: unit tests pin one implementation's intended behavior. This checks whether that intended behavior matches an independent reader.
• vs a golden file generated by the tool under test: self-generated goldens are not a reference. The reference must have a separate code path.

Process

1. Choose a bounded corpus and comparison contract

Start with one format and a corpus small enough to inspect by hand. State the quantities both implementations can meanfully share, such as:

• atom count and normalized element histogram
• cell presence, lattice parameters, and periodic axes
• geometry extents after an explicitly stated unit conversion
• final energy, with units converted to a common basis and a justified tolerance

Do not compare a quantity with different semantics. For example, a reference's last SCF energy and the tool's geometry-optimization energy are not comparable without proving they describe the same step.

2. Select an independent reference without mutating the project environment

Use the project's established environment if it already contains a suitable reference. Otherwise prefer an isolated tool environment over pip install into the project or system Python. For example, the first Orbitron PDB run used:

uv run --with ase \
  python scripts/differential_check.py dist/cartoon-compare \
  --binary target/release/orbitron

Reference backends are optional. A missing backend skips only the formats it cannot read; it is not a successful comparison. If no file can be compared, exit with a distinct nonzero status and say which backend is missing.

3. Prove the tool-under-test contract on one file first

Before writing a corpus loop, run the tool once and inspect its machine output:

• confirm the invocation and argument order
• confirm stdout is parseable machine output and diagnostics are on stderr
• confirm the JSON keys or other schema are real, not inferred from source
• record the exact binary/path being tested

Do this before generalizing. The Orbitron draft had read InfoSummary from source but had never executed orbitron info --json; the first real invocation proved the command and immediately exposed the known PDB atom-loss defect.

4. Build a small, machine-readable harness

Put the harness under the project's existing scripts/test tooling, not in a global scratch directory. It needs one adapter for the tool under test, one per independent backend, normalized comparison records, and a JSON report.

Every file must resolve to exactly one of three outcomes:

1. Agree: both readers accepted it and all comparison fields match.
2. Disagree: both readers accepted it, but at least one pinned field differs.
3. Tool refused: the reference accepted it and the tool under test did not.

The third case is a finding, not a skip. Include it in the JSON report and make the process exit nonzero, exactly as for disagreement. A missing reference is the only skip.

For a structural parser, the minimal normalized record is usually:

atom_count
elements: symbol -> count
has_cell

For QC output, add only quantities whose units and step semantics you have verified. Normalize symbols, units, and tolerances in one place; never compare raw floats in multiple backend adapters.

5. Calibrate with a known fixture, then widen the corpus

The first run should catch a known defect or agree on a hand-checked fixture. Otherwise the harness itself is untrusted. Orbitron's first ASE run compared 1UBQ.pdb: Orbitron reported 581 atoms and ASE 660, a 79-atom (12.0%) mismatch. The five-file bundled corpus then found disagreements in every file.

Inspect the first mismatch manually. Confirm the reference did not choose a different model/record and the tool's JSON was not truncated or misread. Only then run a larger corpus.

6. Preserve the result and decide CI scope

Write the harness report to the project's notes dir or ~/scratch/ for an exploratory run. Keep the harness and a small trusted corpus in the project. The JSON report should contain the checked count, no-reference count, tool-refused count, and one record per disagreement with both values.

Do not gate CI on the first run. First fix or explicitly baseline every known failure, add exact regression tests for the repaired cases, and keep the corpus small and stable. Once trusted, disagreement or tool refusal exits nonzero; missing optional backends can remain an explicit non-gating skip.

Principles baked in

• Independent means independent. A second API over the same parser is not evidence.
• Refusal is data. The reference read a file your users may have.
• Pin user-visible quantities. Counts and element histograms catch more than "parsed successfully."
• Prove the harness on a known case first. A comparison tool that cannot catch a documented bug is just another weak test.
• CI comes last. Differential checks are only gates after their corpus, reference version, and tolerances are trusted.