| name | rocky-test-fixtures |
| description | Dagster integration test fixture lifecycle. Use when regenerating live-binary fixtures via `just regen-fixtures` / `scripts/regen_fixtures.sh`, debugging why a generated fixture differs across runs, or adding scenario data for new CLI commands. Covers determinism (zeroed timings, sentinel timestamps), the relationship between `tests/scenarios.py` and `tests/fixtures_generated/`, and the two POCs that back the corpus. |
Dagster test fixture regeneration
The dagster integration has two parallel sources of test data with different roles. Conflating them causes confusion, so this skill is worth reading before touching either.
The two test-data surfaces
| Path | Role | Edit policy |
|---|
integrations/dagster/tests/scenarios.py | Source of truth for parsing/translator/component tests. Hand-crafted Python dicts exposed as *_json pytest fixtures via conftest.py (which json.dumps each scenario). | Edit by hand. Add new scenarios when introducing a new CLI command or covering a new shape. |
integrations/dagster/tests/fixtures_generated/ | Live-binary corpus. Captured from the running rocky binary against the playground POCs. Used for drift detection — test_generated_fixtures.py re-validates that every captured JSON still parses against the current Pydantic models. | Regenerated by just regen-fixtures. Don't hand-edit — the next regen overwrites your changes. |
Tests in test_types.py, test_translator.py, test_component.py, etc. consume the *_json fixtures backed by scenarios.py. The generated corpus is exercised solely by test_generated_fixtures.py to catch shape drift between the Rust binary and the Pydantic models.
A legacy tests/fixtures/*.json directory previously held the parsing-test corpus; it was removed once scenarios.py became the source of truth. See conftest.py for the migration note. The --in-place flag in scripts/regen_fixtures.sh still targets that directory and is effectively a no-op today.
When to use this skill
- Changing any
*Output struct that affects a fixture's shape (pairs with the rocky-codegen skill — codegen updates the Pydantic models; the captured fixtures should still parse against them)
- Adding a brand new CLI command that needs a parsing test in
test_types.py (you'll add a scenarios.py entry, not a JSON file)
- Investigating why
test_generated_fixtures.py fails after a binary or schema change
- Debugging non-determinism in a captured fixture (timestamps, durations, row counts)
Regeneration workflow
just regen-fixtures
./scripts/regen_fixtures.sh
Writes to integrations/dagster/tests/fixtures_generated/. The captured JSON should git diff cleanly except for fields that legitimately changed; non-trivial diffs warrant inspection.
After regenerating, always run from integrations/dagster/:
uv run pytest tests/test_generated_fixtures.py -v
This catches any captured-output shape that no longer parses against the current Pydantic models.
Prerequisites
The script hard-fails if either is missing:
-
Release binary at engine/target/release/rocky — build with:
cd engine && cargo build --release --bin rocky
or run just codegen (which also builds in release mode and shares the artifact).
-
duckdb CLI on $PATH — used to seed the playground POC's DuckDB file:
brew install duckdb
Two POCs back the corpus
regen_fixtures.sh runs the binary against two playground POCs, captured into different subdirectories of fixtures_generated/:
| POC | Purpose | Fixtures produced |
|---|
examples/playground/pocs/00-foundations/01-replication-basics/ | Baseline full_refresh replication pipeline. Produces most fixtures. (00-playground-default is now a transformation pipeline; this is its verbatim replication clone, kept as the fixture source so captures stay byte-identical.) | discover.json, plan.json, run.json, state.json, compile.json, test.json, ci.json, lineage.json, doctor.json, history.json, metrics.json, optimize.json, etc. |
examples/playground/pocs/02-performance/03-partition-checksum/ | time_interval (partition-keyed) pipeline. Exercises partition-specific shapes. | partition/compile.json, partition/run_single.json, partition/run_backfill.json, partition/run_late.json |
If you need a fixture shape that neither POC produces naturally (e.g. drift.json from a non-zero schema diff), add it as a scenarios.py entry rather than coercing the POC into producing it.
Determinism: why captured JSON is stable across runs
Without post-processing, every regen would produce a slightly different diff because Rocky stamps:
- Duration fields with real wall-clock elapsed time (
_ms, _seconds, _secs suffixes)
- Timestamp fields with
now() (updated_at, started_at, finished_at, timestamp, captured_at)
The capture helper inside regen_fixtures.sh runs a Python normalizer (scripts/_normalize_fixture.py) over each captured JSON file that:
- Zeroes any numeric field whose key ends in
_ms / _seconds / _secs
- Replaces any string field whose key is in the wall-clock set with
"2000-01-01T00:00:00Z"
Important invariants the normalizer preserves:
last_value (watermarks) — NOT touched. It's a logical value derived from seeded data and is genuinely deterministic across runs.- Anything inside a
data payload — NOT touched. Only top-level wall-clock fields get the sentinel.
If you add a new timing or timestamp field to an *Output struct, you may need to extend TIMING_SUFFIXES or WALL_CLOCK_FIELDS in the normalizer. Otherwise your new fixture will drift on every regen.
Exit-code tolerance
The capture helper tolerates non-zero exit codes as long as stdout parses as JSON. This is intentional:
rocky doctor exits 1 when any check is warning, 2 when critical — and both still emit valid JSON.rocky run can exit non-zero on partial success and still emit a valid RunOutput.
The dagster integration's allow_partial=True path mirrors this tolerance in production — see integrations/dagster/src/dagster_rocky/resource.py.
Adding parsing-test coverage for a new CLI command
For a new CLI command whose output the dagster tests need to parse:
- Rust schema exists (from the
rocky-codegen skill).
- Pydantic binding exists (from
just codegen-dagster).
- Add a
scenarios.py entry with a representative output dict, then expose it via a *_json pytest fixture in conftest.py.
- Add a parsing test in
test_types.py that loads the fixture and asserts on key fields.
- Optionally, add a
capture call in scripts/regen_fixtures.sh so the playground POC contributes a live-binary sample under fixtures_generated/. Run just regen-fixtures afterwards.
If the playground POC can't produce the command naturally (e.g. an adapter-specific command that needs Databricks), skip step 5 — the scenarios.py entry is the test source of truth.
Debugging fixture drift
Symptoms and causes:
| Symptom | Likely cause |
|---|
test_generated_fixtures.py fails with ValidationError | Pydantic model changed but captured fixture is stale — run just regen-fixtures |
fixtures_generated/<name>.json differs by a single _ms field | Normalizer missed a new duration field — extend TIMING_SUFFIXES |
| Fixture differs wildly between runs | Playground POC has non-deterministic seed data (rare) or a new timestamp field — extend WALL_CLOCK_FIELDS |
capture prints ==> <name> but the file is empty | Command exited non-zero AND wrote to stderr, not stdout — check the command manually |
rocky binary not found at engine/target/release/rocky | Missing release build — run cd engine && cargo build --release --bin rocky or just codegen |
| Partition fixtures missing but main ones present | 02-performance/03-partition-checksum POC missing or broken — check ls examples/playground/pocs/02-performance/03-partition-checksum/ |
Reference files
scripts/regen_fixtures.sh — the capture pipeline + POC listscripts/_normalize_fixture.py — wall-clock + timing normalizerintegrations/dagster/tests/scenarios.py — hand-crafted Python dict scenarios (test source of truth)integrations/dagster/tests/conftest.py — *_json fixture exposureintegrations/dagster/tests/test_generated_fixtures.py — the test that guards the captured corpusintegrations/dagster/CLAUDE.md — "Adding support for a new Rocky CLI command" checklist (includes the scenarios step)justfile — the regen-fixtures recipe (wraps the script)
Related skills
rocky-codegen — the Rust → Pydantic/TS cascade that fixtures are validated againstrocky-new-cli-command — the end-to-end checklist that includes scenario/fixture creationrocky-poc — authoring the playground POCs that the captured fixture corpus depends on
