ferrogate-test-strategy

name	ferrogate-test-strategy
description	Use when working in the FerroGate repo and deciding how to test a change — which test layer a change needs, adding coverage for a feature or bugfix, whether the harness is sufficient, or when a test run surfaces a bug or new feature that needs tracking. For running the harness CLI itself, use the ferrogate-test skill.

FerroGate Test Strategy

The guiding direction for FerroGate's layered test system: which layer a change needs, how findings feed the issue queue, and how the tooling must grow to keep up. The binding source of truth is AGENTS.md → ## Testing Architecture and docs/testing/testing-architecture.md; this skill is the on-the-spot pointer, kept thin so it does not drift from those. To operate the E2E harness CLI, use the ferrogate-test skill.

Pick the layer for the change

"Unit tests pass" never proves a cross-cutting or runtime-wiring change.

• Pure logic → Static gate + Unit (+ Property if it is a state machine/invariant).
• Unit test implementations live in dedicated sibling *_test.rs files (or crates/*/tests/*.rs), never inline in business-logic files. A production module may keep only the minimal cfg(test) path declaration needed for private-item access.
• Wiring inside a crate → add Crate integration (crates/*/tests/*.rs).
• Provider / guardrail / policy scope / quota surface → add Contract/compliance: prove what it writes is what the runtime reads, and that it emits the telemetry/audit evidence it claims. A 200 with the runtime ignoring the value is a failure, not a pass (the #188 asset-quota-scope failure mode).
• Producer→consumer handoff (billing, evidence, exports) → add Cross-component chain (ferrogate-test gateway-billing-chain / guardrail-supabase).
• Persistence / schema / migration → add Durability (ferrogate-test *-restart).
• Any feature → the E2E harness (ferrogate-test ci) must close the loop.
• Hot-path performance → run the Performance layer, record the number; never a silent PR gate.

Property tests (proptest) belong on state machines/invariants (routing fallback, quota reserve/settle/rollback, streaming stage transitions, ack/settlement ordering), not ordinary branch logic. Streaming and concurrency are correctness surfaces — add a focused async test at the lowest layer that can reproduce the race instead of leaving it to a slow E2E run.

Feed findings back to the issue queue

A bug or missing capability a test surfaces, and that is not fixed in the same change, becomes a house-style GitHub issue before the change lands — never a silent #[ignore], skipped scenario, or inline TODO.

• Real bug, not fixed now → file a bug issue and add a failing regression test at the lowest layer that reproduces it; the fix makes it pass.
• Flaky test → confirm against main, open or link a tracking issue, record it in the affected suite. No blind retries that hide a real race.
• New feature/capability idea → file it and prioritize it in the queue; do not scope-creep it into the current change.
• House style: [area][priority] title; Problem → Scope → Acceptance (checkboxes) → Non-goals body; reuse existing labels; link the issue from the failing suite and from the commit that acts on it.

The harness must keep up with the layers

The methodology is fixed; the tooling is not yet complete. ferrogate-test does not yet support every layer above — treat it as a living harness that must be iterated to satisfy the test system, not a fixed set of scenarios the methodology bends to.

• If a change needs a layer the harness cannot yet express, grow the harness (new subcommand, shared assertion, fixture) and wire it into ferrogate-test ci — do not skip the layer or downgrade the claim to fit the current tool.
• A missing tool never silently shrinks coverage. A layer you cannot yet automate is still owed a manual proof at the affected surface plus a filed tooling issue.
• Quota, canonical provider settlement, and Guardrail allow/block evidence use the reusable ComponentContract executor; quota and Guardrail also have live-Supabase proof. Runtime and the harness share one canonical provider adapter-family registry, and exact matrix equality fails closed before E2E. #214 completed that matrix and #213 completed distinct, idempotent settlement identities for multiple billable attempts in one logical request.
• Live Supabase proof is isolated per scenario: one unique schema is reused for that scenario's restarts and removed through exact-name RAII cleanup. Use --keep-supabase-schema only when retained state is deliberately required for debugging.

Run the narrowest proof

• Local module gate (mirrors CI): scripts/local-test-modules.sh <module> (quality, core-policy, control-plane, agentic-gateway, ai-proxy, cli-tooling, gateway-runtime, e2e-harness, supabase-storage).
• Workspace units: cargo +1.88.0 test --workspace --all-features.
• Static: cargo fmt --all -- --check; cargo clippy --workspace --all-targets --all-features -- -D warnings; python3 scripts/check-openapi.py.
• E2E: the ferrogate-test harness (see the ferrogate-test skill).

Read the output. Do not claim production readiness from unit tests alone when the change touches runtime wiring, live reload, TLS/ACME, provider streaming, or billing settlement.

GitHub Actions are release-only (release: published) and never a per-commit fallback. Missing local infrastructure means the proof remains explicitly unverified; it does not justify adding a push/PR/manual cloud trigger.