brokk-adversarial-test-sweep

name	brokk-adversarial-test-sweep
description	Harden a test suite with a risk-ranked adversarial audit, root-cause fixes, minimized regression tests, redundancy review, and complete validation. Use for adversarial test audits, test-suite hardening, regression hunting, or comprehensive test cleanup.

Adversarial Test Sweep

Outcome

Within the authorized scope, find meaningful gaps in the test suite, reproduce real defects, fix their root causes, retain the smallest useful regressions, and finish with clean validation and no unexplained failures. Optimize for defended behavior, not test count or a coverage percentage.

Scope and authority

• Follow repository instructions and preserve unrelated work.
• For an audit or plan request, inspect and report without changing code.
• For a hardening, change, or fix request, make in-scope local edits and run non-destructive tests without pausing for approval.
• Ask before external writes, destructive or costly actions, or material scope expansion. Stop when a newly found bug requires an unresolved product or architectural decision.
• Model choice and reasoning effort belong to the host configuration. Do not ask the model to “think harder” or claim that the prompt changes its effort.

For a repository-wide or multi-milestone sweep, create or update the repository's required execution plan. Define a bounded current milestone and continue through it; keep later waves in the plan instead of silently widening the active scope.

Non-goals

• Exhaustive Cartesian combinations with no plausible failure model.
• Tests that mirror implementation-shaped lists unless membership or order is a user-visible contract.
• Real host exhaustion, timing-only assertions, long sleeps, retries, ignored tests, or weakened assertions used to manufacture a green result.
• Unrelated refactors discovered during the sweep.

Workflow

1. Establish the baseline

1. Inspect the worktree, repository instructions, supported platforms, feature combinations, and external dependencies.
2. Inventory the scoped production surfaces and existing tests. Use Bifrost code intelligence for symbols, usages, and related files when available; use shell search for arbitrary text, test names, and bounded counts. Keep discovery commands limited to relevant paths and refs, and verify optional paths exist before including them.
3. Select the narrowest exact test target that exercises the scope. Confirm how the framework applies name filters before starting a costly build so a unit filter does not enumerate unrelated test targets.
4. Complete and record the scoped baseline before editing files. Do not mutate sources while a baseline command is still running.
5. Record exact baseline commands and outcomes, including ignored tests, flakes, timeouts, and environment limitations. Do not treat an existing failure as background noise.
6. State the current milestone, required gates, and evidence needed to call it complete.

For Bifrost itself, treat AGENTS.md and .agents/PLANS.md as authoritative. Prefer tests/common/inline_project.rs::InlineTestProject for small analyzer projects, use scripts/with-isolated-cargo-target.sh for isolated Cargo builds, and include the feature-complete nlp,python suite for repository-wide full validation. A user-defined bounded milestone uses its stated gates instead.

2. Build a risk-ranked matrix

For each scoped subsystem, select only categories with a plausible failure model. Rank by impact, likelihood, and missing evidence:

• Empty, missing, duplicate, reordered, malformed, truncated, oversized, and minimally valid inputs.
• Zero, one, maximum, off-by-one, overflow, underflow, Unicode, and path boundaries.
• Races, re-entrancy, cancellation, deadlocks, stale generations, and nondeterministic ordering.
• Deep or wide structures, many files or symbols, response budgets, timeouts, cleanup, and other controlled resource pressure.
• Partial writes, stale or incompatible caches, interrupted initialization, invalidation, corruption, and retry after failure.
• Ambiguity and invalid assumptions across APIs, languages, platforms, and optional features.

Record each selected probe, the existing evidence, the expected behavior, and the result. Briefly justify skipped high-risk categories; do not document every obviously inapplicable pairing.

Prefer deterministic controls such as barriers, channels, injected providers, explicit low budgets, and bounded watchdogs.

3. Make each test earn its place

Retain or add a test only when it protects a distinct observable contract, would fail for a plausible defect, has discriminating assertions, and uses the smallest realistic fixture. Remove or consolidate a test only after mapping its behavior to stronger retained coverage.

Before adding coverage without a reproduced defect, name the missing behavioral evidence and a plausible defect that the test would catch. A boundary value is not sufficient justification by itself. Tests that merely restate an existing private assertion, internal collection invariant, or panic message are normally implementation mirrors; retain them only when that fail-fast behavior is itself an important observable contract.

4. Close each finding

For every defect:

1. Minimize a meaningful reproduction and verify it fails before the fix.
2. Trace the structured code and data flow to the root cause.
3. Fix production behavior or correct a demonstrably wrong test contract.
4. Keep the minimized regression and name the behavior it protects.
5. Run the focused test, neighboring suites, and affected cross-feature or cross-language coverage before continuing the matrix.

5. Validate from narrow to broad

Run the narrowest exact focused target first, then neighboring suites, applicable formatting and scoped linting, and finally the complete matrix required by the milestone. Avoid generic filters that build or enumerate unrelated targets. Repeat concurrency or flake tests only when multiple runs provide relevant evidence. Review every ignored, skipped, filtered, timed-out, or environment-gated result before calling the milestone clean.

Completion and report

Mark the milestone complete only when every selected matrix entry is resolved, all required gates pass, every bug has a permanent regression, removed tests have replacement coverage, and no failure or skip remains unexplained.

Report, in this order:

1. Scope and completion status.
2. Bugs found, root causes, fixes, and regression tests.
3. Tests removed or consolidated and their replacement coverage.
4. Exact validation commands and outcomes.
5. Remaining blockers or deliberately deferred risks with evidence and next action. Omit this section when there are none.

Do not claim completion when required evidence is missing. Report the precise blocker or incomplete milestone and preserve the smallest reproduction.

Attribution

This workflow is adapted from a prompt shared by Paul Hudson in his original post:

Run a comprehensive adversarial unit test sweep that deliberately targets edge cases, malformed inputs, race conditions, boundary values, resource exhaustion, state corruption, and invalid assumptions; ensure every test earns its place by being necessary, non-duplicative, and validating meaningful behavior as completely as is practical; remove or consolidate weak or redundant tests; fix every failing test, add permanent regression tests for each bug found, and repeat until the suite passes cleanly with no unexplained failures and strong coverage.