forge-verification

name	forge-verification
description	WHEN: About to claim implementation is complete or working. HARD-GATE: Run verification, see output, THEN claim success. Never "should pass", "confident", "should work".
type	rigid
version	1.0.1
preamble-tier	3
triggers	["run verification","verify implementation","check implementation is complete"]
allowed-tools	["Bash","Read"]

Verification Before Completion

Iron Law: Evidence before assertions. Always.

Anti-Pattern Preamble: Why Agents Skip Verification

Rationalization	The Truth
"I'm confident the code works, the logic is sound"	Confidence is not evidence. Production requires proof from actual test runs.
"The previous test run passed, this shouldn't have changed"	Previous runs prove nothing about current state. Dependencies, cache, timing all matter.
"The changes are small, I should verify manually"	Manual spot-checks miss edge cases. Automated verification is the only complete record.
"Unit tests pass, that's sufficient proof"	Unit tests verify components in isolation. Full verification tests integration, real conditions, end-to-end flows.
"It should work based on my understanding of the code"	Understanding is subjective. Output is objective. Never substitute belief for evidence.
"Verification takes too long, I'll assume it passes"	The time cost of verification is negligible vs. the cost of undetected failures in production.
"The code review looked good, that's verification enough"	Code review is about correctness; verification is about observable behavior. They are orthogonal.
"I tested locally, it works there"	Local testing != production verification. Environment differences, ordering, concurrency, state matter.
"These are boring infrastructure tests, probably fine"	Boring tests catch boring bugs that break production in boring ways. Skip none.
"I can just describe what passed instead of showing logs"	Logs are the only source of truth. Descriptions are filtered through human interpretation.
"Running verification a second time would just duplicate effort"	Running verification confirms repeatability. Flaky tests need investigation. Do not skip.

Iron Law

RUN THE VERIFICATION COMMAND AND OBSERVE THE OUTPUT BEFORE CLAIMING SUCCESS. CONFIDENCE IS NOT EVIDENCE — OBSERVED OUTPUT IS EVIDENCE.

Cross-cutting: brain codebase scans

When work depends on ~/forge/brain/products/<slug>/codebase/ (council, tech plans, eval paths): if SCAN.json exists, run python3 tools/verify_scan_outputs.py <codebase-dir> and require exit 0 (retry up to 3× with 1s delay before claiming the scan is load-bearing). The forge_scan.py CLI already verifies after each run unless FORGE_SCAN_SKIP_VERIFY=1. Treat verify failure like missing tests — no success claims until the brain tree is whole or the gap is explicitly waived in writing.

When a task-id has ~/forge/brain/prds/<task-id>/tech-plans/*.md and you are about to claim plans are merge-ready or REVIEW_PASS: run python3 tools/verify_forge_task.py --task-id <id> --brain ~/forge/brain --strict-tech-plans (or python3 tools/verify_tech_plans.py alone) and require exit 0 — see docs/forge-task-verification.md. Teams that hit structural PASS / semantic thin slips should add --strict-0c-inventory in CI (inventory GAP rows + required citations when Confluence mirror, touchpoints/, or QA CSV exist). This catches missing Section 0c anchors and misplaced ### 1b.2a; it does not replace reading prd-locked.md end-to-end, but it blocks the common “chat-only self-review” slip.

Red Flags — STOP

If you notice any of these, STOP and do not proceed:

• Agent says "should pass", "confident it works", or "looks correct" — These are beliefs, not evidence. STOP. Run the verification and show actual output before any claim of success.
• Verification output is summarized in words instead of shown as logs — Summaries hide failures. "Tests passed" is not evidence. STOP. Show the raw test runner output.
• Test count drops between runs without explanation — Tests are being silently skipped or filtered. A lower count is not a better result. STOP. Investigate why tests disappeared.
• Agent runs only a subset of tests ("these are the relevant ones") — Selective verification misses cross-cutting regressions. STOP. Run the full suite or explicitly justify each exclusion.
• Infrastructure is unreachable and verification is skipped — "Can't connect to DB" means the test ran in an invalid environment. STOP. Fix infrastructure and re-run from scratch.
• A "flaky" test is dismissed without investigation — Flakiness is a real bug. A flaky test means the system has nondeterministic behavior. STOP. Investigate and fix before claiming pass.
• Verification is claimed complete but checklist items are unmarked — Partial verification is not verification. STOP. Every checklist item must be independently confirmed.

Detailed Workflow

Phase Authority Check (before running any verification command): Read ~/forge/brain/prds/<task-id>/conductor.log tail to confirm the current phase:

• Verification during TDD RED phase: conductor.log should show [P4.0-TDD-RED] logged for the repo (tests written and confirmed failing — no production feature code yet)
• Verification for post-dispatch TDD: conductor.log should show [P4.1-DISPATCH] for the relevant repo (implementation started, not yet eval)
• Verification for eval: conductor.log should show [P4.0-SEMANTIC-EVAL] or [P4.1-DISPATCH] complete
• Verification for pre-PR: conductor.log should show [P4.4-EVAL-PASS] before forge-verification is used as final gate

Do not claim verification results as eval evidence if conductor.log shows the task is still in a TDD phase ([P4.0-TDD-RED] present, [P4.1-DISPATCH] absent).

---

Staleness Check (run FIRST, before any verification work)

Before beginning a new verification run, check whether a PASS verdict already exists for the current commit and is less than 7 days old. If so, you may skip re-running and cite the existing evidence — but only if no code has changed since that commit.

TASK_ID="${FORGE_TASK_ID:-unknown}"
BRAIN="${FORGE_BRAIN:-${FORGE_BRAIN_PATH:-$HOME/forge/brain}}"
VERDICT_FILE="$BRAIN/prds/$TASK_ID/review-verdicts.jsonl"
CURRENT_COMMIT=$(git rev-parse HEAD 2>/dev/null || echo "")
NOW_EPOCH=$(date -u +%s)
SEVEN_DAYS=$((7 * 24 * 3600))

if [ -f "$VERDICT_FILE" ] && [ -n "$CURRENT_COMMIT" ]; then
  MATCH=$(grep "\"$CURRENT_COMMIT\"" "$VERDICT_FILE" | grep "\"PASS\"" | tail -1)
  if [ -n "$MATCH" ]; then
    TS=$(echo "$MATCH" | sed 's/.*"timestamp":"\([^"]*\)".*/\1/')
    ENTRY_EPOCH=$(date -u -d "$TS" +%s 2>/dev/null || date -u -j -f "%Y-%m-%dT%H:%M:%SZ" "$TS" +%s 2>/dev/null || echo 0)
    AGE=$(( NOW_EPOCH - ENTRY_EPOCH ))
    if [ "$AGE" -lt "$SEVEN_DAYS" ]; then
      echo "VERIFICATION PASS (cached — verified $(( AGE / 3600 ))h ago for commit $CURRENT_COMMIT)"
      echo "Stop here. Re-run /forge-verification explicitly to force a fresh check."
    else
      echo "WARNING: Last PASS for this commit is $(( AGE / 86400 )) days old — re-verification required."
    fi
  fi
fi

If the output prints "VERIFICATION PASS (cached…)" — stop. If it prints nothing or the warning — proceed with the full workflow below.

---

Identify What to Verify

• Input: Task completed, code written, tests written
• Action: Enumerate all verification targets (unit tests, integration tests, e2e scenarios, performance baselines)
• Output: Verification checklist (tasks below depend on this list)

Execute Verification

For each verification target:

1. Run the command exactly as written (no shortcuts, no assumptions)

   # Example: full test suite
   npm test -- --coverage --verbose
   
   # Example: integration suite
   cargo test --release -- --test-threads=1
   
   # Example: e2e eval
   /invoke forge-eval-gate
   

2. Observe and capture output

   • Take note of test framework output (PASS/FAIL counts)
   • Capture error messages (stack traces, assertions)
   • Note timing and performance metrics
   • Record environment info (Node version, database state, network availability)

3. Do NOT interpret — just record facts:

   • ✗ "This probably works"
   • ✓ "9/9 tests passed, 0 skipped, 0 flaky"

Surface-Specific Verification Commands

Run the command appropriate for the surface being verified. Capture full output — do not paraphrase.

API / Backend (HTTP):

# Health check
curl -f -s -o /dev/null -w "%{http_code}" http://localhost:<PORT>/health
# Endpoint smoke test
curl -X POST http://localhost:<PORT>/api/<endpoint> \
  -H "Content-Type: application/json" \
  -d '{"key": "value"}' | jq .

Web Frontend:

# Check dev server is running
curl -f http://localhost:<PORT>/ | grep -o '<title>[^<]*</title>'
# Screenshot via CDP (if eval driver available)
node -e "const CDP=require('chrome-remote-interface'); CDP(async c=>{ await c.Page.navigate({url:'http://localhost:<PORT>'}); const s=await c.Page.captureScreenshot(); require('fs').writeFileSync('verify-screenshot.png', Buffer.from(s.data,'base64')); process.exit(0); })"

Android:

# Verify app is running on device/emulator
adb shell dumpsys activity activities | grep -i "<package>"
# Check logs for errors
adb logcat -d -s "<TAG>" | tail -50
# Assert no crash
adb shell am start -n "<package>/<activity>" && echo "Launch OK"

iOS:

# Verify simulator is booted
xcrun simctl list devices | grep "Booted"
# Launch and verify app
xcrun simctl launch booted <bundle-id> && echo "Launch OK"

Database:

# Check schema matches expected
psql $DATABASE_URL -c "\dt" | grep "<expected_table>"
# Verify migration ran
psql $DATABASE_URL -c "SELECT version FROM schema_migrations ORDER BY version DESC LIMIT 1;"

Cache (Redis):

redis-cli -h localhost -p 6379 PING   # expect PONG
redis-cli -h localhost -p 6379 INFO server | grep redis_version

HARD-GATE: Paste the actual command output in your verification report. "It works" without output is not verification.

---

Validate Against Expectations

• Input: Verification output + acceptance criteria

• Check each criterion:

  • Expected pass count == actual pass count
  • Expected failures (if any) are present and explain-able
  • No unexpected warnings or deprecations
  • Performance within SLA (if benchmarked)
  • Coverage meets minimum threshold (if set)

• If gap found:

  • Do NOT claim partial success
  • Document the gap (test name, assertion, expected vs. actual)
  • Return to implementation phase
  • Re-run verification after fix
  • Repeat until all criteria met

Claim Success (Evidence-Backed)

• Output statement format:

  VERIFICATION PASS
  - Test suite: 47/47 passed, 0 skipped
  - Coverage: 92% lines, 88% branches
  - E2E eval: all 8 scenarios green
  - Performance: p95 latency 145ms (SLA: 200ms)
  Evidence: [link to full test output log]
  

• Persist the verdict to the review dashboard (REQUIRED on every PASS):

TASK_ID="${FORGE_TASK_ID:-unknown}"
BRAIN="${FORGE_BRAIN:-${FORGE_BRAIN_PATH:-$HOME/forge/brain}}"
VERDICT_FILE="$BRAIN/prds/$TASK_ID/review-verdicts.jsonl"
mkdir -p "$(dirname "$VERDICT_FILE")"
COMMIT=$(git rev-parse HEAD 2>/dev/null || echo "unknown")
TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
echo "{\"commit\":\"$COMMIT\",\"timestamp\":\"$TIMESTAMP\",\"verdict\":\"PASS\",\"task_id\":\"$TASK_ID\"}" >> "$VERDICT_FILE"

Edge Cases & Fallback Paths

Case 1: Flaky Tests (Intermittent Failures)

• Symptom: Test passes on 1st run, fails on 2nd run
• Do NOT: Ignore flakiness, run once and claim success
• Action:
  1. Run verification 3 times in succession
  2. If all 3 pass: document flakiness in brain (decision link)
  3. If any fail: investigate timing, isolation, state leaks
  4. Fix root cause or quarantine test (don't hide it)
  5. Re-run 3x again after fix

Case 2: Verification Infrastructure Down (CI/CD, test runner)

• Symptom: "Cannot reach test server", "Docker daemon not running"
• Do NOT: Claim verification passed because "it should work"
• Action:
  1. Restore infrastructure (restart service, reconnect VPN, etc.)
  2. Re-run verification after restoration
  3. If cannot restore: escalate as BLOCKED
  4. Document the infrastructure dependency in brain

Case 3: Test Timeouts or Resource Exhaustion

• Symptom: Tests hang, OOM, or exceed timeout
• Do NOT: Reduce timeout, remove resource-heavy tests
• Action:
  1. Investigate: is this a real bottleneck or a test environment issue?
  2. If test is too slow for product: refactor code or test
  3. If environment is too constrained: upgrade infrastructure
  4. Do not reduce verification rigor to fit constraints

Case 4: Verification Passes but Contradicts Earlier Assumptions

• Symptom: "Tests pass but I thought this would fail", or "More tests pass than I expected"
• Do NOT: Assume you miscounted or misunderstood earlier
• Action:
  1. Re-examine the actual test code (not names, not descriptions)
  2. Verify the test environment hasn't changed unexpectedly
  3. Update your mental model based on evidence
  4. Link this decision in brain (why assumption was wrong)

Case 5: Partial Verification (Some Suites Pass, Others Blocked)

• Symptom: Unit tests pass, but integration tests can't run (DB unavailable)
• Do NOT: Claim PASS based on unit tests alone
• Action:
  1. Restore the blocking constraint (DB, service, API)
  2. Re-run all suites
  3. Output is PASS only if ALL relevant suites pass
  4. If blocking constraint cannot be restored: escalate to BLOCKED

Verification Checklist

Before claiming success, complete all items:

• All test suites identified and listed
• Each suite executed (not assumed, not predicted)
• Full output captured (logs, metrics, assertions)
• Expected vs. actual reconciled (no gaps)
• No flaky tests (or documented + escalated)
• Performance within SLA (if applicable)
• Coverage meets threshold (if set)
• Environment stable (no transient failures)
• Success statement includes evidence (not confidence)
• Brain decision recorded (verification outcome link)

Additional Edge Cases

Edge Case 1: Verification Tool Is Broken (False Positives, Crashes, Invalid Output)

Situation: Verification tool itself is buggy or invalid. It crashes, produces nonsensical output, or has false positives.

Example: Test framework hangs on invalid assertion syntax. Verification script crashes mid-test. Coverage report shows 200% coverage (mathematically impossible).

Do NOT: Trust output from broken tools or claim verification passed based on suspect output.

Action:

1. Identify: is the tool broken or is the code broken?
   • Run tool on known-good code (previous commit, simple example)
   • Does tool work on that code? Then your code is broken (not the tool)
   • Does tool still fail on known-good code? Then tool is broken
2. If tool is broken:
   • Do NOT use it to verify your code
   • Escalate as BLOCKED (verification tool unavailable)
   • Find alternate verification method or wait for tool fix
   • Document tool failure in brain
3. If code is broken:
   • Fix code
   • Re-run verification
   • Proceed normally
4. Never ship based on verification from broken tools

---

Edge Case 2: Verification Output Is Unreadable (Logs Too Large, Format Unclear, Summary Insufficient)

Situation: Verification runs, produces output, but output is hard to parse or understand.

Example: Test output is 50MB log with no summary. Test names are cryptic (test_0001, test_0002). Output format differs from expected (JSON instead of TAP, etc.).

Do NOT: Summarize output in words ("looks like it passed"). Output must be human-readable.

Action:

1. Examine raw output:
   • Is it parseable? (valid JSON, TAP format, etc.)
   • Is it complete? (no truncation, all assertions visible)
   • Is it meaningful? (test names describe what they test)
2. If output is too large:
   • Filter to relevant sections (failures, summary, stats)
   • Or re-run with verbose=false, summary-only
   • Document what was filtered and why
3. If output format is unclear:
   • Re-run with standard format (TAP, JSON, human-readable)
   • Or write parser to convert to readable format
4. If output is still unclear:
   • Escalate as BLOCKED (verification output uninterpretable)
   • Tool maintainers must improve output readability
5. Document readable output in brain (with context)

---

Edge Case 3: Verification Passes but Humans Spot Issues (Tool Missed Bugs)

Situation: Verification suite passes all tests, but human testing or production finds bugs. Tool missed real issues.

Example: All unit tests pass. Integration tests pass. Manual testing discovers "user can't export data" feature doesn't work.

Do NOT: Ignore human findings. Verification tools are not infallible.

Action:

1. Investigate: why did verification miss this?
   • Was the scenario not tested? (missing test case)
   • Was test written incorrectly? (false positive)
   • Was behavior regression not caught? (old test, changed code)
2. Add test case:
   • Write test that reproduces the human-found issue
   • Verify test fails on current code
   • Fix code
   • Verify test passes
   • Re-run all verification (not just new test)
3. Document in brain:
   • What was missed by verification
   • Why test was missing
   • Action taken (added test, fixed code)
4. Review verification coverage:
   • Are there other scenarios humans would test that verification misses?
   • Add them to verification suite
5. Key lesson: Verification catches what you test. Humans catch what you didn't test.

---

Output: PASS (with evidence from working tools) or BLOCKED (verification tool broken, output unreadable, infrastructure down, unresolvable flakiness)

---

Edge Case 4: Test Count Drops Between Runs (Silent Test Deletion)

Symptom: Verification passes on run N (47 tests, all pass). On run N+1 after a code change, 44 tests pass — but the implementer claims "nothing was deleted." The 3 missing tests were silently removed.

Do NOT: Accept a passing run if the test count has decreased without a documented reason.

Action:

1. Compare test counts between the prior run and the current run — flag any decrease
2. Ask the implementer: which tests were removed, and why? Require a commit that shows the deletion
3. If tests were removed because they were "duplicates" or "flaky," require evidence before accepting the removal
4. If tests were removed because the feature was descoped, verify against the spec that the descope was intentional
5. A decrease in test count is a red flag, not a neutral fact — treat it as BLOCKED until explained
6. Escalation: BLOCKED if test count decreases without justification committed to git

---

Edge Case 5: Verification Passes Locally but CI Shows Different Results

Symptom: Local npm test shows 47/47 passing. CI pipeline shows 41/47 with 6 failures. The implementer argues "it works on my machine."

Do NOT: Accept local verification as evidence when CI contradicts it.

Action:

1. CI is authoritative — it runs in a clean environment without local dev state
2. Identify the failing tests in CI output — they often fail due to environment differences (missing env vars, different Node version, different file permissions)
3. Require the implementer to reproduce CI failures locally: CI=true npm test or use the same env vars as CI
4. Do not approve or merge until CI passes — "works locally" is not verification evidence
5. Common CI-only failures: hardcoded localhost references, missing .env entries, implicit dependency on global tools not in CI
6. Escalation: BLOCKED until CI passes; NEEDS_COORDINATION if CI infrastructure itself is broken (not the code)

---

Checklist

Before claiming a task complete:

• Verification command executed and raw output observed (not summarized)
• Test count is equal to or higher than the previous run (no silent skips)
• Full test suite run, not a subset
• No "should pass" or "confident it works" language used — evidence shown
• Infrastructure is reachable (DB, cache, services confirmed up before testing)
• If human found an issue not caught by verification: test added to prevent recurrence

Post-Implementation Checklist

• Every claim in the implementation report has a corresponding command run and output logged.
• No "should work" or "probably passes" — only confirmed outputs count.
• Test suite run and actual pass/fail counts recorded (not "tests pass").
• verify_forge_task.py was run and its output included, not assumed.
• Zero unverified assertions in the implementation summary.