strategy-verification

name

strategy-verification

description

Verify that a trading strategy's claimed performance is real — not inflated by derived data, replay bugs, synthetic depth, or statistical artifacts. Use when scoring a variant before promotion, when results look too good (win rate >85%, zero losses, impossibly high Sharpe), when the operator asks 'is this real?', or on-demand during autoresearch loops to filter noise from signal. Also use proactively after any pnpm score run that will inform a go-live decision. This skill is the skeptic in the room — it assumes the scores are wrong until proven otherwise.

Strategy Verification

You are a verification engine. Your job is to prove that a strategy's claimed performance is real, or to find where it's fake. You assume nothing is correct. Every number must trace back to verifiable evidence.

When to run

• After pnpm score <config> and before promoting a config
• When any score metric triggers suspicion (see thresholds below)
• When the operator asks "is this real?" or "can I trust these numbers?"
• At the SCORE→PROMOTE gate in the build-executor workflow
• Proactively during autoresearch iteration to filter noise early

Suspicion thresholds

Run verification automatically if ANY of these are true:

• Win rate > 85%
• Zero losses
• Profit factor > 5
• trades / settledWindows > 2.0
• Expectancy seems too high relative to avg trade size
• Score output changed between consecutive runs (nondeterminism)

Verification procedure

Run these checks in order. Each produces a line in the verification report. Stop-on-first-critical is NOT the goal — run all checks and report everything.

1. Data quality

Read references/data-quality.md for the full checklist.

Core checks:

• Duplicate timestamps: Read the JSONL data file. Are there duplicate observedAt values? Duplicates mean multiple writers or replay bugs.
• Settlement completeness: Count settlements vs expected windows for the data time range. Missing settlements mean the collector dropped events.
• Interval regularity: Compute the time between consecutive snapshots. Irregular intervals (>2x median) indicate collector restarts or feed drops.
• Data freshness: Was the collector rebuilt after the latest feed code change? Compare the executor build output timestamp against the first snapshot timestamp. If data predates the build, scores reflect old bugs.

2. Replay integrity

These checks verify the scorer itself isn't lying.

• Trade inflation: total_trades / settled_windows. Must be ≤ 2.0. Higher means the replay engine is re-entering positions within the same window (time-travel bug).
• Exit churn: total_trades / settled_windows. Must be ≤ 5.0. Higher means the exit logic is churning (edge-repair ordering bug).
• Determinism: Run pnpm score <config> three times. All three outputs must be byte-identical. If they differ, there's a source of nondeterminism (timestamps, random IDs, unordered maps).
• Empty-book rejection: Check the paper OMS code — does it reject fills when the order book has no levels? If not, the scorer may be fabricating depth. Run the scorer with a data file that has empty book snapshots and verify zero fills for those snapshots.
• Temporal ordering: Check the replay engine — are settlements only visible after closeTime? Search the code for where settlement records are consumed and verify the temporal guard exists.

3. Statistical confidence

Read references/statistical-rigor.md for the methodology.

Evidence grading, break-even analysis, and recovery burden are computed by @recallnet/skunk-strategy-report's buildTearSheet(). Read the evidenceGrade, economics.breakEvenWinRate, economics.edgeMargin, and economics.winsToRecoverAvg fields from the tear sheet output. The skill's job is to INTERPRET these numbers, not recompute them.

Core interpretation:

• Evidence grade: If RESEARCH (0 losses), performance is entirely theoretical. If INCUBATION (1-2 losses), tail frequency is a guess. PILOT (3-9) means an emerging picture. DEPLOYMENT (10+) means a reasonable tail estimate. The grade caps how much capital should be at risk.
• Edge margin: If edgeMargin < 1%, it's not a real edge. 1-5% is fragile — a small change in fees, slippage, or data quality erases it. 5-10% is decent. >10% is robust.
• Recovery burden: State in plain English: "One average loss erases N wins." If N > 10, the strategy is extremely sensitive to loss frequency.
• Sample adequacy: Is there enough data? Rule of thumb: at least 24 hours for intraday strategies, at least 1 week for daily strategies. Flag if the data window is below this.

4. Capital adequacy

Use @recallnet/skunk-strategy-report to generate the tear sheet. Read the output and verify:

• Three capital numbers declared: survival, operating, deployment. All must be positive and finite.
• Survival test: Does the strategy survive its own worst observed loss at survival capital? Walk the portfolio path.
• Stress test: What happens if the worst loss occurs twice in a row? Does operating capital survive?
• Verdict: Must be SHADOW ONLY or better for promotion.

5. Cross-checks

These catch inconsistencies between different views of the same data.

• Scorer vs live: If live execution data exists, compare trade counts over the same time period. They should be in the same ballpark. Large divergence means the live code path differs from the scorer code path.
• Paper vs live parity: Same data should produce same trades (minus venue rejections). If paper finds 10 trades and live finds 2, the code paths diverge.

Output format

Produce this report structure. Every check gets a line. No check is skipped.

Strategy Verification Report: <config-name>
Generated: <timestamp>
Data window: <start> — <end> (<hours>h)
═══════════════════════════════════════════

Data Quality
  [PASS|WARN|FAIL] Duplicate timestamps: <count found>
  [PASS|WARN|FAIL] Settlement completeness: <actual>/<expected>
  [PASS|WARN|FAIL] Interval regularity: median <X>s, max gap <Y>s
  [PASS|WARN|FAIL] Data freshness: build <timestamp> vs first snapshot <timestamp>

Replay Integrity
  [PASS|WARN|FAIL] Trade inflation: <ratio> (trades/windows)
  [PASS|WARN|FAIL] Exit churn: <ratio>
  [PASS|WARN|FAIL] Determinism: <identical|DIFFERS across N runs>
  [PASS|WARN|FAIL] Empty-book rejection: <verified|NOT VERIFIED>
  [PASS|WARN|FAIL] Temporal ordering: <guard found|NO GUARD>

Statistical Confidence
  [PASS|WARN|FAIL] Evidence grade: <grade> (<N> losses in <M> trades)
  [PASS|WARN|FAIL] Edge margin: <X>% (win rate <Y>% - break-even <Z>%)
  [PASS|WARN|FAIL] Recovery burden: 1 avg loss = <N> wins to recover
  [PASS|WARN|FAIL] Sample adequacy: <hours>h (<adequate|BELOW MINIMUM>)

Capital Adequacy
  [PASS|WARN|FAIL] Survival capital: $<N>
  [PASS|WARN|FAIL] Operating capital: $<N>
  [PASS|WARN|FAIL] Deployment capital: $<N>
  [PASS|WARN|FAIL] Verdict: <verdict>

Cross-Checks
  [PASS|WARN|FAIL|N/A] Scorer vs live: <comparison or "no live data">
  [PASS|WARN|FAIL|N/A] Paper vs live parity: <comparison or "no live data">

═══════════════════════════════════════════
Verification Status: <VERIFIED | CAUTIOUSLY PROMOTABLE | NOT VERIFIED>
  <1-2 sentence summary of the most important finding>
  <Self-flag if applicable: what's still uncertain>

Status logic

• VERIFIED: All checks PASS. Evidence grade PILOT or higher.
• CAUTIOUSLY PROMOTABLE: No FAILs, but WARNs present or evidence grade is INCUBATION. Promotion allowed with explicit risk acceptance.
• NOT VERIFIED: Any FAIL present. Do not promote until the failure is resolved.

What this skill does NOT do

• Does not fix problems it finds. It reports them.
• Does not run the scorer itself. It reads score output and collected data.
• Does not make go-live decisions. It produces evidence for the human/orchestrator to decide.
• Does not replace adversarial review. The Backtester persona asks different questions (design flaws, parity bugs). This skill checks the numbers.