grade-experiment

name

grade-experiment

description

Walk through the experiment grading rubric for one variant and update its frontmatter + Reading + Change Log. Use when asked to 'grade this experiment', 'update the grade', 'what grade is this', 'score this variant', 're-grade', or whenever an experiment doc needs its grade/confidence/expectancy updated. Also use proactively when you notice an experiment doc's grade is missing, stale, or contradicts its evidence.

Grade Experiment

Walk through the grading rubric for one experiment doc, compute or assess the metrics, assign the grade, and update the doc in one pass.

When to use

• Experiment doc has no grade yet (gate is blocking)
• New session evidence arrived and grade may have changed
• Operator asks to re-grade after a pivot or new data
• You're building/auditing an executor and need to assess where a variant stands
• The Reading section contradicts the current grade

The fields you're updating

grade: "II-B"
confidence: pilot
expectancy: "+$11.44/session (session-summary.json, 7 sessions since 2026-04-19), 7W/2L, max DD $93"
version: 2026-04-19

Field	Question	Type
grade	What class + how is it doing?	I-A through III-F (see promotion-metrics.md)
confidence	How tested at this class level?	Judgment: hypothesis/pilot/proven/battle-tested
expectancy	What's the actual number?	Quantitative one-liner with source + window
version	When did the current config generation start?	Date: YYYY-MM-DD

The grade has two parts: class (I/II/III) + letter (A–F).

Class	Stage	What the letter grades
I	Discovery — searching for signal	How promising is the signal?
II	Validation — backtesting/dry-running	Does edge survive replay rigor?
III	Production — live capital	Do live economics work?

The grade reflects the CURRENT VERSION only. When the config changes, bump version: and reset the letter to D at the SAME class (not back to Class I). Old sessions are Change Log history.

Example Change Log entries:
- 2026-04-18: created at I-D. Exploring thesis: 3m window captures
  settlement-adjacent spread.
- 2026-04-19: signal confirmed in replay. Promoted I-D → II-B.
  Edge survives fees in 200-window backtest.
- 2026-04-19: promoted to live pilot. II-B → III-D. Accumulating
  live sessions.
- 2026-04-20: 7 live sessions since version. Net +$80.
  Grade → III-B (pilot). Edge appears real in production.

Procedure

Step 1: Find and read the experiment doc

Experiment docs live at executors/<name>/experiments/<slug>.md — one file per variant. NOT at executors/<name>/experiment.md (that path is legacy and rejected by the gate).

ls executors/*/experiments/*.md

If no experiment docs exist yet, create one:

mkdir -p executors/<name>/experiments
# Copy the scaffold from the template or write from scratch:
cat > executors/<name>/experiments/<your-variant-slug>.md << 'EOF'
---
status: testing
reviewed: <today>
grade: ""
confidence: ""
expectancy: ""
version: <today>
---

# Experiment: <variant name>

## Thesis

<WHY this variant works — not what config changed>

## Evidence

### Live / paper sessions

| Started (UTC) | Dur (min) | Mode | Entries | Net PnL |
|---|---|---|---|---|

### Reading

<analysis of results — not just numbers>

## Change Log

- <today>: created. Initial thesis: ...
EOF

Then open the doc and read:

• The thesis (what's the claimed edge?)
• The version: date (when did the current config start? only sessions AFTER this count)
• The evidence table (how many sessions SINCE version date? what markets/conditions?)
• The current Reading section (what does it say?)
• The current grade/confidence/expectancy (if any)

Step 2: Gather the numbers — CURRENT VERSION ONLY

Only include sessions that ran under the current config (after the version: date). Sessions from before that date belonged to a previous iteration — they inform the Change Log, not the current grade. If version hasn't bumped since the doc was created, all sessions count.

This is the #1 grading mistake: computing metrics from the full history instead of the current version. If v1 lost $100 and v2 gained $50, the grade is about v2 (+$50), not the combined (-$50). The old v1 loss is in the Change Log. It does not touch the current grade.

Compute from current-version sessions:

Total net PnL:           $___  (sessions since version date only)
Sessions with fills:     ___
Win rate:                ___% (___W / ___L)
Average win:             $___
Average loss:            $___
Win/loss ratio:          ___:1
Max drawdown:            $___
Max DD as % of bankroll: ___%
Fee drag:                ___% of gross PnL

If the doc has a session evidence table, filter rows to those after version: date. If not, ask the operator where session data lives (session-summary.json files, ledger, runtime logs).

Name your source. Do not present a number without stating where it came from:

• session-summary.json files in .runtime/
• Execution ledger JSONL
• Venue API position/balance queries
• Scorer replay output

If multiple sources exist and disagree, note the disagreement — don't silently pick one. State which is the authoritative surface and why.

State the observation window. Record the exact range:

• "7 sessions from 2026-04-19 to 2026-04-20 (since version bump)"
• "all 14 sessions since version 2026-04-18"

Record how to reproduce. Any future agent should be able to re-derive the same numbers:

# Example reproduction command
find .runtime/<executor>/runs -name session-summary.json | \
  xargs grep -l '"variantName":"<slug>"' | \
  xargs jq -s '[.[] | select(.startedAt >= "2026-04-19")]'

If Sortino/Calmar can be computed (≥ 1 loss + ≥ 3 sessions since version), compute them too. They're supporting evidence, not grade determinants.

Step 3: Determine the class

First, determine WHICH class:

If the evidence is...	Class
Signal exploration, thesis analysis, early replay, model outputs	I — Discovery
Rigorous backtesting, dry runs, paper sessions with realistic fills	II — Validation
Live capital deployed, real venue fills, production sessions	III — Production

Don't overclass: replay-only evidence is Class I or II, never III. Don't underclass: if it's running live, it's Class III even if the sample is small.

Step 4: Assess the letter within that class

Ask the class-appropriate question:

Class	The question
I	"Is there a signal here worth investigating?"
II	"Does the edge survive realistic conditions?"
III	"Are the live economics working?"

Then assign:

Letter	Answer
A	"Yes, clearly — strong and ready for the next stage"
B	"Yes — solid, real, worth continuing"
C	"Maybe — something's there but it's thin"
D	"Can't tell yet"
F	"No — thesis disproved at this stage"

Combine: class + letter = grade. e.g. II-B, III-D, I-A.

Check against guardrails:

• Net negative with ≥ 5 data points at this class → cannot be above *-D
• Net negative with ≥ 10 data points at this class → must be *-F
• 0 evidence at this class level → must be *-D
• Class III only: max DD > 50% of bankroll → cannot be *-A

Special cases:

• Edge proved in replay but live can't capture it → Class III, C or D (execution gap, not thesis failure)
• All wins, zero losses → grade from PnL + consistency; don't penalize for not losing
• Config changed → bump version:, reset letter to D at the SAME class

Step 5: Assess confidence

Ask: "Has this strategy been exposed to enough conditions that I'd trust the grade to hold?"

Tag	What qualifies
hypothesis	Barely tested. Few trades, one market condition, one regime, replay-only, or just started.
pilot	Moderate exposure. Real trades across multiple conditions/markets, but hasn't seen serious stress or regime diversity.
proven	Deep exposure. Traded through volatility shifts, drawdowns, different market conditions. Grade survived adversity.
battle-tested	Extensive. Multiple regimes, stress events, long time span, diverse market conditions. Highest conviction.

This is judgment about EXPOSURE DIVERSITY, not a mechanical count. A strategy with 10 sessions across 50 diverse markets in both trending and choppy conditions could be proven. A strategy with 100 sessions all in the same quiet regime is still pilot.

Step 6: Write the expectancy

One line, quantitative, scannable. Include the source and window:

"+$11.44/session (session-summary.json, 7 sessions since 2026-04-19), 7W/2L, max DD $93"

Or for a losing variant:

"-$12.30/session (ledger, 9 sessions since 2026-04-18), 2W/7L, max DD $172"

Or for an inconclusive one:

"+$3.20/session (session-summary.json, 3 sessions since 2026-04-20), 3W/0L, too early"

Include: direction + magnitude, (source, count, since version), W/L ratio, drawdown. An operator scanning 100 experiments reads this without opening the doc AND knows where the number came from.

Step 7: Update the doc

1. Frontmatter: set grade:, confidence:, expectancy:, version:
2. Reading section — must contain ALL of:
   • Source: "computed from session-summary.json files" (or ledger, or venue API — name it)
   • Window: "7 sessions from 2026-04-19 to 2026-04-20 (since version bump)"
   • Numbers: the key metrics that support the letter grade
   • Caveats: any surface disagreements, known limitations, or reasons the grade might be wrong
   • What would change the grade: up or down — what evidence would shift it
   • Reproduction: command or path to re-derive ("files at .runtime/.../session-summary.json filtered by variantName + date")
3. Change Log: append an entry noting the grade change and why.

---
status: testing
reviewed: 2026-04-20
grade: C
confidence: pilot
expectancy: "+$8.50/session (session-summary.json, 5 sessions since 2026-04-19), 5W/2L, max DD $50"
version: 2026-04-19
---

Step 8: Verify consistency

Read the completed doc top-to-bottom. Does the grade match the Reading? Does the expectancy match the evidence table? Does the confidence match the exposure described in the thesis?

If anything contradicts, fix it. The adversarial-verifier will catch inconsistencies later — catch them yourself first.

Quick grading for operators scanning many experiments

When you need to grade or re-grade many experiments at once, for each doc:

1. Read evidence table → compute expectancy one-liner
2. Gut-check: "money behind this?" → letter
3. "Enough conditions seen?" → tag
4. Write all three, update Reading briefly, append Change Log
5. Move to next

Don't overthink it. The grade is a signal, not a PhD defense. It changes as evidence arrives. Assign honestly and move on.

Common mistakes

Mistake	Fix
Flat letter without class	grade: "B" is invalid. Must be "I-B", "II-B", or "III-B". The class prefix is required. The gate rejects grades without it.
Wrong class for the evidence	Replay-only evidence is Class I or II. Live sessions are Class III. Don't overclass or underclass.
Grading from full history	Only sessions since version: date count. If v1 was *-F and v2 is profitable, the grade is about v2.
I-B treated as failing	An I-B is a LEAD — a promising signal worth investigating. It's doing well for its class. Don't undersell discovery-stage work.
III-F applied to Class I work	A thesis in early replay that hasn't shown signal yet is I-D, not III-F. F at the wrong class misrepresents the stage.
*-F for "haven't computed yet"	*-F means thesis disproved at that class. If you haven't computed, it's *-D. Do the computation.
F from old-version sessions	Losses from a previous config → bump version:, reset to *-D. The F belongs to the old version's Change Log.
Grade contradicts Reading	If Reading says "strong signal in replay" but grade is II-F, something's wrong. Fix one or the other.