geode-eval

name

geode-eval

description

Scope, preregister, execute, audit, normalize, and publish GEODE evaluations with research-question, reproduction, attempt-lineage, trajectory, verifier-receipt, and immutable artifact contracts. Use for benchmark runs or comparisons, eval-artifact and cache-accounting audits, GPT or subscription result normalization, research question or hypothesis framing, retry diagnosis, trajectory publication, DPO or reward-data readiness reviews, and requests involving MCPMark, tau2, BFCL, Agent-World, HAL, Terminal-Bench, or Toolathlon.

GEODE Evaluation

Preserve the question, evidence, and judgement as separate authorities. Reuse the existing native result, trajectory, verifier receipt, and release formats; do not create a second copy of raw evidence.

Preserve v1 and derive learning views

Read the packaged schemas before proposing new storage. geode.trajectory@1 is an immutable normalized view over existing execution evidence; it already carries event, session, turn, and call identity, recomputed scope and replay integrity, and digest references to external evidence.

Do not create a unified raw-log store or retrofit new fields into v1. When a downstream episode, attempt, or decision join needs identity or replay preconditions that v1 cannot express, add a new schema version with an explicit migration and admission rule.

The v1 outcome object is permissive, so preserve any reward or outcome summary already present in an immutable artifact without treating it as label authority. For new learning views, keep reward, preference, process, and verifier labels out of the raw trajectory. Store them as digest-bound derived label or analysis artifacts with target identity, evaluator identity and revision, scope, provenance, and write authority. Call the result a post-training candidate, not a training dataset, until policy and environment identity, join closure, transition and replay semantics, privacy, deduplication, lineage-safe splits, and label quality pass.

Start With The Catalog

1. Read docs/eval/index.json.
2. Select documents by triggers, status, and authority.
3. Read docs/eval/README.md only when the catalog does not answer the routing question.
4. Load a specialized skill after the general contract when needed:
   • Agent-World or mean_accuracy@8: .claude/skills/agent-world-benchmark/
   • best-of-N, repair depth, replication, or promotion authority: .claude/skills/stanford-test-time-compute/

Do not scan every eval document or load site/public/llms-full.txt by default.

For token/cache/cost evidence, read the usage accounting contract. Use its actual field map, producer-specific zero/missing rules and source-bound reconciliation gates. Keep accounting coverage separate from the paired score denominator; no post-hoc cache repair replaces frozen raw results or grants publication authority. Load the detailed contract only for that scope.

Freeze Before Execution

Copy docs/eval/eval-run-spec.template.json to the run directory and replace every placeholder before any model or paid service call. Freeze:

• research question, research gap, and measurable hypothesis;
• primary metric with unit, direction, frozen denominator, and aggregation;
• decision, invalidation, and analysis rules;
• GEODE/harness/model/environment revisions and reset evidence;
• ordered workload IDs, canonical hash, seeds, repetitions, budgets, and route;
• comparator class, comparability, promotion authority, artifact destinations, and privacy boundary.

Validate the record:

uv run python scripts/eval/contract.py validate-run-spec <run-dir>/run-spec.json

Live model, account, quota, or remote-service calls require explicit user approval. Approval belongs in the frozen spec; it does not relax redaction or publication gates.

Preserve Attempt Lineage

Create each row from docs/eval/eval-attempt.template.json and append it to attempts.jsonl. Never edit or delete an earlier row to make a retry look like the first attempt.

• Use one run ID and unique, contiguous attempt IDs/sequences.
• Point a retry or repair at an already-recorded parent.
• Record the changed surface and expected effect before interpreting it.
• Keep measurement validity separate from semantic outcome: a valid full run may have mixed task outcomes. Infrastructure contamination is invalid, never a zero score or a semantic failure.
• Record timing as timezone-qualified exact with a known offset, zone-less source-naive plus its source, or all-null unknown; -00:00 is not exact, and a historical source never gains an invented timezone.
• A harness-valid semantic failure may retain a failure class and error reference. An infrastructure-invalid or aborted attempt uses outcome=unknown, never a semantic fail score.
• Reference raw output, errors, trajectories, and receipts by run-directory relative path plus SHA-256 instead of copying their payload into the row. Only portable POSIX-relative paths are accepted; URI, UNC, drive-root, backslash, and parent traversal forms are forbidden. Point at a manifest when evidence is a directory.
• Select attempts using the frozen rule, not the observed score.

Validate the append-only projection:

uv run python scripts/eval/contract.py validate-attempts <run-dir>/attempts.jsonl

Keep Authorities Separate

Surface	Authority	Lifecycle
Checkpoint	resume state	mutable
Session record and attempts	chronological execution and retry lineage	append-only
Native harness result	suite score	append-only producer output
GEODE trajectory	normalized behavior	append-only projection
Verifier receipt or state diff	judgement evidence	append-only producer output
Analysis	answer and decision	digest-bound, immutable after publication
Trajectory release and artifact manifest	admitted public bytes and integrity	immutable

Runtime activity such as streaming chunks, heartbeat, cache internals, and transient retry telemetry stays out of training/evaluation claims unless a specific evidence reference promotes it deliberately.

Analyze Against The Frozen Question

Create analysis.json from docs/eval/eval-analysis.template.json. Bind it to the SHA-256 of the frozen spec and complete attempts JSONL. Report the primary metric with numerator and denominator, answer only the preregistered question, apply the frozen decision/invalidation rules, and preserve limitations.

uv run python scripts/eval/contract.py validate-analysis <run-dir>/analysis.json \
  --run-spec <run-dir>/run-spec.json \
  --attempts <run-dir>/attempts.jsonl

The validator requires selected attempt IDs, evidence digests, primary-metric unit/counts/source pointer, and registration chronology to match the source sidecars. A measured primary metric uses RFC 6901 JSON Pointers to read its value and counts from selected-attempt, digest-bound native JSON, and its denominator matches the explicitly frozen denominator. If an invalid or aborted row is selected for provenance, the primary metric is not-measurable with null counts and cannot drive promotion or rejection. Add new analysis as a superseding artifact; do not rewrite a published analysis in place.

Publish Without Inflating Claims

Follow docs/eval/benchmark-publishing-cycle.md and docs/eval/external-artifact-repository.md.

1. Keep native score, behavior trajectory, and verifier evidence distinct.

2. Stage stable geode.trajectory@1 records and geode.trajectory-release@1 manifests when behavior evidence is published.

3. Build artifact-publish-manifest.template.json, privacy review the exact bytes, publish append-only, and verify remote read-back at the pinned commit. Validate portable paths, classification, byte counts, SHA-256 identities, and prepared/published state before copying:

   uv run python scripts/eval/contract.py validate-publication \
     <run-dir>/publication-manifest.json
   

   Raw prompts/responses, transcripts, messages, SQLite/WAL, evidence JSONL, profiles, usage, diagnostics, and provider payloads remain withheld-private unless their exact bytes receive public approval.

4. Derive the human run report and public page from validated sidecars; do not make either another score authority.

5. Label subscription results as product-route evidence. Keep smoke, directional, paired-runtime, suite-headline, and promotion claims separate.

For historical runs, set preregistration.mode=retrospective, force promotion_authority=none, and add immutable sidecars with explicit unknowns and source digests. Never invent missing timestamps, seeds, budgets, prompts, simulator identity, or task selection, and never rewrite old raw receipts.

Fail Loud

• Reject unresolved <...> placeholders.
• Reject workload hash or seed/repetition mismatch.
• Reject duplicate attempts, forward parent links, sequence gaps, mixed run IDs, or end times before start times.
• Reject analysis whose source digests or selected attempts do not match.
• Reject prospective attempts that predate the frozen spec, promotion or rejection decisions without authority, and authority that does not match a direct named comparator and its claim class.
• Reject human preference, factual correctness, or DPO-readiness claims based only on a terminal score; require explicit preference/process labels and verifier evidence appropriate to the claim.
• Reject publication when secrets, identities, raw private prompts, or any POSIX, Windows-drive, UNC, or home-relative machine path crosses the declared privacy boundary.

Finish by reporting the research question, invalidity status, selected attempt lineage, primary numerator/denominator, artifact commit, comparability class, promotion authority, and the smallest unresolved measurement.