context-intelligence-evaluation-methodology

# Context Intelligence Evaluation Methodology Mode-only skill for *how to measure*. It complements — and never restates — `context-intelligence-eval-design` (which owns scenario mechanics and the two-layer structural/behavioral structure) and `digital-twin-universe` (which owns the DTU machinery). ## Scope ### In scope - **Metric design.** Choose metrics across three axes: **quality** (did it detect what the user means?), **efficiency** (token/tool cost to detect), **efficacy** (does detection drive the right outcome?). - **Measure the precursor, not only the failure.** Prefer leading indicators (e.g. bounded vs climbing context growth) over lagging ones (e.g. a final timeout). The precursor is testable in a short window; the full failure often is not. - **A/B + statistical-N discipline.** A single green run is not proof of a behavioral change. Compare a control arm against a treatment arm; require N independent trials and report the pass rate, not a single anecdote. - **Test data fidelity.** Validate on **real sessions** where available, or on **faithfully modeled synthetic** corpora that preserve the distribution that matters (e.g. heavy-tail file sizes), never on data hand-shaped to pass. ### Out of scope (point elsewhere — do not restate) - **Scenario mechanics, the two-layer structure, DTU profile templates / Gitea URL rewrite** → `context-intelligence-eval-design`. - **DTU lifecycle, launch/exec/profiles** → the `digital-twin-universe` skill. - **"DTU-as-default" rationale and the "artifact-as-success" anti-pattern** → already authoritative in `context-intelligence-eval-design` and `context-intelligence:context/context-intelligence-primitives-reference.md`. Reference them; do not repeat them. ## Method 1. **Name the user-meaningful outcome** the metric must reflect (from `domain-concepts.md`), not what a detector happens to emit. 2. **Pick the cheapest axis that discriminates.** If a deterministic efficiency signal (token growth, tool-call count) separates good from bad behavior, prefer it over an LLM-judged quality rubric. 3. **Identify the precursor.** Ask: what climbs *before* the failure? Measure that in a bounded window. 4. **Design the A/B.** Control = current behavior; treatment = the change. Hold everything else identical. Decide N and the pass threshold *before* running. 5. **Choose the corpus.** Real sessions if available; otherwise synthetic that preserves the load-bearing distribution. State the fidelity assumption explicitly. > **Apply (do not restate) the event-semantics principle:** when a metric depends on what an > event *means*, consult the authoritative ecosystem expert agents for that event's semantics — > the principle is named once in > `context-intelligence:context/context-intelligence-strategy.md`. Do not restate it here.