context-intelligence-eval-design

# Context Intelligence Evaluation Design Phase 3 specialist skill for context-intelligence mode. Called by context-intelligence-tool-designer agent via self-delegation with context_depth="none", scoped to one concept at a time. --- ## Scope **In scope:** - Derive success criteria from `domain-concepts.md` - Design one evaluation scenario per concept - Populate DTU profile templates including mandatory Gitea URL rewrite block **Out of scope:** - Investigating signals - Classifying detection strategies - Running evaluations Note: The evaluation runner is `amplifier-bundle-evaluation` invoked inside a DTU built from this scenario. --- ## Criteria Derivation Rule Success criteria MUST come from the user-confirmed concept definition in `domain-concepts.md` — specifically the `Agreed definition`, `Boundary conditions`, `Data availability`, and `Explicitly excludes` fields. NEVER derive from implementation details. What a detector returns is not the criterion; what correct detection looks like in plain terms from the user's domain perspective is the criterion. If the definition is too vague to produce a testable criterion, return a structured gap entry rather than guessing. --- ## Per-Concept Evaluation Scenario Format Each scenario is ready for insertion into `evaluation-scenarios.md`: ```markdown ## Scenario [N]: [Concept] — [description] **Derived from:** [field reference in domain-concepts.md — Agreed definition / Boundary conditions / Data availability / Explicitly excludes] **Success criterion:** [Plain-language statement of what correct detection looks like from the user's domain perspective. Not what the detector returns — what it means.] **Failing scenario:** [Concrete example of a case where detection fails or produces a false positive, derived from Explicitly excludes or Boundary conditions.] **DTU environment:** [Reference to DTU profile template produced by this skill for this concept.] **Pass threshold:** [Quantitative or qualitative threshold — e.g., "detects in ≥8 of 10 representative sessions" or "no false positives on the Explicitly excludes examples".] **Iteration question:** [The question this scenario answers. Used to route delta back through Phase 4. E.g., "Does the signal correctly distinguish X from Y as the user defined it?"] ``` --- ## DTU Profile Template — Mandatory Gitea URL Rewrite Every DTU profile generated by this skill MUST include the URL rewrite block below. This block is non-optional. The explanatory comment is included inline so readers cannot remove the block without understanding why it exists. ```yaml # MANDATORY: Gitea URL rewrite for local branch skill resolution. # The bundle ships skills with a git+https://github.com/microsoft/amplifier-bundle-context-intelligence@main#subdirectory=skills source in behaviors/context-intelligence.yaml. # When running in DTU, local working branch is source of truth — NOT GitHub @main. Without rewrite, skill registration inside DTU fetches from GitHub @main and silently runs stale code, producing test results that do not reflect local changes. # The digital-twin-universe:dtu-profile-builder agent mirrors the local amplifier-bundle-context-intelligence submodule to Gitea on current branch. These rewrites point every reference at that mirror. url_rewrites: - from: "git+https://github.com/microsoft/amplifier-bundle-context-intelligence@main" to: "git+http://{gitea_host}/microsoft/amplifier-bundle-context-intelligence@main" ``` This block must appear in EVERY profile this skill produces. `{gitea_host}` is resolved by `digital-twin-universe:dtu-profile-builder` from the running Gitea instance. --- ## Two-Layer Validation Structure ### Layer 1 — Structural Validation Seven deterministic test points encoded as assertions in the DTU profile: 1. **Mode inactive blocks delegate.** When context-intelligence mode is not active, delegate calls to mode-scoped agents are blocked. 2. **Mode activates correctly.** Activating the mode enforces tool policies and injects both @mention context files into the session. 3. **Both specialist agents available under mode.** `context-intelligence-design-facilitator` AND `context-intelligence-tool-designer` are invokable while mode is active. 4. **Tool-designer NOT accessible without mode.** `context-intelligence-tool-designer` is not invokable when mode is inactive. 5. **`domain-concepts.md` exists after Phase 0 with all required fields per entry.** Required fields: `User intent`, `Agreed definition`, `Boundary conditions`, `Nesting`, `Data availability`, `Explicitly excludes`. 6. **`domain-signals.md` enriched after Phase 2.** Each signal entry contains: `detection_strategy`, `detection_notes`, `ai_dependency`, `reasoning_requirement`, `suggested_primitive`. 7. **`context-intelligence-primitives-reference.md` NOT in session context when mode inactive.** The primitives reference must not leak into sessions where mode has not been activated. 8. **`context-intelligence-strategy.md` absent when mode inactive.** The thin mode-orientation strategy file must NOT appear in session context unless the context-intelligence mode is active (parallels point 7; guards mode-only injection via `contributes.context:`). 9. **Strategy-file pointer correctness.** Every referenced path and skill name inside `context-intelligence-strategy.md` resolves to an existing file/skill in the bundle (guards against rename-induced dead pointers). These are NON-loading references (plain paths / skill names), in contrast to the LOADING `@mention` that `session-navigator` uses for the discipline file — this check covers the non-loading pointers. 10. **Tool-design enrichment marker present in-mode / absent in baseline.** The new R1 marker string **"module vs CLI"** is PRESENT in tool-design context when the mode is active, and ABSENT from the always-on baseline. (Anchors the test on a concrete, buildable marker introduced by this change, rather than the non-falsifiable "enriched content loads only when mode active".) ### Layer 2 — Behavioural Evaluation Run by `amplifier-bundle-evaluation` inside the same DTU, after Layer 1 passes. **Scenario A — concept elicitation:** User enters mode and says: *"investigate how testing sessions are performing."* Pass criterion: `domain-concepts.md` contains a user-confirmed definition for "testing session" with non-empty `Boundary conditions`, `Data availability`, and `Explicitly excludes`. The definition must not reference Amplifier agent names. **Scenario B — signal gap loop:** Tool-designer encounters an ambiguous concept during Phase 2 or Phase 3. Pass criterion: A `signal-gaps.md` entry is written with `Gap type`, `Question`, and `Blocks` fields; the facilitator resolves it; the entry status changes to `resolved`. **Scenario C — bounded navigation (discipline still fires after extraction):** Reuses the workspace harness `evaluations/01-delegation-runaway` with `SCENARIO=disk_absent` (the `after` arm carries the local working branch). `session-navigator`, given a non-existent session ID against a seeded corpus, must terminate in ≤3 strategies with "not found". Pass criterion: `runaway_detected=false`, `meta.exit_code=0`, and bounded `context_health` (peak/total tokens not climbing; `compaction_count` 0–1). This proves the `@mention`ed `navigation-budget-discipline.md` is actually followed. --- ## Iteration Loop ``` Layer 1 → Layer 2 → compare against success criteria in evaluation-scenarios.md → route delta back via mode's Phase 4 validation gap protocol ``` - **Signal or concept definition wrong** → re-enter Phase 0 or Phase 1 via Facilitator. - **Detection strategy or tool choice wrong** → re-enter Phase 2 or Phase 3 via Tool Designer. **Iteration is the user's call, not the agent's.** This skill produces scenarios; the user decides when to iterate.