| name | context-intelligence-eval-design |
| version | 1.0.0 |
| description | Use when designing evaluation scenarios for a context-intelligence tool signal — derives success criteria from domain-concepts.md and produces evaluation-scenarios.md entries and DTU profile templates. |
| user-invocable | false |
| allowed-tools | read_file, glob, grep, delegate, load_skill, todo |
| model_role | reasoning |
| license | MIT |
Context Intelligence Evaluation Design
Phase 3 specialist skill for context-intelligence mode. Called by context-intelligence-tool-designer agent via self-delegation with context_depth="none", scoped to one concept at a time.
Scope
In scope:
- Derive success criteria from
domain-concepts.md
- Design one evaluation scenario per concept
- Populate DTU profile templates including mandatory Gitea URL rewrite block
Out of scope:
- Investigating signals
- Classifying detection strategies
- Running evaluations
Note: The evaluation runner is amplifier-bundle-evaluation invoked inside a DTU built from this scenario.
Criteria Derivation Rule
Success criteria MUST come from the user-confirmed concept definition in domain-concepts.md — specifically the Agreed definition, Boundary conditions, Data availability, and Explicitly excludes fields. NEVER derive from implementation details.
What a detector returns is not the criterion; what correct detection looks like in plain terms from the user's domain perspective is the criterion.
If the definition is too vague to produce a testable criterion, return a structured gap entry rather than guessing.
Per-Concept Evaluation Scenario Format
Each scenario is ready for insertion into evaluation-scenarios.md:
## Scenario [N]: [Concept] — [description]
**Derived from:** [field reference in domain-concepts.md — Agreed definition / Boundary conditions / Data availability / Explicitly excludes]
**Success criterion:** [Plain-language statement of what correct detection looks like from the user's domain perspective. Not what the detector returns — what it means.]
**Failing scenario:** [Concrete example of a case where detection fails or produces a false positive, derived from Explicitly excludes or Boundary conditions.]
**DTU environment:** [Reference to DTU profile template produced by this skill for this concept.]
**Pass threshold:** [Quantitative or qualitative threshold — e.g., "detects in ≥8 of 10 representative sessions" or "no false positives on the Explicitly excludes examples".]
**Iteration question:** [The question this scenario answers. Used to route delta back through Phase 4. E.g., "Does the signal correctly distinguish X from Y as the user defined it?"]
DTU Profile Template — Mandatory Gitea URL Rewrite
Every DTU profile generated by this skill MUST include the URL rewrite block below. This block is non-optional.
The explanatory comment is included inline so readers cannot remove the block without understanding why it exists.
url_rewrites:
- from: "git+https://github.com/microsoft/amplifier-bundle-context-intelligence@main"
to: "git+http://{gitea_host}/microsoft/amplifier-bundle-context-intelligence@main"
This block must appear in EVERY profile this skill produces. {gitea_host} is resolved by digital-twin-universe:dtu-profile-builder from the running Gitea instance.
Two-Layer Validation Structure
Layer 1 — Structural Validation
Seven deterministic test points encoded as assertions in the DTU profile:
- Mode inactive blocks delegate. When context-intelligence mode is not active, delegate calls to mode-scoped agents are blocked.
- Mode activates correctly. Activating the mode enforces tool policies and injects both @mention context files into the session.
- Both specialist agents available under mode.
context-intelligence-design-facilitator AND context-intelligence-tool-designer are invokable while mode is active.
- Tool-designer NOT accessible without mode.
context-intelligence-tool-designer is not invokable when mode is inactive.
domain-concepts.md exists after Phase 0 with all required fields per entry. Required fields: User intent, Agreed definition, Boundary conditions, Nesting, Data availability, Explicitly excludes.domain-signals.md enriched after Phase 2. Each signal entry contains: detection_strategy, detection_notes, ai_dependency, reasoning_requirement, suggested_primitive.context-intelligence-primitives-reference.md NOT in session context when mode inactive. The primitives reference must not leak into sessions where mode has not been activated.context-intelligence-strategy.md absent when mode inactive. The thin mode-orientation strategy file must NOT appear in session context unless the context-intelligence mode is active (parallels point 7; guards mode-only injection via contributes.context:).- Strategy-file pointer correctness. Every referenced path and skill name inside
context-intelligence-strategy.md resolves to an existing file/skill in the bundle (guards against rename-induced dead pointers). These are NON-loading references (plain paths / skill names), in contrast to the LOADING @mention that session-navigator uses for the discipline file — this check covers the non-loading pointers.
- Tool-design enrichment marker present in-mode / absent in baseline. The new R1 marker string "module vs CLI" is PRESENT in tool-design context when the mode is active, and ABSENT from the always-on baseline. (Anchors the test on a concrete, buildable marker introduced by this change, rather than the non-falsifiable "enriched content loads only when mode active".)
Layer 2 — Behavioural Evaluation
Run by amplifier-bundle-evaluation inside the same DTU, after Layer 1 passes.
Scenario A — concept elicitation:
User enters mode and says: "investigate how testing sessions are performing."
Pass criterion: domain-concepts.md contains a user-confirmed definition for "testing session" with non-empty Boundary conditions, Data availability, and Explicitly excludes. The definition must not reference Amplifier agent names.
Scenario B — signal gap loop:
Tool-designer encounters an ambiguous concept during Phase 2 or Phase 3.
Pass criterion: A signal-gaps.md entry is written with Gap type, Question, and Blocks fields; the facilitator resolves it; the entry status changes to resolved.
Scenario C — bounded navigation (discipline still fires after extraction):
Reuses the workspace harness evaluations/01-delegation-runaway with SCENARIO=disk_absent
(the after arm carries the local working branch). session-navigator, given a non-existent
session ID against a seeded corpus, must terminate in ≤3 strategies with "not found".
Pass criterion: runaway_detected=false, meta.exit_code=0, and bounded context_health
(peak/total tokens not climbing; compaction_count 0–1). This proves the @mentioned
navigation-budget-discipline.md is actually followed.
Iteration Loop
Layer 1 → Layer 2 → compare against success criteria in evaluation-scenarios.md
→ route delta back via mode's Phase 4 validation gap protocol
- Signal or concept definition wrong → re-enter Phase 0 or Phase 1 via Facilitator.
- Detection strategy or tool choice wrong → re-enter Phase 2 or Phase 3 via Tool Designer.
Iteration is the user's call, not the agent's. This skill produces scenarios; the user decides when to iterate.
