context-intelligence-eval-design

name	context-intelligence-eval-design
version	1.0.0
description	Use when designing evaluation scenarios for a context-intelligence tool signal — derives success criteria from domain-concepts.md and produces evaluation-scenarios.md entries and DTU profile templates.
user-invocable	false
allowed-tools	read_file, glob, grep, delegate, load_skill, todo
model_role	reasoning
license	MIT

Context Intelligence Evaluation Design

Phase 3 specialist skill for context-intelligence mode. Called by context-intelligence-tool-designer agent via self-delegation with context_depth="none", scoped to one concept at a time.

---

Scope

In scope:

• Derive success criteria from domain-concepts.md
• Design one evaluation scenario per concept
• Populate DTU profile templates including mandatory Gitea URL rewrite block

Out of scope:

• Investigating signals
• Classifying detection strategies
• Running evaluations

Note: The evaluation runner is amplifier-bundle-evaluation invoked inside a DTU built from this scenario.

---

Criteria Derivation Rule

Success criteria MUST come from the user-confirmed concept definition in domain-concepts.md — specifically the Agreed definition, Boundary conditions, Data availability, and Explicitly excludes fields. NEVER derive from implementation details.

What a detector returns is not the criterion; what correct detection looks like in plain terms from the user's domain perspective is the criterion.

If the definition is too vague to produce a testable criterion, return a structured gap entry rather than guessing.

---

Per-Concept Evaluation Scenario Format

Each scenario is ready for insertion into evaluation-scenarios.md:

## Scenario [N]: [Concept] — [description]

**Derived from:** [field reference in domain-concepts.md — Agreed definition / Boundary conditions / Data availability / Explicitly excludes]

**Success criterion:** [Plain-language statement of what correct detection looks like from the user's domain perspective. Not what the detector returns — what it means.]

**Failing scenario:** [Concrete example of a case where detection fails or produces a false positive, derived from Explicitly excludes or Boundary conditions.]

**DTU environment:** [Reference to DTU profile template produced by this skill for this concept.]

**Pass threshold:** [Quantitative or qualitative threshold — e.g., "detects in ≥8 of 10 representative sessions" or "no false positives on the Explicitly excludes examples".]

**Iteration question:** [The question this scenario answers. Used to route delta back through Phase 4. E.g., "Does the signal correctly distinguish X from Y as the user defined it?"]

---

DTU Profile Template — Mandatory Gitea URL Rewrite

Every DTU profile generated by this skill MUST include the URL rewrite block below. This block is non-optional.

The explanatory comment is included inline so readers cannot remove the block without understanding why it exists.

# MANDATORY: Gitea URL rewrite for local branch skill resolution.
# The bundle ships skills with a git+https://github.com/microsoft/amplifier-bundle-context-intelligence@main#subdirectory=skills source in behaviors/context-intelligence.yaml.
# When running in DTU, local working branch is source of truth — NOT GitHub @main. Without rewrite, skill registration inside DTU fetches from GitHub @main and silently runs stale code, producing test results that do not reflect local changes.
# The digital-twin-universe:dtu-profile-builder agent mirrors the local amplifier-bundle-context-intelligence submodule to Gitea on current branch. These rewrites point every reference at that mirror.
url_rewrites:
  - from: "git+https://github.com/microsoft/amplifier-bundle-context-intelligence@main"
    to:   "git+http://{gitea_host}/microsoft/amplifier-bundle-context-intelligence@main"

This block must appear in EVERY profile this skill produces. {gitea_host} is resolved by digital-twin-universe:dtu-profile-builder from the running Gitea instance.

---

Two-Layer Validation Structure

Layer 1 — Structural Validation

Seven deterministic test points encoded as assertions in the DTU profile:

1. Mode inactive blocks delegate. When context-intelligence mode is not active, delegate calls to mode-scoped agents are blocked.
2. Mode activates correctly. Activating the mode enforces tool policies and injects both @mention context files into the session.
3. Both specialist agents available under mode. context-intelligence-design-facilitator AND context-intelligence-tool-designer are invokable while mode is active.
4. Tool-designer NOT accessible without mode. context-intelligence-tool-designer is not invokable when mode is inactive.
5. domain-concepts.md exists after Phase 0 with all required fields per entry. Required fields: User intent, Agreed definition, Boundary conditions, Nesting, Data availability, Explicitly excludes.
6. domain-signals.md enriched after Phase 2. Each signal entry contains: detection_strategy, detection_notes, ai_dependency, reasoning_requirement, suggested_primitive.
7. context-intelligence-primitives-reference.md NOT in session context when mode inactive. The primitives reference must not leak into sessions where mode has not been activated.
8. context-intelligence-strategy.md absent when mode inactive. The thin mode-orientation strategy file must NOT appear in session context unless the context-intelligence mode is active (parallels point 7; guards mode-only injection via contributes.context:).
9. Strategy-file pointer correctness. Every referenced path and skill name inside context-intelligence-strategy.md resolves to an existing file/skill in the bundle (guards against rename-induced dead pointers). These are NON-loading references (plain paths / skill names), in contrast to the LOADING @mention that session-navigator uses for the discipline file — this check covers the non-loading pointers.
10. Tool-design enrichment marker present in-mode / absent in baseline. The new R1 marker string "module vs CLI" is PRESENT in tool-design context when the mode is active, and ABSENT from the always-on baseline. (Anchors the test on a concrete, buildable marker introduced by this change, rather than the non-falsifiable "enriched content loads only when mode active".)

Layer 2 — Behavioural Evaluation

Run by amplifier-bundle-evaluation inside the same DTU, after Layer 1 passes.

Scenario A — concept elicitation: User enters mode and says: "investigate how testing sessions are performing."

Pass criterion: domain-concepts.md contains a user-confirmed definition for "testing session" with non-empty Boundary conditions, Data availability, and Explicitly excludes. The definition must not reference Amplifier agent names.

Scenario B — signal gap loop: Tool-designer encounters an ambiguous concept during Phase 2 or Phase 3.

Pass criterion: A signal-gaps.md entry is written with Gap type, Question, and Blocks fields; the facilitator resolves it; the entry status changes to resolved.

Scenario C — bounded navigation (discipline still fires after extraction): Reuses the workspace harness evaluations/01-delegation-runaway with SCENARIO=disk_absent (the after arm carries the local working branch). session-navigator, given a non-existent session ID against a seeded corpus, must terminate in ≤3 strategies with "not found".

Pass criterion: runaway_detected=false, meta.exit_code=0, and bounded context_health (peak/total tokens not climbing; compaction_count 0–1). This proves the @mentioned navigation-budget-discipline.md is actually followed.

---

Iteration Loop

Layer 1 → Layer 2 → compare against success criteria in evaluation-scenarios.md
        → route delta back via mode's Phase 4 validation gap protocol

• Signal or concept definition wrong → re-enter Phase 0 or Phase 1 via Facilitator.
• Detection strategy or tool choice wrong → re-enter Phase 2 or Phase 3 via Tool Designer.

Iteration is the user's call, not the agent's. This skill produces scenarios; the user decides when to iterate.