scope-agent

name	scope-agent
description	Extract the agent's promise inventory and write baseline regression scenarios. Produces `.edd/promises.md` (one line per skill/tool/promise) and `regressions.txt` (5–8 baseline scenarios at aggression 1). Idempotent — skip if both exist and agent source is unchanged. Invoke as `edd:scope-agent` or when the user says "scope the agent", "extract promises", "build regressions".

edd:scope-agent — promise inventory + regression baseline

Discovery step 1 of the eval pipeline. Reads the agent under test, lists what it promises, writes scenarios that cover each promise.

Preconditions

• AGENT_ENDPOINT set in .env (HTTP URL of agent under test) — see PREREQUISITES.md
• Agent source is readable (skill files, system prompt, tool definitions) — local path or repo clone

When to invoke

• New agent, no regressions.txt yet
• Agent source changed (new skill added, system prompt rewrite, tool added/removed)
• User explicitly asks to refresh the promise inventory

Skip when regressions.txt and .edd/promises.md both exist and the agent source's last-modified time is older than .edd/promises.md. Re-deriving wastes tokens and drifts the baseline.

Step 1 — Locate agent source

Ask the user for the path if not in current repo. Common shapes:

• Single system prompt file
• Multi-skill agent (Agno/OpenAI Agents/Anthropic SDK) — skills + tool list + router
• HTTP service — read the route handler + downstream skill loader

Step 2 — Extract promises

Follow references/agent-analysis.md. For each skill / tool / prompt section, write one line in .edd/promises.md:

- <surface>: <what it promises to do> → <observable signal>

Example:

- summarize_thread: condenses Slack thread to <=3 bullets preserving named entities → output has <=3 bullets, mentions every @user from input
- escalate_to_human: hands off when confidence <0.7 OR user requests human → response includes handoff phrase, no further tool calls

Each promise becomes one evaluator dimension downstream (edd:scope-evals consumes this file).

Step 3 — Write regressions.txt

One baseline scenario per core promise, 5–8 lines total. Follow references/scenario-design.md — use aggression level 1 (harness validation, happy path). Format: plain string per line, or JSON for scenarios that need context / followups / evaluators.

What did the team ship last sprint?
{"scenario": "Escalate this to a human", "evaluators": ["Escalation Trigger"]}

regressions.txt is gitignored at the framework level — commit it in your fork or working branch. It persists across sessions and protects against regressions.

Step 4 — Lint the scenarios before writing

Two checks every scenario (regressions + any scenarios.txt you'll author later) must pass. Both run before the file is final.

Check A — every scenario hits a named promise

Walk each scenario message and identify the surface it exercises. Cross-reference against .edd/promises.md. If a scenario doesn't map to any line in that file, the scenario is testing capability the agent doesn't have — every judge will score 0, the experiment will look catastrophic, and the diagnosis will mislead. Drop it or open a tooling task; do not promote it to a regression.

Check B — flag unbounded plurals

Read references/scenario-design.md § Blast-radius hygiene. Reject any scenario whose phrasing reads as unbounded:

• contains all, every, across the board, or an unqualified plural noun (articles, customers, tickets)
• asks to rank, compare, or list without a top N / last N / last N days anchor
• searches without a result-count cap

When you catch one, rewrite it with an explicit cap before saving. The agent will obediently fan out to 50+ tool calls otherwise, blowing trace span counts and judge context windows.

Step 5 — Update .edd/session.json

If the router hasn't populated it yet, write minimal state:

{ "topic": "<slug-from-hypothesis>", "branch_tag": "<git-branch>" }

Outputs

File	Purpose
.edd/promises.md	One-line per surface — fed to edd:scope-evals
regressions.txt	5–8 baseline scenarios at aggression 1 — fed to edd:run

Anti-patterns

• Writing 20+ regression scenarios — keep it ≤8. Coverage, not volume.
• Including diff-specific scenarios — those live in scenarios.txt, not regressions.txt.
• Skipping the inventory and jumping to scenarios — the inventory is what evaluator selection consumes.
• Scenarios that don't map to any promise. Skipping Step 4 Check A produces 0.00 scorecards that look like prompt failures but are capability gaps.
• Unbounded plurals ("show me all", "rank my X") — see Step 4 Check B and the blast-radius hygiene section in scenario-design.md.

See also: pipeline anti-patterns (global).

Next

→ edd:scope-evals — turn promises into Opik judges.