agentsop-regression-gate

# regression-gate — Eval Set + Metric + Threshold, Wired Into CI > "Every subsequent change must be gated on these numbers." > — Synthesized from [[llamaindex]] Stage 2 (eval loop *before* optimizing) [llamaindex-sop-skill/SKILL.md:114-126] > "Compiled program beats baseline on a *held-out* test set (not the val set used in optimization)." > — [[dspy]] Stage 3 exit criterion [dspy-sop-skill/SKILL.md:101] This is an **enhancement overlay**. The regression-gate SOP exists only as fragments scattered across base skills — [[llamaindex]] `OP-10 EvalLoop` ("gate every change"), [[dspy]] train/dev/test split + metric — and is never assembled as a standalone cross-framework discipline. It is the discipline that turns a one-off eval into a *gate*: a test suite that runs in CI on every prompt/model/retriever change and fails the build on regression. It consumes a metric from `[[agentsop-metric-design]]` and, for domain-specific held-out sets, hands off to `[[agentsop-domain-eval-set]]`. --- ## 1. 何时激活 (When to Activate) Activate when **any** of these is true: - **Any prompt change you want to ship safely**: a prompt edit, a system-message tweak, a few-shot-demo swap is about to merge and you have no automated way to know if it made things worse. - **Any model change**: swapping GPT-4o → a cheaper/newer model, a temperature change, a provider migration. An LM change silently shifts the whole output distribution. [[dspy]] Case B: "If you optimize a complex pipeline for GPT-4, it usually breaks on a smaller model" [dspy-sop-skill/SKILL.md:184]. - **Any retriever/chunking/reranker change in a RAG pipeline**: every such change needs a quantitative gate. [[llamaindex]] `OP-10`: "Quantitative regression test for every chunking / embedding / retriever / prompt change" [llamaindex-sop-skill/SKILL.md:235]. - **Recurring "it got worse" surprises**: the team keeps shipping changes that users report as regressions after the fact. The fix is a gate, not more careful review. - **Setting up CI for an LLM app** and there is no eval job in the pipeline. **Do NOT activate for:** - **One-shot tasks** with no production surface — there is nothing to regress. ([[dspy]] boundary: "Summarize this email once → raw API call" [dspy-sop-skill/SKILL.md:259].) - **The signature/task is still changing daily** — gate only after the I/O contract stabilizes, else you re-baseline every commit. ([[dspy]] boundary [dspy-sop-skill/SKILL.md:260].) - **No willingness to define any success criterion** — without a metric there is nothing to gate. Route to `[[agentsop-metric-design]]` first; if the user refuses, this skill cannot help. --- ## 2. 核心心智模型 (Core Mental Model) ### **"An LM change is a code change. Gate it with a test suite: eval set + metric + threshold."** You already gate code with unit tests in CI: a change that breaks a test fails the build. A prompt edit, a model swap, a chunk-size tweak are *also* changes to the system's behavior — but they slip through review because their effect is statistical, not a stack trace. The regression gate is the missing unit test for LM behavior. The gate is exactly three artifacts plus a wiring step: ``` eval set metric threshold CI wiring (held-out QA) (ex,pred)->score (fail if <X / drop>Y) (block merge) │ │ │ │ └──────────────┴──────────────────┴────────────────────┘ REGRESSION GATE ``` Three load-bearing principles: 1. **The eval set is held out and frozen.** It is a labelled, version-controlled fixture that the prompt/model under test has *never seen*. [[dspy]] is explicit: the compiled program must beat baseline on a held-out test set "not the val set used in optimization" [dspy-sop-skill/SKILL.md:101]. The split is `train / dev / test`; the **gate runs on test only**. If the eval set leaks into the prompt (few-shot demos, instructions), the gate measures memorization, not quality. 2. **The metric comes from `[[agentsop-metric-design]]`, not invented here.** This skill does not design metrics — it consumes one. A bad metric makes the gate theatre: it will pass changes that hurt users and block changes that help them. The metric must be human-calibrated *before* it gates anything ([[agentsop-metric-design]] `OP-M05`). 3. **The threshold is a policy, not a number you guess.** Two common shapes: an **absolute floor** (fail if score < X) and a **relative no-regression** (fail if score drops > Y from the committed baseline). Relative is the regression gate proper; absolute is a quality bar. Most teams use both: a floor for "never ship below this," plus a no-regression delta for "this PR must not make it worse." ### Build the eval loop *before* you optimize anything [[llamaindex]] Stage 2 is named "Build the eval loop **before** optimizing anything" [llamaindex-sop-skill/SKILL.md:114]. The anti-pattern it names is A3: "No eval loop; debug by anecdote" [llamaindex-sop-skill/SKILL.md:348]. The gate is the institutional form of that loop — once it exists, every change is debugged by number, not by vibe. --- ## 3. SOP (Standard Operating Procedure) ``` 0. Confirm activation (§1); confirm a metric exists or invoke [[agentsop-metric-design]] 1. BUILD eval set: generate candidates -> curate to a golden set -> freeze + version 2. SPLIT: train / dev / test; the GATE runs on TEST only 3. PICK metric: consume from [[agentsop-metric-design]] (do not invent here) 4. SET threshold: absolute floor AND/OR relative no-regression delta 5. WIRE into CI: run eval on every prompt/model/retriever PR; fail on regression 6. HANDLE flakiness: pin seeds/temp, average N runs, separate flaky from real drops ``` ### Stage 1 — Build the eval set (generate + curate) Two stages, never one. **Generation** gives coverage cheaply; **curation** gives trust. - **Generate** candidates from your corpus. [[llamaindex]] `DatasetGenerator.from_documents(docs).generate_dataset_from_nodes(num=50)` produces labelled QA pairs from the documents themselves [llamaindex-sop-skill/SKILL.md:121]. promptfoo and synthetic-data generators do the same for non-RAG tasks. - **Curate** the generated set into a *golden set*: a human reviews, fixes wrong labels, drops ambiguous items, and adds known hard/edge cases the generator missed. A purely generated set inherits the generator model's blind spots and tends to be too easy (Dilemma 1). - **Freeze and version**. The golden set is a committed fixture (`eval/golden_v1.jsonl`), tagged with the date and the generator model. Changing it is a versioned event, not an edit. Size: [[dspy]] documents the sweet spot — "30 examples = minimum useful, 300 = recommended" [dspy-sop-skill/SKILL.md:87]. For a held-out *gate*, 50–200 curated domain examples is the working range; descend to `[[agentsop-domain-eval-set]]` for domain-specific construction. ### Stage 2 — Split: train / dev / test The gate runs on **test only**. Keep test sealed from anything that touches the prompt: - `train` — feeds optimizers / few-shot demo selection. - `dev` — tuning and threshold-setting. - `test` — the gate. Never used to author prompts, pick demos, or tune. ([[dspy]] held-out exit criterion [dspy-sop-skill/SKILL.md:101].) ### Stage 3 — Pick the metric (cross-link `[[agentsop-metric-design]]`) Do not invent a metric here. Consume one from `[[agentsop-metric-design]]`: - RAG → the Faithfulness + Relevancy + Retriever(MRR/hit-rate) triad ([[llamaindex]] `OP-10` [llamaindex-sop-skill/SKILL.md:234]). - Exact-answer → exact-match ([[dspy]] §4.3 [dspy-sop-skill/SKILL.md:137]). - Open-ended → decomposed sub-judges, bool-in-compile/float-in-eval, length penalty ([[agentsop-metric-design]] `OP-M01`/`OP-M02`/`OP-M03`). A metric that has not been human-calibrated must not gate ([[agentsop-metric-design]] `OP-M05`). An uncalibrated gate is worse than no gate — it gives false confidence. ### Stage 4 — Set the threshold | Threshold shape | Rule | Use when | |---|---|---| | Absolute floor | fail if `score(test) < X` | "never ship below this quality bar" | | Relative no-regression | fail if `baseline − score > Δ` | "this PR must not make it worse" (the gate proper) | | Per-slice floor | fail if any slice (e.g. lexical-query subset) drops > Δ | aggregate hides a regressed minority | Set Δ above measured run-to-run noise (Stage 6), else the gate flaps. Commit the current `test` score as `baseline.json` next to the eval set; the gate compares against it. ### Stage 5 — Wire into CI - Add an `eval` CI job that runs on every PR touching prompts, model config, retriever/chunking config, or the program graph. - Job: load frozen eval set → run pipeline at PR's config → compute metric → compare to `baseline.json` → exit non-zero on threshold breach → post the before/after table as a PR comment. - On merge to main with an *intended* improvement, bump `baseline.json` in the same PR (reviewed, not silent). ### Stage 6 — Handle flaky evals LLM outputs are nondeterministic; a naive gate flaps and gets disabled. Mitigations: - **Pin** `temperature=0` and seeds where the provider supports them; disable response caching in CI ([[dspy]] AP-10: "Forgetting `cache=False` in stateless deploys" [dspy-sop-skill/SKILL.md:255]). - **Average N runs** (e.g. 3) and gate on the mean; report variance. - **Separate flaky from real**: if the same config scores differently across reruns by more than Δ, the *gate* (Δ too tight) or the *metric* (noisy judge) is the problem — fix those before trusting a single red build. A judge-based metric with high variance should be hardened in `[[agentsop-metric-design]]`, not papered over by widening Δ. --- ## 4. 操作模型 (Operations) ### OP-01 — GenerateEvalCandidates - **Trigger**: Need a held-out eval set; no labelled fixture exists yet. - **Action**: Generate QA candidates from the corpus — [[llamaindex]] `DatasetGenerator.generate_dataset_from_nodes(num=50)`, promptfoo synthetic generation, or task-specific synthesis. Tag with generator model + date. - **Output**: A raw candidate set (unvalidated), ready for curation. - **Evidence**: [[llamaindex]] Stage 2 [llamaindex-sop-skill/SKILL.md:121]; `OP-10` [llamaindex-sop-skill/SKILL.md:234]; external "eval set generation". ### OP-02 — CurateGoldenSet - **Trigger**: A generated candidate set exists; it has not been human-reviewed. - **Action**: Human reviews every item: fix wrong labels, drop ambiguous/duplicate items, inject known hard cases and past production failures. Freeze as a versioned fixture (`golden_vN.jsonl`). - **Output**: A trusted, frozen golden eval set that the gate runs against. - **Evidence**: [[dspy]] metric human-validation discipline [dspy-sop-skill/SKILL.md:212]; [[agentsop-metric-design]] `OP-M05`; Dilemma 1. ### OP-03 — SplitTrainDevTest - **Trigger**: Eval set built; about to use it for both tuning and gating. - **Action**: Partition into train/dev/test. Seal `test` from all prompt-authoring. The gate reads only `test`. (Note [[dspy]]'s reversed 20/80 train/val split for *prompt optimizers* [dspy-sop-skill/SKILL.md:96] — that is an optimizer concern; the gate still needs an untouched test slice.) - **Output**: Three disjoint splits; a sealed test set for the gate. - **Evidence**: [[dspy]] Stage 3 held-out exit criterion [dspy-sop-skill/SKILL.md:101]; AP-9 [dspy-sop-skill/SKILL.md:254]. ### OP-04 — SetRegressionThreshold - **Trigger**: Have a `test` score; need a pass/fail policy. - **Action**: Commit current `test` score as `baseline.json`. Define absolute floor X and/or relative no-regression Δ (Δ > measured noise). Optionally per-slice floors. - **Output**: A versioned threshold policy the CI job enforces. - **Evidence**: [[dspy]] Stage 3 exit "by ≥ task-relevant delta" [dspy-sop-skill/SKILL.md:101]; [[llamaindex]] gate-every-change [llamaindex-sop-skill/SKILL.md:124]. ### OP-05 — WireCIGate - **Trigger**: Eval set + metric + threshold exist; not yet enforced automatically. - **Action**: Add a CI job triggered by prompt/model/retriever/graph changes: run eval at PR config → compute metric → compare to baseline → fail on breach → comment the before/after table. - **Output**: Merges that regress quality are blocked; every change carries a number. - **Evidence**: [[llamaindex]] `OP-10` regression-test framing [llamaindex-sop-skill/SKILL.md:233-235]; external "llm regression testing CI", "promptfoo". ### OP-06 — BumpBaselineOnIntendedWin - **Trigger**: A PR *intentionally* raises quality; the gate would otherwise pin the old baseline forever. - **Action**: In the same reviewed PR, update `baseline.json` to the new `test` score. Never let CI auto-bump silently. - **Output**: Baseline ratchets upward deliberately; future regressions are caught against the new bar. - **Evidence**: [[dspy]] "keep both `program.gpt4o.json` and `program.llama8b.json`; A/B" [dspy-sop-skill/SKILL.md:191] (versioned-artifact discipline). ### OP-07 — StabilizeFlakyEval - **Trigger**: The gate flaps — same config, different verdicts across reruns. - **Action**: Pin temperature/seed, disable CI caching, average N runs and gate on the mean, report variance. If variance > Δ, fix the metric (harden judge via `[[agentsop-metric-design]]`) or widen Δ — never disable the gate. - **Output**: A stable gate whose red builds are trustworthy. - **Evidence**: [[dspy]] `cache=False` AP-10 [dspy-sop-skill/SKILL.md:255]; [[agentsop-metric-design]] judge-bias hardening; [[dspy]] Stage 2 exit "stable across two runs" [dspy-sop-skill/SKILL.md:91]. ### OP-08 — SliceTheEvalSet - **Trigger**: Aggregate score is flat but a subpopulation (lexical queries, a tenant, a topic) silently regressed. - **Action**: Tag eval items by slice; compute and gate per-slice. A win on the majority must not mask a regression on a minority slice. - **Output**: Slice-level regression detection; aggregate no longer hides harm. - **Evidence**: [[llamaindex]] per-query-type taxonomy [llamaindex-sop-skill/SKILL.md:280]; [[agentsop-metric-design]] triad (no single number). --- ## 5. 困境决策案例 (Dilemma Cases) ### Dilemma 1 — Generated eval set vs hand-curated golden set **困境**: Team needs an eval set fast. `DatasetGenerator` produces 200 QA pairs in minutes ([[llamaindex]] Stage 2 [llamaindex-sop-skill/SKILL.md:121]). Hand-curating 200 examples costs days of human time. Ship the generated set as the gate, or pay for curation? **约束**: The generator is the same model family that powers the pipeline → its questions are answerable by exactly the kind of reasoning the pipeline already does (self-preference leakage). Generated sets skew easy and miss the long-tail failures users actually hit. But zero eval set means shipping blind (anti-pattern). **决策步骤**: 1. **Generate for coverage, never gate on raw generation.** Use the generated set as a *candidate pool*, not the gate. 2. **Curate a golden subset** (`OP-02`): a human keeps the good items, fixes labels, drops the trivially-easy and the ambiguous, and *injects* known production failures and adversarial/edge cases the generator never proposes. 3. **Cross-family generation** where possible: generate with a *different* model family than the task model to reduce self-preference leakage (mirrors [[agentsop-metric-design]] `OP-M04`). 4. **Size to budget**: 50 curated > 200 raw. [[dspy]] floor is 30 useful examples [dspy-sop-skill/SKILL.md:87]; a tight, curated, edge-case-loaded 50 gates better than a bloated easy 200. 5. **Version both**: keep the raw generated pool (regeneratable) and the frozen golden set (the gate fixture). **结果**: The golden set is the gate; the generated pool is scaffolding. Teams that gate on raw generated sets ship regressions that the easy set never exercised — the gate was green while users churned. **可提取的操作**: `OP-01 GenerateEvalCandidates`, `OP-02 CurateGoldenSet`. **Lesson: generation buys coverage, curation buys trust. A gate needs trust — never gate on an uncurated generated set.** ### Dilemma 2 — Threshold too strict blocks good changes **困境**: A no-regression gate is set at Δ = 0 (any drop fails). A genuinely *good* refactor — simpler prompt, 40% cheaper model — scores 0.81 vs the 0.83 baseline: a 2-point drop within run-to-run noise. The gate blocks a change that is net-positive (equal quality, far cheaper). The team starts overriding the gate, and soon ignores it entirely.