agentsop-per-model-artifacts

# Per-Model Prompt Artifacts — SOP > *"Prompts are effectively the weights of an LLM application."* > — DSPy core philosophy [arxiv.org/abs/2310.03714] > > *"Treat the compiled program as a (program × LM) pair. Changing the LM invalidates the artifact — recompile."* > — dspy-sop SKILL, Dilemma Case B --- ## 1. 何时激活 (When to activate) Activate this skill when **any** of the following appears in the user's intent, codebase, or workflow: | Trigger | Signal | |---|---| | Adopting compiled prompts | `compiled.save("v1.json")`, `dspy.load_program()`, `BootstrapFewShot`, `MIPROv2`, `GEPA`, LangChain Hub `hub.push/pull`, prompt files checked into `prompts/` or `artifacts/` | | Multi-LM production | The same program runs against ≥2 of: `gpt-4o-*`, `gpt-4o-mini-*`, `gpt-4.1-*`, `claude-3-5-sonnet-*`, `claude-3-7-sonnet-*`, `claude-3-opus-*`, `Llama-3-*`, `Llama-3.1-*`, `DeepSeek-V3`, `gemini-2.5-pro` | | Provider deprecation | OpenAI/Anthropic deprecation notice mentioning a pinned snapshot; alias rollover (`gpt-4o` → new dated snapshot); silent model behavior drift reports | | Framework deprecation | LlamaIndex `ServiceContext` → `Settings`; LangChain `LLMChain` → LCEL; DSPy major version bump; aider edit-format default change | | Symptoms | `prompts/system_prompt.txt` (no model in path), generic alias pins (`model="gpt-4o"`), missing `parent_artifact` lineage, hand-edited compiled JSON, no held-out test set re-runs | | Cross-skill bridges | DSPy compile produced output → ship via this SOP. LlamaIndex index baked with embed model → tag artifact per this SOP. Aider edit-format pin → per-model config artifact per Recipe 6 (R2) | **Do NOT activate** when: - Raw, one-off prompt edits with no compile step and no production deploy. - A genuinely model-agnostic system prompt that has been **swap-tested** across ≥3 LMs with <2-point dev metric drift. - Prompts that must remain verbatim human-authored for compliance — versioning still matters, but the optimizer/recompile loop does not apply. - The whole pipeline lives behind a vendor's managed prompt (e.g. OpenAI's Prompt Library) where the vendor owns the artifact. --- ## 2. 核心心智模型 (Core mental model) ### The artifact is a triple, not a string ``` ┌────────── ARTIFACT ──────────┐ │ │ program × LM (snapshot) × dataset (hash) → compiled.json + metadata (code) (provider+date) (canonical hash) │ │ └─ change any axis → new artifact ─┘ ``` Three load-bearing claims: 1. **A compiled prompt is not portable.** The DSPy doctrine (sibling skill, Case B): swap the LM → recompile. Reusing GPT-4o-compiled `program.json` on Llama-3-8B typically loses 15–30 points (R1 §5). The asymmetry is real and documented. 2. **The snapshot, not the alias, is the identity.** `gpt-4o` is a moving alias; `gpt-4o-2024-08-06` is an immutable identifier. Pinning to the alias means your behavior changes silently when OpenAI rolls the alias forward. Anthropic does not even roll aliases — Sonnet 3.5 (`claude-3-5-sonnet-20240620`) and Sonnet 3.5 v2 (`claude-3-5-sonnet-20241022`) are different models with different behavior; the org that pinned the wrong one in 2024-10 ate a regression in 2025-10 when the older one was retired (R1 §4). 3. **The dataset is part of the identity.** Compiled prompts overfit their training distribution. Two artifacts compiled from the "same dataset" that turn out to differ by 50 examples produce silently different programs. A `sha256` over canonicalized dataset bytes, embedded in the artifact path, makes this impossible to confuse (OP-9). ### The PyTorch checkpoint analogy | PyTorch checkpoint | Per-model prompt artifact | |---|---| | `.pt` weights file | `program.json` (compiled prompt) | | Architecture (forward pass) | DSPy Signature + Module graph (source-of-truth Python) | | Dataset version | `dataset_sha256_8` in path | | Hyperparameters | optimizer config (`MIPROv2 auto="light"`, seed, demos) | | Framework version | `dspy_version`, `python_version` in metadata | | Eval score on val | `dev_score` in REGISTRY | | Final test score | `test_score` in REGISTRY | | Model registry (MLflow) | `artifacts/REGISTRY.jsonl` or MLflow (Recipe 5) | | Promotion to prod | git tag `prompt/<program>/<snapshot>/v<n>` | The same engineering discipline applies. Anyone who would not check a `.pt` into prod without a checkpoint registry should not check a compiled prompt into prod without an artifact registry. ### Why "alone is not enough" A bare `compiled.save("v1.json")` produces a JSON file that, viewed in isolation, looks model-agnostic. The instructions and demos are text — they could be for any model. But they were *selected by an optimizer running calls against a specific LM*. The model's distributional response to the bootstrapping queries shaped which demos got kept. The artifact is implicitly LM-conditioned without saying so. The SOP exists to make this conditioning explicit and auditable. --- ## 3. SOP 工作流 (SOP workflow) A five-stage lifecycle. Each stage has an exit criterion. ### Stage 1 — Initial compile 1. Source-of-truth Python module (`src/<program>/program.py`) defines the DSPy module graph. 2. Pin the **dated snapshot** of the LM (not the alias): `dspy.LM("openai/gpt-4o-2024-08-06")`. 3. Pin the dataset: canonicalize → `sha256` → use first 8 hex chars (`a8f1c2e9`) as the dataset tag. 4. Run the optimizer (start with `MIPROv2(auto="light")` per dspy-sop). Record `cost_usd`, `wall_seconds`. 5. Evaluate on a **held-out test set** (not the optimizer's val split). **Exit:** Held-out test score recorded. No artifact saved yet. ### Stage 2 — Version (write artifact + registry) 6. Compute the path: `artifacts/<program>/<provider>/<model-snapshot>/<dataset>-<sha8>/v<n>.json` where `n` is the next integer after existing files in that directory. 7. `compiled.save(path)` for the JSON; optionally also `compiled.save(path_dir, save_program=True)` for whole-program reproducibility. 8. Append one line to `artifacts/REGISTRY.jsonl` with the schema in R2 Recipe 1. 9. Commit both the artifact AND the REGISTRY line in one PR (gated by Stage 3). **Exit:** Artifact + REGISTRY entry on disk, both staged for PR. ### Stage 3 — Register (regression gate) 10. CI loads the new artifact, identifies its `parent_artifact` (previous artifact for same `program × model × dataset`), re-runs held-out test set, compares. 11. Fail PR if `new_test_score - parent_test_score < -GATE` (default `GATE = 0.02`, 2 points on 0-1 scale). 12. Block merge unless gate passes or a human attaches the `artifact-override` label (with justification in PR body). **Exit:** PR merged to main, REGISTRY canonical. ### Stage 4 — Swap-test (on model swap or new snapshot) 13. New LM snapshot announced (e.g. `gpt-4o-2024-11-20` replacing `2024-08-06`). Or considering family swap (GPT-4o → Sonnet 3.5). 14. Load existing artifact unchanged. Configure DSPy to the new LM. Run held-out test set. 15. Compute Δ-score vs original. - `|Δ| ≤ 2 points`: tag artifact `transferable[old→new]` in REGISTRY. May ship without recompile. - `|Δ| > 2 points`: tag `recompile_required[old→new]`. Proceed to Stage 5. 16. Emit `swap-report-<old>-to-<new>.md` artifact regardless of outcome (audit trail). **Exit:** Swap report committed, recompile decision recorded. ### Stage 5 — Recompile or accept 17. If `recompile_required`: re-run the same optimizer config against the new LM. Produces a new artifact at the new model's path. Increment `v<n>`. 18. If `transferable`: keep using the old artifact; mark in REGISTRY which model snapshots it's certified-transferable to. 19. Keep BOTH old and new artifacts checked in. Production deploy script reads the git tag, not HEAD. 20. Loop back to Stage 2 for the new artifact. **Exit:** Production deploys an artifact whose REGISTRY entry shows passing test score against the model it will actually run on. ### Quarterly maintenance loop - Run deprecation scan (R2 Recipe 4). Open issues for artifacts within 90 days of model retirement. - Run canary re-eval: re-run held-out test set against the pinned snapshot to detect silent provider-side drift. Threshold same as regression gate. --- ## 4. 操作模型 (Trigger / Action / Output / Evidence) ### 4.1 Storage operations | Trigger | Action | Output | Evidence | |---|---|---|---| | Compile finishes successfully | Path `<program>/<provider>/<snapshot>/<dataset-sha8>/v<n>.json`; append REGISTRY line with `parent_artifact` pointer | Grep-able path + lineage | R2 Recipe 1; OP-1, OP-9 | | About to pin a model | Use **dated snapshot** never an alias: `gpt-4o-2024-08-06` not `gpt-4o`; `claude-3-5-sonnet-20241022` not `claude-3-5-sonnet` | Metadata header inside artifact + REGISTRY | R1 §4 (alias rollover); OP-2 | | Multiple compiles share a program code | All keyed by `program` field in REGISTRY (string), `commit` field links to source-of-truth Python at compile time | Reproducible: `git checkout <commit>` restores the program code that produced it | OP-3 | | Promoting to production | `git tag prompt/<program>/<snapshot>/v<n>`. Deploy reads the tag, not main HEAD | Atomic rollback via `git checkout <tag>` | MLflow Stage analogy (R1 §7); OP-10 | | Need whole-program portability | `compiled.save(dir, save_program=True)` for pickled program + state; ship both | Self-contained; survives source-tree reorgs (until Python/DSPy version mismatch) | DSPy save_program docs [dspy.ai/tutorials/saving/] | ### 4.2 Swap-test operations | Trigger | Action | Output | Evidence | |---|---|---|---| | Provider releases new snapshot | OP-4 swap-test old artifact against new model on held-out test set | Δ-score; recompile_required boolean | R2 Recipe 2 | | Family swap candidate (GPT-4o → Sonnet) | Swap-test BOTH directions (artifact-from-A run on B; artifact-from-B run on A); record asymmetry | Two Δ-scores; choose recompile direction | R1 §5 (down-transfer 15-30pt drop) | | Aider edit-format default change | Re-run aider edit benchmark on fixed task subset for affected model | edit-format-bench delta; pin if changed | aider SKILL line 302 (udiff 20→61%) | | LlamaIndex `Settings.embed_model` change candidate | **Refuse to swap without rebuilding index.** Re-embed against full corpus; tag new index artifact | New `indices/<name>/openai__<embed-model>__<dim>/<corpus-sha8>/` | LlamaIndex SKILL anti-pattern A2; R2 Recipe 7 | ### 4.3 Gate operations | Trigger | Action | Output | Evidence | |---|---|---|---| | PR touches `artifacts/**/v*.json` | CI regression-gate: re-run held-out test against new artifact and parent artifact | CI status `artifact-regression: pass\|fail\|overridden` | R2 Recipe 3; OP-5 | | PR tries to add file under `artifacts/` without REGISTRY append | Pre-commit hook fails | Local hook error; can't push | OP-8 | | PR pins a snapshot ≤90 days from deprecation | Pre-commit hook warns; CI fails unless override label | Surfaces deprecation debt at compile time | OP-7; R2 Recipe 4 | | Hand edit to compiled JSON | Pre-commit hook fails: artifact files are read-only except via compile.py | Forces recompile path | OP-8; dspy-sop anti-pattern #3 | ### 4.4 Lineage operations | Trigger | Action | Output | Evidence | |---|---|---|---| | Looking up "which artifact is in prod for this program?" | `git ls-remote --tags origin 'prompt/<program>/*'` | Tag list with snapshot + version | OP-10 | | Reproducing a past compile | REGISTRY entry has `commit`, `dspy_version`, `optimizer_config`, `dataset_sha256_8`, seed; replay against original LM snapshot | Bit-identical or near-identical artifact | R2 Recipe 1; OP-3 | | Auditing "what changed when score regressed" | Diff REGISTRY entries; if `dataset_sha256_8` changed → dataset drift; if `optimizer_config` changed → config drift; if `dspy_version` changed → framework drift; if `commit` changed without code changes → environment drift | Root-cause class identified | OP-3 | --- ## 5. 困境决策案例 (Dilemma cases) ### Case A — "GPT-4o deprecation: artifact loses 8 points on Sonnet 3.7" **困境 (Dilemma):** Production runs a DSPy-compiled program against `gpt-4o-2024-05-13` at 82% test score. OpenAI announces deprecation in 60 days. The team wants to move to Sonnet 3.7 (`claude-3-7-sonnet-20250219`) for unrelated cost/latency reasons. Swap-test (OP-4) shows the unchanged artifact scores 74% on Sonnet 3.7 — Δ = −8 points. Recompile, accept, or hybrid? **约束 (Constraints):** - Recompile cost on Sonnet 3.7 (200-example trainset, MIPROv2 light): ~$4 and 20 minutes. - Customer SLA is "≥80% on the public eval set". - Sonnet 3.7 has extended thinking budget — different output distribution from GPT-4o. - Old artifact's instructions reference "be terse" — Sonnet 3.7 with thinking ignores terseness hints. - Team has 60 days, not 60 minutes. **决策步骤 (Decision steps):** 1. **Do not accept the −8 swap as-is.** The artifact is below SLA on the target model. Per dspy-sop Case B doctrine, "Changing the LM invalidates the artifact — recompile" is the default. 2. **Recompile against Sonnet 3.7 with the same dataset.** Run `MIPROv2(auto="light")` first (R2 cost guardrail). If light gives ≥80%, stop. If <80%, escalate to `medium` only after Stage 1 review of the program/metric. 3. **Tune the optimizer for Sonnet 3.7 specifics.** Set `max_bootstrapped_demos` lower (Sonnet 3.7's thinking emits richer reasoning per-demo, so fewer richer demos > more thin demos — R1 §1). 4. **Keep BOTH artifacts** in `artifacts/<program>/`. The GPT-4o-2024-05-13 artifact stays available for canary-rollback during the cutover window. 5. **Git tag the new artifact** as `prompt/<program>/claude-3-7-sonnet-20250219/v1` only after the regression gate passes against the Sonnet 3.7 held-out test set. 6. **Emit `swap-report-gpt-4o-2024-05-13-to-claude-3-7-sonnet-20250219.md`** showing: transfer Δ −8, recompiled Δ +3 vs original. Audit trail. **结果 (Outcome):** Recompiled Sonnet 3.7 artifact lands at 83% on the same held-out set. Cost: $4 + reviewer time. The +1 point over the GPT-4o original is bonus; the value was avoiding the −8 cliff. **可提取的操作 (Extractable operation):** **A negative swap-test Δ exceeding the regression gate is a recompile signal, not a "ship anyway" signal. The artifact path makes both old and new available simultaneously — there is no migration risk to keeping both.** --- ### Case B — "Silent 4o-patch: same snapshot string, different behavior" **困境:** Quarterly canary re-runs the held-out test set against the production artifact (`gpt-4o-2024-08-06`, no model change). Score dropped from 78% to 73% over three months. No code changed. No artifact changed. No registry entry changed. What happened? **约束:** - The pinned snapshot string did not change. - OpenAI's policy is "snapshots are stable" but in practice server-side mitigations (safety, hallucination patches, throughput) ship without bumping the snapshot ID. - Customer noticed before the team did. SLA at risk. - Reverting the model is not possible — the canary IS the latest behavior on the same snapshot. **决策步骤:** 1. **Verify the canary against a stored sample-output hash.** Per OP-2, REGISTRY entries log a sample-output hash from compile time. Re-run against the same 20 sample inputs; compare current outputs to stored hashes. Divergence → confirms behavior shifted on the same snapshot. 2. **Run the regression gate locally** with current canary outputs as the "new artifact" and compile-time outputs as "parent". This isolates whether the metric also dropped or just the outputs changed (semantically-equivalent rewrites would shift hashes but not score). 3. **Recompile against the same snapshot.** Even though the snapshot string is unchanged, recompiling lets the optimizer re-bootstrap demos against the new server-side behavior. Cost is the same $2–4 as initial light compile. 4. **Tag the new artifact as `v<n+1>`** under the same model directory: `artifacts/<program>/openai/gpt-4o-2024-08-06/<dataset-sha>/v<n+1>.json`. The `compiled_at` field in REGISTRY makes the temporal lineage clear even though the snapshot string is unchanged. 5. **Add the canary cadence to weekly** for 30 days post-incident; revert to quarterly when stable. 6. **File issue with OpenAI** (if support contract): provide REGISTRY entries showing same snapshot string, same code commit, same dataset hash, drift > regression gate. **结果:** Recompile recovers score to 79% (+1 over baseline, because the optimizer found demos that work with the new server behavior). Lesson: snapshot pins protect against MOST drift but not all. Canary catches what pins miss.