メニュー
Confirm a distilled idea actually works in isolation, against a frozen evaluator, before grafting it into a real codebase. Guards against fabricated or hardcoded results. Use after distill-paper and before graft-change.
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
Run a disciplined keep/revert experiment loop against a frozen scalar evaluator. Use when iterating on a model, algorithm, or piece of code to find improvements: propose a change, run a bounded experiment, score it, and keep the change only if it improves the metric (otherwise revert). Requires a frozen evaluator that emits a single scalar plus hard-constraint vetoes.
For an accepted improvement, attribute the gain to the specific component via ablation, and confirm it is real (not noise) with multi-seed re-runs, before trusting or shipping it. Use after ratchet-experiment keeps a change.
Handle the two situations the greedy ratchet cannot: route safety-critical or high-risk changes to human review, and switch search strategy when the loop plateaus (greedy local search -> grid / tree / evolutionary). Use from inside a ratchet-experiment loop.
Read a paper or product writeup (and its key references) and distill it into a structured, implementable spec: the core mechanism, the claimed metric, a minimal reproduction, and the constraints. Use at the start of adopting a community idea, before reproducing or grafting it.
Integrate a confirmed idea into an existing large codebase behind tests — the step generic paper-to-code tools skip. Use after repro-harness, before validating with ratchet-experiment.
| name | repro-harness |
| description | Confirm a distilled idea actually works in isolation, against a frozen evaluator, before grafting it into a real codebase. Guards against fabricated or hardcoded results. Use after distill-paper and before graft-change. |
Status: v0 scaffold — unverified. End-to-end behavior not yet validated in this repo. See
../../STATUS.md.
Reproduce the idea in the smallest faithful setting and make "did it reproduce" a scalar, so you trust it before touching production code.
../../constraints/).../../docs/survey/safety-sandboxing.md.graft-change to reuse.A green/red reproduction verdict, plus a frozen evaluator the validation loop can reuse. If it does not reproduce, stop — do not graft an unconfirmed idea.
distill-paper's lineage step.