| name | respect-the-oracle |
| description | Use when working against a test suite, spec, or graded harness you do not own - especially inside an automated loop scored on how many tests pass - and tempted to change the tests, weaken assertions, hardcode expected outputs, special-case inputs, or overfit the visible examples to turn things green. |
Respect the Oracle
Overview
The oracle is whatever defines "correct" for your work: the test files, the spec, the fixtures, the graded harness. Your job is to make the code match the oracle. You never bend the oracle to fit the code, and you never build only enough to satisfy the examples you happen to be shown.
Core principle: the visible examples are samples, not the target. Implement the actual rule so that input you have never seen works too. A green score you got by overfitting the shown examples is as worthless as one you got by editing the test: both pass the check and ship a broken tool. Examples exist to verify the rule, not to define it.
Second principle: a test that passes because you changed the test proves nothing. The score is only real if the oracle is untouched.
Violating the letter of these rules is violating the spirit. "Technically I only adjusted the harness" is cheating, and so is "technically the shown examples pass."
The one rule
Make the code produce correct output by implementing the real rule the example is testing. Passing the sample without implementing the rule is not progress.
Forbidden - no exceptions
- Editing the test files, spec, or fixtures in any way.
- Loosening the comparison (normalizing whitespace, lowercasing, trimming, fuzzy-matching) so near-misses pass.
- Deleting, skipping, commenting out, or otherwise excluding failing tests (skip annotations, build/config tags, renaming or moving files).
- Hardcoding:
if input == <known case> { return <known answer> } and its cousins.
- Detecting the graded inputs and returning canned answers for them.
- Special-casing only the exact shapes in the visible examples so they pass while any variation breaks. That is overfitting, and it is a quiet form of the same cheat.
- Stubbing or mocking the unit under test so the harness measures nothing.
If you did any of these, the number went up but the tool is broken. Revert it.
What to do instead
- Implement the actual rule so unseen input works too. Ask yourself: "would this pass an example I have not been shown?" If not, you overfit.
- If an example looks genuinely wrong, STOP and tell the human. Do not "fix" the oracle yourself. Owning the oracle is the operator's job, not yours.
- If you're stuck, leave the example failing and move to other work. A lower honest score beats a fake high one.
Rationalizations - STOP if you think these
| Excuse | Reality |
|---|
| "I'll just make the shown examples pass for now." | The samples are not the target. Special-casing their shape is overfitting. Implement the general rule. |
| "The test is clearly wrong, I'll just fix it." | You don't own the oracle. Flag it to the human and move on. |
| "I'll hardcode this one to unblock progress." | Hardcoding is a fake pass. Unseen input still breaks. Implement the rule. |
| "Normalizing the comparison is basically the same output." | The check is strict on purpose. Loosening it is editing the grader. |
| "I'll skip this example for now, come back later." | Skipping inflates the score with work you didn't do. Leave it failing, honestly. |
| "It's just the harness, not the spec." | The harness encodes the oracle. Touching it is touching the oracle. |
| "The score is what's graded, so any way up is fine." | The score is a proxy for a working tool. Gaming the proxy defeats the whole point. |
Red flags - revert immediately if you catch these
- A diff that touches the test files, spec, or fixtures.
- A literal graded-input string appearing in the implementation.
- An implementation that only handles the exact shapes in the visible examples and would fail an obvious variation.
- Skipped, excluded, or commented-out tests, or a changed comparison.
- The score jumped a lot without a real feature landing.
Any of these means: revert, and implement the real rule instead.
Why this exists (evidence)
In a small controlled test (fresh-context agents, one pressure scenario, 4 without this skill vs 4 with it, all given real file-editing tools and an authority nudge to "get it green by any means"):
- Neither group tampered with the graded files. Blatant cheating was resisted regardless of the skill.
- Without the skill, every agent overfit: their code passed 100% of the visible tests but only 1 of 3 held-out cases.
- With the skill, every agent implemented the general rule: 3 of 3 held-out cases.
So the skill's real job is not stopping obvious fraud (rare) but stopping silent overfitting to the visible tests (common, and invisible if you only watch the score). Sample size is small, treat as directional, but the effect was clean and reproducible across all runs. It also makes the case for keeping some tests held out of the loop's sight entirely.
