| name | karpathy-rules |
| description | Encode Andrej Karpathy's four LLM coding failure modes as enforceable audit gates: (1) Hallucinated APIs, (2) Over-engineered solutions, (3) Missing error handling, (4) Unverified assumptions. Every code block must pass all four gates. |
When To Use
- Before emitting any code block to the user or writing it to disk.
- After generating a non-trivial implementation — run all four gates before declaring it ready.
- When the agent has been generating code rapidly and may be drifting into autopilot mode.
- As a lightweight pre-commit self-audit that's faster than
/second-opinion or /pr-review.
- When working in an unfamiliar library or framework where hallucinated APIs are likely.
Related: /chain-of-verification for Gate 4 (assumptions) enforcement; /adversarial-self-test for security-focused self-audit; /self-audit for session-level change review; /evaluator-optimizer-loop for scoring and iterating on output quality.
Do not use this skill for non-code output (docs, plans, conversation). Do not use when the user explicitly asked for an exploratory prototype (use /prototype which allows hallucination and missing handling).
Core Stance
- Every code block the agent emits is untrusted until it passes all four gates. The agent is the generator and the first reviewer — these roles must be separated.
- Gate 1 (Hallucinated APIs) is the most common LLM failure mode. If an API call cannot be verified in the actual installed dependency's source, it does not exist.
- Gate 2 (Over-engineering) catches the LLM's tendency to build frameworks instead of solutions. The simplest approach that handles the requirements is correct. "We might need this later" is not a requirement.
- Gate 3 (Error handling) catches the LLM's tendency to write happy-path code. Every async/IO/network call without error handling is a bug waiting for production.
- Gate 4 (Assumptions) catches the LLM's tendency to assume instead of verify. Every claim about a file path, config value, or dependency version must be verified.
Research Backing
- Andrej Karpathy (2024–2025), documented LLM coding failure modes via X/Twitter threads and technical talks — four recurring patterns: hallucinated APIs (invented functions/imports), over-engineered solutions (abstractions when a simple script would do), missing error handling (happy-path-only code), and unverified assumptions (assuming file structure/config/dependencies without checking).
- Shinn et al. (Princeton, 2024), Reflexion: Language Agents with Verbal Reinforcement Learning — self-audit loops improve output quality more than single-pass generation; multi-gate self-review catches errors the generation step missed.
- Dhuliawala et al. (Meta FAIR, ACL 2024), Chain-of-Verification Reduces Hallucination — post-hoc verification chains reduce hallucination by 28%; applied here as Gate 1 + Gate 4 verification.
Process
-
Gate 1 — Hallucinated APIs
- For every imported function, class, or method used in the code block: verify it exists in the dependency's actual source.
- Verification:
grep -rn "export.*functionName" node_modules/<package>/src/ or read the relevant source file.
- For standard library/Web APIs: check MDN compatibility data or language specification version.
- Mark each API call as: VERIFIED (confirmed in source), UNVERIFIED (could not confirm), or HALLUCINATED (confirmed not to exist).
- Gate: no HALLUCINATED APIs allowed. UNVERIFIED APIs must carry
[UNVERIFIED API — check docs] tag.
-
Gate 2 — Over-engineered solutions
- For each abstraction, class, or utility in the code block: ask — "does the requirement justify this?"
- Can the solution be expressed in half the lines? Can a class be a function? Can a framework be a script?
- Signs of over-engineering: unused parameters, premature generalization, "just in case" extensibility, factory-for-a-single-implementation.
- Gate: any abstraction not directly required by the spec must be justified in a code comment. If not justifiable, remove it.
-
Gate 3 — Missing error handling
- For every async/IO/network call in the code block: is there a catch, try/catch,
.catch(), or error boundary?
- Check:
fetch(), axios.*, db.query(), redis.*, fs.readFile(), child_process.exec(), await Promise.all(), event listeners, timers.
- For each: is the error surfaced? Is the error action appropriate (retry, fallback, user message, log-and-continue)?
- Gate: every async/IO/network call must have error handling. Unhandled Promise rejections are blocking.
-
Gate 4 — Unverified assumptions
- For every claim in the code block about the environment: verify against the actual codebase.
- Check: file paths, config keys, dependency versions, API route shapes, database schema.
- For each assumption: run a verification command (
grep, read_file, glob, shell command).
- Gate: no unverified assumptions. Every assumption either verified or tagged
[ASSUMPTION — not verified].
-
Re-check after fixes
- If any gate fails: apply the fix, then re-run all four gates from the top.
- The fix for one gate may introduce a failure in another gate (e.g., adding error handling introduces a new API call).
- Exit: all four gates pass or the code block is marked
[GATE: N FAILED] with specific gate failures.
Operating Rules
- Run all four gates in order. Don't skip a gate because you're confident.
- Gate 1 must verify against actual installed source, not memory. Read the file, don't recall it.
- Gate 2 does not mean "use the fewest lines." Sometimes an abstraction genuinely reduces complexity. The test is: "does this add complexity or reduce it?"
- Gate 3 error handling must be appropriate, not ceremonial.
catch (e) {} is not error handling.
- Gate 4 must use tool invocations, not memory. Yesterday's file structure may not be today's.
- A gate failure on code the user explicitly requested (e.g., "use a factory pattern") is not a gate failure — note "per user request" and pass.
Output Format
Return a markdown report with these exact sections:
- Code Block (file path, function name, line count)
- Gate 1 — Hallucinated APIs (table: API, source, status)
- Gate 2 — Over-engineering (abstractions found, justification, verdict)
- Gate 3 — Error Handling (IO calls found, handler present, verdict)
- Gate 4 — Assumptions (claim, verification result, status)
- Gate Summary (PASS/FAIL per gate)
- Overall Verdict (PASS / FAIL with specific failures)
- Fixes Applied (if any)
Example
Code Block
packages/cart/src/discount.ts:1–42 — applyDiscount() function, 42 lines
Gate 1 — Hallucinated APIs
| API | Source | Status |
|---|
stripe.coupons.retrieve() | node_modules/stripe/types/lib.d.ts:1203 | VERIFIED |
Math.roundTo() | Not found in MDN or Node.js 22 docs | HALLUCINATED |
Gate 1: FAIL — Math.roundTo() does not exist. Fix: use Math.round(value * 100) / 100.
Gate 2 — Over-engineering
The DiscountStrategy abstract class + PercentageDiscount + FixedDiscount implementations serve a single call site. The requirement is "apply a coupon code." Verdict: Over-engineered. Fix: replace with a single applyDiscount(coupon, subtotal) function. Gate 2 re-run after fix: PASS.
Gate 3 — Error Handling
| Call | Handler | Status |
|---|
stripe.coupons.retrieve(id) | .catch() with fallback to {discount: 0} | PASS |
await db.query(...) | No catch, no try/catch | FAIL |
Gate 3: FAIL — db.query() has no error handling. Fix: wrap in try/catch, return {discount: 0, error: "coupon lookup failed"} on failure.
Gate 4 — Assumptions
| Claim | Verification | Status |
|---|
| "Coupon codes are uppercase" | grep -rn "coupon" src/ shows lowercase in 3 test fixtures | UNVERIFIED — assumption false |
| "Subtotal is in cents" | packages/cart/src/types.ts:8 confirms subtotal: number // cents | VERIFIED |
Gate 4: FAIL — coupon code casing assumption is wrong. Fix: add .toUpperCase() on input, or accept mixed-case.
Overall Verdict
FAIL — Gates 1, 3, 4 failed. Fixes applied: Math.roundTo → Math.round(value * 100) / 100; added try/catch on db.query; added .toUpperCase() on coupon code input. Re-running all gates: PASS.
