murphy-hallucinated-output

name

murphy-hallucinated-output

description

What goes wrong when a model invents things in its own generated output — nonexistent code/APIs/functions, fabricated package names, made-up citations, and the 90%-correct answer with one invisible invented detail (wrong default, flag, or signature). Use when generating or accepting AI-written code, working with copilots, acting on AI-suggested dependencies/installs, trusting AI-written API calls or citations, or trusting LLM output by reading it instead of running it. A failure-domain lens in the murphy-jutsu pre-mortem family.

Murphy: Hallucinated Output

A failure-domain lens in the murphy-jutsu pre-mortem family — the pessimistic twin of brainstorming. Mined from ec1-dev videos.

When this lens applies

• AI-generated code is accepted, edited, or shipped — copilots, "just fix it" / "make it work" loops, autocompleted method calls.
• A human "reviews" AI output by reading it rather than compiling, importing, or running it.
• The model suggests dependencies, install commands, or imports that someone might run blind.
• AI produces citations, references, version numbers, case names, or DOIs used in a high-stakes context (legal, medical, compliance, docs).
• The work touches obscure/niche libraries, internal tooling, or rare APIs where training data is thin.
• Specificity is being read as a signal of accuracy ("it gave an exact flag, so it must be right").

Failure modes

Hallucinated code / APIs

• How it happens: The model predicts the most plausible next token, not the true one. When a method "ought to exist" by pattern, it emits a call to it — truth is not a dial in the system, and there is no "I don't know," so it auto-completes the gap with something answer-shaped.
• Warning signs: Code that reads perfectly — clean imports, error handling, idiomatic style — but references methods, fields, or signatures that don't exist; calls that fail only at compile/import time; the AI "fixing" a gap by inventing the missing helper.
• Mitigations: Gate ALL AI code behind a compile/import/test before any human trusts it — "the compiler is the one thing it cannot charm." Prefer asking for runnable code over claims. Feed the model the real API docs/source in-context instead of letting it answer from memory.

Hallucinated package install / slopsquatting

• How it happens: The model recommends installing a package name that doesn't exist. Attackers monitor commonly-hallucinated names, register them in the public registry, and stuff them with malware — so running the suggested install pulls a malicious package: a self-inflicted supply-chain attack.
• Warning signs: Install instructions for a package you can't find in the official registry; a name that's almost a real one; suggested dependencies nobody on the team has heard of.
• Mitigations: Verify every AI-suggested dependency actually exists before install — never run AI-suggested install commands blind. Check the registry entry, repo, maintenance status, age, and download counts. Use allowlists / private mirrors where possible.

Hallucinated citations / sources

• How it happens: Same next-token plausibility applied to references — titles, authors, links, version numbers generated in flawless format with zero backing reality. Confidence scales with formatting polish, not truth.
• Warning signs: Perfectly formatted citations; specific case names, DOIs, or version numbers nobody has clicked through; fluency mistaken for accuracy.
• Mitigations: Treat every citation as guilty until proven real — open each one and confirm it exists and says what's claimed. Never file or ship citation-bearing output without independent lookup (the lawyer who didn't check got caught by the judge).

Mostly-correct with one invented detail

• How it happens: 90% of the output is correct, but a single fabricated detail — a wrong default value, a flag that isn't real, an off-by-one signature — is stitched invisibly into the middle. It's hard to catch because everything around it checks out. "The lie wears the truth as camouflage."
• Warning signs: Mostly-correct output where one config value, default, or flag is subtly off; bugs that surface only at runtime in a specific branch; the more obscure the detail, the more confidently it's stated.
• Mitigations: Verify the load-bearing details — defaults, flags, signatures, magic values — not just the overall shape. Don't let a long correct-looking answer earn trust for its one rare claim. Externalize the check: run it, test it, diff it against real docs.

Thin-training-data overconfidence

• How it happens: The model hallucinates hardest exactly where real data is thinnest (niche libs, internal/obscure tech, post-cutoff topics) — and dresses those guesses in the most confident, specific detail. It sounds identically sure whether right or inventing.
• Warning signs: Crisp, specific answers about rare or proprietary things; rising confidence on increasingly obscure questions; no hedging anywhere, ever.
• Mitigations: Treat model confidence in thin-data areas as a red flag, not reassurance — "precision on something rare is a red flag, not a green light." Ground answers about niche topics in supplied source. Assign a designated doubt-bringer who asks "is that actually real?" for every AI-asserted fact.

Pre-mortem questions

1. Does every piece of AI-generated code pass a compile/import/test gate before any human trusts it, or are we trusting it by reading?
2. Which of our dependencies, APIs, or citations originated from AI suggestions, and has each been independently verified to exist?
3. Do we have a hard rule against running AI-suggested install commands without confirming the package's provenance (slopsquatting)?
4. Where is our training data thinnest (niche libs, internal/obscure tech), and are we treating confidence there as a red flag rather than reassurance?
5. For mostly-correct output, are we verifying the small load-bearing details — defaults, flags, signatures — or just the overall shape?
6. Are we feeding the model the real source/docs in-context, or letting it answer from memory (which invents more)?
7. Who is the designated doubt-bringer who stops and asks "is that actually real?" for every AI-asserted fact?

Mitigations checklist

• Gate all AI code behind a compile/import/test run before any human trusts or ships it.
• Prefer runnable artifacts over claims — ask for code reality can check, not prose assertions.
• Verify every AI-suggested dependency exists, is maintained, and matches the project before install; never run AI install commands blind.
• Open and confirm every AI-produced citation, link, version, or reference before filing or shipping.
• For mostly-correct output, independently verify the load-bearing details: defaults, flags, signatures, magic values.
• Feed the model real docs/source in-context instead of relying on its memory.
• Treat high confidence on obscure/thin-data topics as a warning, not a green light.
• Name a doubt-bringer responsible for fact-checking AI-asserted facts.

Recurring motifs

• "It sounds exactly as confident when it's right as when it's inventing... You cannot tell the truth from the fiction by how sure it sounds."
• "Precision on something rare is a red flag, not a green light."
• "The lie wears the truth as camouflage" — the 90%-right answer with one invented detail is more dangerous than 100% wrong.
• The model as an improv actor ("yes, and" — never breaks character to say "I don't know") and as an overconfident intern who has never once said "I'm not sure": it brings the confidence, you bring the doubt.
• "The compiler is the one thing it cannot charm" — externalize verification to reality (compile, run, test, ground).

Source videos

• why does my ai invent code that doesn't exist — transcripts/14_ai-invents-nonexistent-code/