murphy-retrieval-trust

name

murphy-retrieval-trust

description

How RAG, grounding, and retrieval pipelines fail — the wrong chunk retrieved on keyword-not-meaning matches, the right document misread on a negation or table, the caveat truncated at a chunk boundary, a stale index serving obsolete facts, top-k bloat burying the signal, and citations that make a WRONG answer look MORE credible. Use when designing or reviewing RAG, vector search, embeddings, reranking, citations, document Q&A, knowledge bases, second-brain corpora, or any LLM grounding over a document store. A failure-domain lens in the murphy-jutsu pre-mortem family.

Murphy: Retrieval Trust

A failure-domain lens in the murphy-jutsu pre-mortem family — the pessimistic twin of brainstorming. Mined from ec1-dev videos.

When this lens applies

• A system answers questions by retrieving chunks from a corpus and feeding them to an LLM (classic RAG).
• Answers come with citations, footnotes, or source labels that humans treat as proof.
• Retrieval is embedding/similarity-driven, possibly with no reranker or relevance check.
• The corpus contains negations, tables, hedged findings, or caveats that span boundaries.
• Source documents change over time (prices, policies, configs, schedules) and the index may lag.
• An LLM is pointed at an unfiltered or stale personal/team corpus (notes, wikis, "second brain") with no trust or recency signals.

Failure modes

Wrong chunk retrieved (keyword match, not meaning)

• How it happens: Embedding similarity returns a doc that shares vocabulary with the query but is topically off. Generation faithfully summarizes a real-but-wrong source and has no signal that it was irrelevant — retrieval failed silently, the model did its job.
• Warning signs: Confident, well-cited answers that are tangential; cited sources share words but not topic; "the AI is stupid" complaints that vanish when you paste the correct passage by hand; eval failures cluster on keyword-overlapping-but-semantically-different queries.
• Mitigations: Log and inspect retrieved chunks for every answer, not just the final text; add a reranker or relevance/grounding check; evaluate retrieval precision separately from generation, using known-answer gold chunks; before blaming the prompt, confirm "did we even hand it the right chunk?"

Right document, wrong read (negation / table / caveat)

• How it happens: Retrieval is correct but the model misreads the source — it drops a "no" ("found no significant effect" → "there is a significant effect"), misaligns a table cell, or ignores a hedge, while still citing accurately.
• Warning signs: The cited source, read by a human, contradicts the model's claim; errors concentrate on negated statements, tabular data, and qualified findings; the model paraphrases instead of quoting.
• Mitigations: Apply the quote test — require the exact supporting sentence; if the quote contradicts the summary, the read was wrong; give extra scrutiny to negations, tables, and caveats; use structured extraction for tabular sources; test chunking and reading against a fixture set of negations/tables/caveats.

Truncated chunk (the caveat got cut off)

• How it happens: Naive fixed-size chunking splits the text before the qualifier ("...however..."). The model answers from half the sentence, confidently, because the caveat is literally absent from its context window.
• Warning signs: Answers omit known caveats/exceptions; the missing qualifier sits just past a chunk boundary or on the next page; the model cannot produce a complete supporting quote.
• Mitigations: Use semantic/sentence-aware chunking with overlap; don't split tables, functions, or clauses mid-thought; verify boundaries don't sever qualifiers; the quote test catches it (no complete quote available); review chunking on caveat-heavy documents.

Missing chunk (answer in corpus, never retrieved)

• How it happens: The correct fact exists but the chunk holding it is never returned — poor chunking, awkwardly-formatted content that embeds badly (tables/footnotes/code), or top-k too small to reach it. The model fills the gap with plausible fiction.
• Warning signs: "It got this wrong but I know the doc covers it"; correct fact lives in a table/footnote/edge section; quality degrades on questions whose evidence is awkwardly formatted.
• Mitigations: Measure retrieval recall — for a sample of known answers, confirm the gold chunk is actually retrievable at current chunk size, overlap, and top-k; chunk with intent (keep related things together, add overlap); tag chunks with provenance so gaps are diagnosable.

Stale index (grounded in something no longer true)

• How it happens: Source documents change but the vector store isn't re-indexed, so retrieval serves obsolete chunks. The model grounds confidently in last quarter's numbers — "a confident time capsule."
• Warning signs: Correct-looking answers reflecting outdated numbers/policies; mismatches between live docs and what the system cites; freshness bugs correlating with recent doc edits.
• Mitigations: Define a re-index trigger (event- or schedule-driven on document change); track index build timestamp vs source last-modified and surface staleness; add freshness cases to the eval set; treat "the source must actually be the source" as a hard requirement.

Top-k bloat (more is not safer)

• How it happens: Retrieving many chunks "just in case" floods the context window, raises token cost, and buries the one relevant chunk among noise — deliberately recreating lost-in-the-middle.
• Warning signs: Rising token bills; answers getting vaguer as you add context; one good chunk diluted by many irrelevant ones; large top-k defaults left untuned.
• Mitigations: Tune top-k down and rerank; measure whether each added chunk helps or hurts on your eval; prefer precision over volume; don't bury signal under 50 chunks.

Citations make wrong answers more credible

• How it happens: Across every mode above, a real citation lets a wrong answer pass the smell test, so it survives review that would have caught a bare hallucination. Garbage retrieval produces garbage answers with citations — worse than garbage alone.
• Warning signs: Reviewers approve answers because "it's cited" without opening the source; trust calibrated to citation presence rather than citation content.
• Mitigations: Never treat a citation as proof — verify the cited text actually supports the claim; make the quote test and chunk logging part of the pipeline/eval, not an afterthought.

Unfiltered corpus: indexed blind spots (knowledge hygiene)

• How it happens: Pointing an LLM at an unfiltered/stale note pile imports every model failure and aims it at your own memory. With no trust/recency signals, it weighs a 2 a.m. throwaway equally with a week-refined idea and quotes both with machine confidence. It supplies context, not judgment, so it indexes your blind spots and hands them back sorted and confident.
• Warning signs: AI confidently answering with an opinion you abandoned years ago; stale/superseded versions surfacing over current ones; uniform confidence regardless of source quality.
• Mitigations: Encode trust/recency/status signals (draft vs settled, superseded markers, dates) so retrieval can weight them; curate or filter the corpus instead of indexing everything; don't outsource judgment to the index — the AI gives context, you supply the verdict; verify "current" answers aren't pulled from abandoned notes.

Pre-mortem questions

1. When an answer is wrong, will it be wrong with a citation — and what in our process actually opens the source to confirm the citation supports the claim?
2. Are we measuring retrieval quality (does the chunk answer the question?) separately from generation, or only inspecting the final answer?
3. For our known-answer set, is the gold chunk actually retrievable at our current chunk size, overlap, and top-k? (recall check)
4. Does the pipeline force the model to quote the exact supporting sentence, so we can tell synthesis from grounding?
5. How does our chunking handle negations, tables, and caveats — could a "however" land just past a boundary and vanish?
6. What triggers a re-index, and how stale can the vector store be relative to the live source documents?
7. Is top-k tuned, or are we dumping 50 chunks and burying the signal (and paying for it)?
8. When we point an LLM at this corpus, what tells it which entries are trustworthy, current, or superseded — or will it weight a 2 a.m. throwaway equally with a settled idea?

Mitigations checklist

• Log and inspect retrieved chunks for every answer, not just the final text.
• Evaluate retrieval precision and recall separately from generation, using known-answer gold chunks.
• Add a reranker / relevance check; confirm the chunk answers the question before trusting the answer.
• Force the model to quote the exact supporting sentence (quote test) so grounding is checkable.
• Test chunking against a fixture set of negations, tables, and caveats; use sentence-aware chunking with overlap.
• Never split tables, functions, or clauses mid-thought; tag chunks with provenance.
• Define a re-index trigger and track index staleness (build time vs source last-modified); add freshness eval cases.
• Tune top-k down and rerank; measure whether added context helps or hurts — don't bury signal under 50 chunks.
• Never treat a citation as proof; verify cited text supports the claim in review.
• Encode trust/recency/superseded signals on corpus entries; curate before indexing; let the AI give context, you give the verdict.

Recurring motifs

• "RAG doesn't make wrong answers less confident. It makes them more credible." A wrong answer with a real citation passes the smell test.
• "RAG does not kill the hallucination, it relocates it... Now it lives in your retrieval pipeline. Congratulations. You own the bug now." — and "most 'the AI is stupid' bugs are retrieval bugs in a costume."
• "Ask it to quote, not summarize — quote. If it can't produce a real quote, it doesn't know. It's synthesizing."
• "More is not safer" — top-k bloat recreates lost-in-the-middle on purpose.
• "Context is not judgment." A second brain "isn't a wiser you, it's past you with a search index" — the model "indexes your blind spots and hands them back sorted, searchable, and wearing the confidence of a machine."

Source videos

• my rag model cited a source that says the opposite — transcripts/03_rag-cited-opposite-source/
• rag made my ai hallucinate with citations — transcripts/13_rag-hallucinate-with-citations/
• my Obsidian second brain has 4000 notes and none of them are useful — transcripts/07_obsidian-4000-useless-notes/