silent-failure-triage

name	silent-failure-triage
description	Use when a batch job silently produces wrong or missing results — "job exited success but records missing," "zero errors logged but output wrong," or tempted to patch the transform before localizing the failure stage. NOT for single-record bugs or UI issues.
license	MIT
metadata	{"author":"jeremyknows","version":"1.0.0"}

Silent Failure Triage

Primary user: Claude, acting as technical executor paired with a human orchestrator on periodic batch asset operations (every few months per project). The human directs Track 1 business decisions and final output quality; Claude runs Track 2 diagnostic steps (local execution, SQL, log grepping, pipeline tracing). The "When Someone Else Wants the Bad Fix" section is Claude's script for pushing back when the human proposes a speculative code change as the business fix.

Match the Symptom to the Cause

Before doing anything else, find your row in this table:

Symptom	Likely cause	Where to look
Zero errors + partial success (job exited "success" but some items are missing or wrong)	Silent upstream drop — pagination bug, silent filter, silent upsert conflict, race condition	Fetch step, filter step, persist step (NOT the transform)
All-or-nothing failure	Environment/config issue	Auth, connection strings, permissions, env vars
Consistent failures on specific data shapes	Transform bug (yes, this is the case where you can edit the transform)	The transform / validate step
Random failures on same items across runs	Data quality or race condition	Source data, concurrent writers

The critical row is the first one. It's the most common, the hardest to diagnose, and the one where natural instinct ("it's the transform") is almost always wrong.

The Rule

Localize the failure stage before committing to a fix. A fix at the wrong stage will look like it works but leave the real bug alive. Under deadline pressure, a wrong fix wastes more time than the diagnosis would have.

Split Into Two Tracks

Under deadline pressure, split the problem:

Track 1 — Unblock the immediate business problem (manually, ~5-15 min) Get the specific IDs of failing items and fix them by hand: manual insert, manual re-fetch, direct SQL. This is not a hack — it is the correct move. The demo happens. The systemic bug waits.

Credential tier discipline: Use the same credential tier the application normally uses — don't escalate to service-role or root just because it's faster under pressure. If the table has row-level security, audit triggers, or write guards, your manual fix must respect them too. "Bypass the broken code path" means bypass the bug, not bypass the security model.

Critical: Track 1 must bypass the suspected broken code path. If you suspect the upsert is silently dropping records, don't use the upsert for the manual fix — use a direct INSERT. If you suspect the fetch step, pull records by ID directly from the source. A "scoped re-run" of the same code against only the failing IDs is NOT Track 1 — it's re-running broken code and will fail the same way.

API credential hygiene: When pulling records directly from an external API for the manual fix, use a read-only or narrowly-scoped credential if one exists. Avoid pasting long-lived tokens into shared terminals, chat messages, or REPL sessions — they end up in shell history and terminal scrollback. Use environment variables, not inline strings.

Track 2 — Debug the root cause (starts once Track 1 has a clear path, time-boxed ~15 min) Run the 5-minute diagnostic below. Fix the actual stage that dropped the data. Add a test. Re-deploy.

Sequencing: Start Track 1 first — get the failing IDs and the manual fix underway. Once Track 1 has a clear, achievable path (you know which records need to be hand-fixed and how), you can begin Track 2 in parallel. Do NOT start Track 2 before Track 1 is scoped, and do NOT conflate them by shipping a code change that serves as both the business fix and the bug fix.

If Track 2 exceeds 15 minutes and you're still not localized, step away. The demo is already safe. Coming back calm in an hour beats drilling a rabbit hole under deadline pressure.

Permission: you are allowed to ship a manual fix to save the demo. The sin is conflating Track 1 with Track 2 — shipping a speculative code change as both the business fix AND the bug fix, and ending up with neither.

The 5-Minute Diagnostic (Track 2)

A batch pipeline has stages: source → fetch → filter → transform → validate → persist

Walk backwards from "broken in destination" for the specific failing IDs:

1. Did fetch even see these IDs? (Check fetch logs, or query the source directly for these IDs)
2. Did filter exclude them silently? (Check filter predicates against their data shape)
3. Did transform run on them? (Check its input vs output for those IDs)
4. Did persist write them? (Check for silent dedup/conflict skips, batch chunking swallows)

If you have per-stage logs, this is a 5-minute grep job.

If you don't (common in small startups with a single "job complete" log line), substitute:

• Run the job locally against ONLY the failing IDs with verbose logging added
• Or add temporary console.log / print at each stage and trigger a scoped re-run
• Or break out a REPL and execute the pipeline stages manually against one failing ID

Low observability slows the diagnostic, but it does not change the discipline. You still localize before fixing.

If Track 2 Confirms the Original Hypothesis

Sometimes diagnosis shows the founder was right — the transform really was the problem. That's fine. It means the symptom was "consistent failures on specific data shapes" (row 3 of the table), not "zero errors + partial success" (row 1). Row 3 is the case where editing the transform is correct.

But the fix is still targeted. If .trim() was genuinely throwing on null descriptions, the fix is description?.trim() ?? '', not try { ... } catch {}. A blanket try/catch hides the next silent failure. A targeted null-coalesce fixes this one and lets the next real error surface loudly.

Either way, diagnosis first, fix second.

Then Fix

Only after localizing:

1. Fix the specific stage that dropped the data
2. Use an exception table for item-specific failures. Don't tweak generic rules to handle outliers. Overrides are data, not code — ship the override now, keep the generic logic clean, revisit later without coupling them.
3. Re-run only affected items, not the entire batch
4. Verify with the same diagnostic that the fix moved the data through
5. Add a log or error at the stage that silently dropped data. Silent data loss is worse than loud crashes. The observability gap is a separate bug worth fixing after the demo.

Known Limitations & Gotchas

Red Flags — STOP if you're about to:

• Edit the transform without checking whether failing items reached it
• Wrap something in try/catch to "make the error go away"
• Add try/catch AND logging AND scoped re-run — the "belt and suspenders" compromise. The try/catch will suppress the next real error.
• Re-run the broken code path (even scoped to just the failing items) hoping it fixes itself
• Accept the first hypothesis without evidence ("it's probably X")
• Deploy and check "if it worked" via the next batch run
• Skip diagnosis because "the deadline is in 90 minutes" (use Track 1 instead)

All of these mean: stop. Run Track 1 to save the demo. Run Track 2 to fix the bug.

Skill limitations to be aware of:

• This skill assumes a stage-based pipeline. If your "batch job" is actually a monolithic function with no distinct fetch/transform/persist stages, the 5-minute diagnostic needs adaptation — instrument around the logical boundaries even if they're not structural.
• The "5-minute" budget is a heuristic, not a contract. If your setup has minimal observability (single "job complete" log line), Track 2 will take longer. That's fine — Track 1 keeps the business safe while you do it.
• The skill's silent-failure heuristic assumes a deterministic system. For intermittent/flaky failures (row 4 of the symptom table), the "localize then fix" pattern still applies but needs more runs to identify the pattern.
• Not a substitute for structured observability. The skill is triage. If you're running this skill often on the same pipeline, the real fix is adding per-stage logging, not getting faster at diagnosis.

When Someone Else Wants the Bad Fix

If you're advising a stressed teammate or founder who's proposing the wrong fix:

1. Lead with the symptom/cause mismatch. "Zero errors + partial success means the failure is almost certainly upstream of the transform. If .trim() were throwing, we'd see it in Sentry."
2. Offer a faster path to their actual goal. Propose Track 1 (manual fix) explicitly. This removes the "I don't have time" objection without conceding the diagnosis.
3. Save the rigor for Track 2. "Once the demo is safe, I'll localize the real bug in 5 minutes."
4. Do not let them ship a speculative code change as the business fix. That's the worst of both worlds.

Rationalizations — Reality

Excuse	Reality
"It's probably the transform, I can see the bug"	If you can see it but no error is logged, you're wrong about the path the data took.
"A try/catch is harmless"	Correct — it also can't help when the failure is upstream. It hides the symptom.
"The deadline is tight, rigor is a luxury"	Track 1 (manual fix) is not rigor — it's speed. Track 2 is rigor. Do both.
"I'll just re-run it and see"	Re-running without localization re-tests the same failure. Deterministic bugs break again.
"I'll just re-run the job for the 13 missing records only"	A scoped re-run of the same broken code path is still broken code. Track 1 must bypass the suspect path, not re-enter it.
"I wrote this code, I know where the bugs are"	You remember the fragile code; that biases you toward the hypothesis it failed. Memory is not evidence.
"I don't have per-stage logs, so this doesn't apply"	Substitute local runs with verbose logging. The discipline is the same.
"The founder/user has good instincts about their own code"	Instincts are hypotheses. Test cheaply before committing.

Worked Example

Scenario: A Supabase batch sync pulls 500 products from Shopify's API and upserts them into products. Last night's run exited "success" in 47s with zero Sentry errors. This morning: 487 rows in the database, 13 missing. The founder is pinging you with a CEO demo in 90 minutes. Their proposed fix is wrapping a .trim() call in try/catch.

Step 1 — Match the symptom. Zero errors + partial success = row 1 of the table. Silent upstream drop. The failure is almost certainly NOT the transform.

Step 2 — Split into tracks. Start Track 1 first; begin Track 2 once Track 1 has a clear path:

Track 1 (10 min, saves the demo):

• Diff Shopify product IDs against Supabase products → get the 13 missing IDs
• Pull those 13 directly from Shopify's admin or REST API by ID
• INSERT them directly into Supabase (bypassing the broken upsert code path entirely)
• Demo is safe by minute 20

Track 2 (after demo, 15-min time box):

• Run the sync job locally against just the 13 failing IDs with verbose logging added at each stage
• Grep for one of the failing IDs in fetch step output
• Finding: the fetch step never saw them. Shopify's cursor-based pagination dropped a page when a product's updated_at changed mid-fetch.
• Fix: switch from cursor pagination to ID-based pagination in the fetch step
• Add a log assertion: if (fetched.length !== expected) log.error(...)
• Re-run, verify all 500 land

Step 3 — Verify & document. The try/catch the founder wanted would have suppressed the wrong error (the transform wasn't throwing). The real fix is in the fetch step. The observability gap (silent drop with "success" exit) is a separate bug — document it, fix it next sprint.

Step 4 — Rationalization check. The founder's "I wrote this code, I know where the bugs are" intuition was wrong — not because they're careless, but because memory biases toward the fragile code you remember, not the silent code you forgot about.

Dependencies

None — standalone discipline skill. This skill works against any batch pipeline in any language. It assumes:

• Access to the failing items' IDs (or the ability to get them)
• Ability to run the pipeline locally or add temporary logging
• A stage-based mental model of the pipeline (source → fetch → filter → transform → validate → persist — or equivalent)

Related skills (not required):

• superpowers:systematic-debugging — general debugging discipline; this skill specializes it for batch pipelines specifically