evidence-first-debugging

name	evidence-first-debugging
description	General debugging methodology for complex or intermittent failures in any multi-layer system (pipelines, builds, network, DB, queues, integrations, CI). Use it when the symptom is generic/compressed (exit code, "error 500", timeout), when a theory-based fix doesn't change the behavior, or when there is more than one component between the cause and the symptom.

Evidence-First Debugging — general methodology

It applies to any system where the symptom shows up far from the cause: build pipelines, CI jobs, data replication, queues, authentication, third-party integrations.

Golden rule: raw evidence before theory

Every symptom is compressed on its way to you (an exit code summarizes an HTTP status, which summarizes a response, which summarizes which component responded). Every theory built on top of a compressed symptom inherits that loss of information.

Before any hypothesis, answer with data, not with deduction:

1. What exactly was returned? (the full payload, not the error code)
2. Who responded? (the source of the response identifies the layer — format, headers, the signature of the error message)
3. When exactly? (timestamps with enough precision to correlate)

If you can't answer all three, the next step is not to theorize — it's to instrument.

The process (7 steps)

1. Enumerate the full path

Draw every hop between whoever observes the symptom and whoever produces the behavior: client → intermediaries (proxies, queues, caches, forwards, wrappers, hooks) → target → the target's internal logic (config, env, routing). Each hop is an independent hypothesis. There may be more than one bug at once with the same external symptom — don't stop at the first finding if the evidence doesn't close the case 100%.

2. Bisect from the cheapest observation point

For every boundary between layers there is a place to observe from. Test the SAME target from two or more points of view (from the inside, from the outside, from a neighbor, skipping an intermediary). The divergence between two observation points localizes the broken hop far more strongly than any log.

3. Validate the instrument before the measurement

Fixtures, images, mocks, build caches, seed data: confirm that the test tool is what you think it is (inspect the artifact, not the recipe that produced it). A measurement taken with a poisoned instrument invalidates the entire round — and worse, it generates a false theory.

4. Preserve the crime scene

Disable automatic cleanup on diagnostic runs (don't let cleanup wipe the original state). It failed → freeze the state and do the autopsy on the live system. Deducing from destroyed state is the fastest route to the wrong theory.

5. Correlate timelines

The failure window (start/end, on the observer's side; start/end on the executor's side). The event at the edge of the window is the trigger. Without timestamps from both sides that correlation is impossible — instrument both.

6. Falsify cheaply and honestly

A theory-based fix is an experiment: define BEFORE which result would confirm it and which would kill it. Classic signs of a dead theory:

• the fix changes nothing;
• the fix gets worse in proportion to what it adds (e.g. +N s of waiting = +N s of failures → the failures cover the whole window, not the theorized event);
• the observed "recovery" is an artifact of the observer (the probe stopped, not the problem). Dead theory → go back to the golden rule: there is evidence nobody captured.

7. Only fix after a live, binary confirmation

The cause is proven when a test on the live system produces a reproducible binary result (the value is there or it isn't; the key is X or Y). Only then: implement, add a regression test that would fail without the fix, and re-verify the complete scenario from scratch — twice if the failure was intermittent.

Universal anti-patterns

• Debugging the exit code instead of the response.
• Adding wait/retry/timeout without knowing what is being waited on.
• Trusting a fixture/cache/mock without inspecting it.
• Letting the test teardown destroy the evidence of the failure.
• Treating "the failures stopped" as "the problem is over".
• Sticking with a theory because it already cost a lot (sunk-cost fallacy).