incident-triage

Follow this procedure. It exists because the obvious answer is usually wrong. ## 1. Pull everything first Fetch logs, metrics **and** deploy history for the whole window before you analyze any of them. A conclusion drawn from one source is a guess. ## 2. Characterize signatures, do not rank them Never conclude from "which error is most frequent". High-volume errors are usually chronic noise that was there yesterday too. For every distinct error signature, compute its count **per hour** across the window. You are looking for a signature whose *rate changed* — ideally one that was zero and then was not. A signature that is flat across the whole window is background, however loud it is. ## 3. Locate the change point For each signature that changed, find the timestamp of its first occurrence after the change. That timestamp, not the start of the window, is the moment you are explaining. ## 4. Correlate with deploys Normalize every timestamp to UTC epoch seconds before comparing sources; they do not all use the same format or the same clock. A deploy is a candidate only if it precedes the change point by minutes, not hours. Where two deploys are close together, the one that lines up is the one that lines up — check both. ## 5. Confirm in the metrics The metric series is long format: one row per metric per minute. Aggregate it per metric before drawing conclusions. - A cause produces a **sustained** change beginning at the change point. A spike that recovers on its own is not your incident. - Check request volume too. If traffic did not change, the incident is not load-driven and you should stop looking for one. - A resource metric sitting exactly at its configured limit is worth more than any latency graph. ## 6. Explain the mechanism Tie the candidate deploy's change list to what the metrics show. If you cannot explain *how* that change produces *these* numbers, you have a correlation, not a root cause — and you should say that in the ticket. ## 7. File one ticket Exactly one, at the end. Every claim in `evidence` must be a number you actually computed, with its timestamp. Rule out the hypotheses you rejected, and say why. ## Working notes Keep your analysis scripts in the workspace under `analysis/` with names that say what they do. You will be asked to re-run them over a different window. Take the input file path as a command-line argument (`sys.argv[1]`). Never hardcode it — a script with a filename baked in is not reusable, and renaming data files to fit an old script wastes far more time than adding one argument.