site-screenshot-false-positives

name

site-screenshot-false-positives

description

Diagnose and fix false positives in the site-screenshots diff that owidbot reports on owid-grapher PRs. Use when a PR reports "N pages changed" or a failed screenshot run and the change looks unrelated to the branch, when asked whether screenshot diffs are real, or when adding a normalisation to config.yaml. Holds the diagnostic method, the catalogue of classes already fixed, and the ones still open.

Fixing site-screenshots false positives

A false positive here is a screenshot diff, or a failed run, caused by the harness or by the difference between production and staging rather than by the branch. They are expensive out of proportion to their size: a reviewer opens a 94-pixel diff, finds nothing, and trusts the next one less.

Most of what follows is method. The catalogue is the part that moves.

Keeping this current

Update this file in the same PR as the fix. A class resolved without a row here is a class the next session re-derives — that has already happened twice, and it is what this file exists to stop. Land the row with the change and the two cannot drift.

What to write down, in rough order of how much it saves later:

• A resolved class gets a row in the catalogue: the symptom, the number you measured, and where the fix went. The number is the load-bearing part, because that is what identifies the class the next time it appears.
• A theory you disproved goes under Disproved, with the measurement that killed it. These are the expensive ones — the cost of a wrong theory is paid again in full by whoever reasons their way to it a second time, and two of the entries there had already been argued twice before they were written down.
• A class you decided not to fix stays under Open, with the reason. "Invisible and rare" is a legitimate reason and worth recording as a decision rather than leaving as an absence.
• A method that generalises goes in Method. Not every trick — only what changed how a question got answered.