render-reftest

# CG Rendering Tests — Reftests & Golden Tests How to design, name, and review visual rendering tests in this repo. ## When to Use This Skill - Adding a new pixel-comparison test to `crates/grida/` - Reviewing a PR that adds or updates golden images - Deciding whether a test should be a reftest, golden test, or probe test - Designing a fixture that can be verified without vision input - Investigating a flaky or platform-dependent visual test - Setting up tolerance thresholds for image diff - Writing honest PR descriptions for rendering changes --- ## Terminology Use these terms precisely. Misusing them erodes trust in test results. | Term | Definition | | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Reftest** | A test that compares renderer output against an **independent reference** (oracle) whose correctness is established outside this project — e.g. a W3C-provided PNG for an SVG test case. The oracle is the source of truth; a mismatch means our renderer is wrong. | | **Independent reference / oracle** | A rendering produced by a separate, trusted implementation or defined by a specification. We do not control its content. | | **Golden test** | A test that compares renderer output against a **previously accepted snapshot** produced by our own renderer. There is no external truth — the golden file _is_ the expected output because a human reviewed and approved it. Also called a snapshot test. | | **Snapshot test** | Synonym for golden test. The snapshot is a frozen output that we assert has not changed. | | **Render regression test** | Any test whose purpose is to detect _unintended changes_ in rendering output. Golden tests are regression tests. Reftests are correctness tests. | | **Pixel diff** | Byte-level comparison of two raster images. A single differing channel value is a failure (at zero tolerance). | | **Perceptual diff** | Comparison in a perceptual color space (e.g. YIQ via the `dify` crate). Weights differences by human visual sensitivity. More forgiving than raw pixel diff but still quantifiable. | | **rendiff** | Rust crate (`rendiff` v0.2) for histogram-based pixel diffing. Computes a per-channel difference histogram; thresholds are expressed as `[(max_diff, max_count), ...]` pairs. Used in `flatten_rendiff.rs` for equivalence tests. Dep in `crates/grida/`. | | **dify** | Rust crate for perceptual image comparison in YIQ color space. Used by `grida_dev reftest` for SVG reftests. Supports `--threshold` and `--aa` (anti-aliasing detection) flags. | | **pixelmatch** | Pure-JS perceptual image comparison library. YIQ-based, AA-aware. Used by `@grida/reftest`. Zero native deps; same conceptual model as dify, slightly different threshold semantics — see parity notes below. | | **`@grida/reftest`** | General-purpose, language-agnostic TS reftest CLI + library at `packages/grida-reftest/`. Takes two directories of PNGs, diffs, scores, writes the same bucket layout and JSON report as the Rust `grida_dev reftest`. Does NOT render anything — producers upstream. | | **`grida_dev reftest`** | Rust reftest runner at `crates/grida_dev/src/reftest/`. SVG-specific: renders SVG via our own grida pipeline, then diffs against a reference PNG. Canonical for SVG. For non-SVG formats, use `@grida/reftest` with an upstream renderer. | | **refig** | Short for "Figma reftest." Fixture suites under `fixtures/local/refig/` containing `.fig` + `document.json` + `images/` + `exports/` (oracle PNGs from Figma's Images API). Consumed by a TS render step + `@grida/reftest`. See `fixtures/local/refig/README.md`. | | **refbrowser** | Short for "headless-browser reftest." HTML/CSS fixtures under `fixtures/test-html/L0/` rendered by Playwright Chromium as the oracle vs. our `grida` htmlcss renderer. Producer script: `.agents/skills/render-reftest/scripts/refbrowser_render.ts`; diff via `@grida/reftest`. | | **Tolerance / fuzz** | A configured threshold below which pixel differences are ignored. Expressed as a histogram threshold (rendiff) or a YIQ distance (dify / pixelmatch). Required when rasterization is non-deterministic across platforms. | | **Data test** | A test that asserts on the scene graph or computed values directly — no rendering needed. E.g. bounding box, resolved transform matrix, computed style. The cheapest possible assertion. | | **Probe test** | A test that asserts correctness by reading pixel values at specific coordinates in the rendered output. Requires a purpose-built fixture with a minimal color palette and documented probe points. No full-image comparison needed. | | **Probe-friendly fixture** | A fixture explicitly designed for probe testing: minimal colors, no decorative elements, shapes at known coordinates. Often accompanied by a `.probe.json` file declaring expected pixel values at specific points. | --- ## The Core Decision Rule ``` can correctness be asserted without rendering? ├─ YES → data test (assert on scene graph, computed values, bounding boxes) └─ NO → need pixels ├─ independent oracle exists? │ ├─ YES → reftest (compare against external truth) │ └─ NO │ ├─ probe-friendly fixture? (minimal palette, known coords) │ │ ├─ YES → probe test (read pixel values at specific points) │ │ └─ NO → golden test (diff against prior accepted output) │ └─ (when in doubt → golden test) ``` Prefer the cheapest method that still validates the behavior. Data tests are free (no render), probe tests are cheap (render but no diff), golden tests and reftests are expensive (full image comparison). --- ## When to Use Each ### Data tests — no render needed Use when the behavior can be verified on the **data model directly**: - Computed bounding boxes, resolved transform matrices, layout positions. - Scene graph structure (parent/child, ordering, visibility flags). - Style resolution (computed fill, stroke, opacity values). - Path math (point-on-curve, intersection, winding number). If the assertion doesn't need pixels, don't render. Data tests are instant, deterministic, and platform-independent. ### Reftests — spec-backed or standard formats Use when a **trusted external reference** exists: - **SVG rendering** — W3C SVG 1.1 Test Suite provides reference PNGs. Our `grida_dev reftest` runner compares against these. See `docs/wg/feat-svg/testing.md` and `crates/grida_dev/TESTING.md`. - **resvg test suite** — feature-focused SVG tests with author-provided reference PNGs. - **Figma files (refig)** — Figma's own Images API renders every node with an `exportSettings` preset; those PNGs are the oracle. Consumed by a TS render step + `@grida/reftest`. See the Figma section below. - **CSS properties** — when a CSS spec defines exact rendering behavior and a reference implementation provides ground truth. A reftest failure means **our renderer has a bug** (or the spec changed, or — for refig — Figma's server-side renderer changed). ### Two reftest runners, one report format Two tools implement the diff/score/report side of reftests. Pick by the producer: | Runner | Language | Render path | Diff engine | Use when | | ------------------- | -------- | -------------------------------- | ----------- | ------------------------------------------------------------------- | | `grida_dev reftest` | Rust | SVG → grida → PNG (built-in) | dify | SVG suites (W3C, resvg); renderer runs in-process | | `@grida/reftest` | TS | None — you produce PNGs upstream | pixelmatch | Figma refig, any cross-language producer, re-diff without re-render | Both write the **same bucket layout** (`S99/S95/S90/S75/err`) and the **same `report.json` schema**. A parity test at `packages/grida-reftest/__tests__/parity.test.ts` asserts the two tools grade a fixture pair to within ±0.005 similarity and the same bucket. Do not let them drift — update this table and both tools if the contract ever changes. --- ## Oracle Strategies by Input Format The oracle source depends on the format being tested. Choose the right strategy before authoring a new fixture. ### SVG — pre-baked reference vs. resvg-generated reference Three strategies, depending on what oracle data is available: **Strategy A — pre-baked reference (single oracle)** Use if the SVG comes from a test suite that ships reference PNGs alongside it (W3C SVG 1.1, Oxygen Icons). The oracle is the co-located PNG; `grida_dev reftest run` picks it up automatically via `reftest.toml`. ```sh # W3C suite — reference PNGs are in png/ next to svg/ cargo run -p grida_dev --release -- reftest run \ --suite-dir fixtures/local/W3C_SVG_11_TestSuite --bg white ``` **Strategy A+ — pre-baked reference + Chrome bake (multi-oracle)** Use for `resvg-test-suite`. The vendored `expected.png` is the suite author's read of the spec, but for ~12% of fixtures Chrome diverges from it. The harness ingests the suite's `results.csv` (a 9-renderer status matrix) and a baked Chrome PNG to classify each fixture into **consensus** / **disputed** / **UB** buckets. Effective per-fixture score is `max(vs_expected, vs_chrome)`; the headline parity number is the consensus pass-rate (excludes disputed and UB). ```sh # One-time: bake Chrome PNGs (puppeteer; deterministic per Chrome version) cargo run -p grida_dev --release -- reftest bake # Run + summarize cargo run -p grida_dev --release -- reftest run \ --suite-dir fixtures/local/resvg-test-suite --renderer htmlcss cargo run -p grida_dev --release -- reftest summary ``` For the full driver loop (audit → ground → fixture → impl → verify against the multi-oracle gate), see [`dev-render-htmlcss-svg-feature`](../dev-render-htmlcss-svg-feature/SKILL.md). **Strategy B — resvg as dynamic oracle (when no pre-baked image exists)** For arbitrary SVG files with no reference PNG, resvg is an independent, conformant SVG renderer that can serve as the oracle. Run resvg on the SVG to produce a reference PNG, commit it, then run the normal reftest against it. ```sh # Install resvg CLI cargo install resvg # or: brew install resvg # Render SVG → reference PNG at the exact viewport size resvg input.svg reference.png --width 512 --height 512 # Store alongside the SVG fixture; reference is the oracle ``` This is still a genuine **reftest** — resvg is an independent implementation, not our own renderer. A mismatch means our renderer diverges from the SVG spec (or resvg has a bug, which can be checked upstream). **Choosing between strategies:** - Pre-baked PNG present? → use it (Strategy A). - No pre-baked PNG, SVG content is standard/spec-compliant? → generate via resvg (Strategy B). - SVG content is non-standard or uses Grida-specific extensions? → golden test (see below). --- ### Figma — the refig reftest pipeline Figma content has **no pre-baked oracle** — the oracle is Figma's own server-side renderer, accessed via the Images API. The full, end-to-end refig suite has four pieces, all living under `fixtures/local/refig/` (gitignored): ``` fixtures/local/refig/<name>.<filekey>/ ├── document.fig .fig binary (manual File → Save local copy in Figma desktop) ├── document.json REST: GET /v1/files/:key?geometry=paths ├── images/ Image fills from GET /v1/files/:key/images └── exports/ Oracle PNGs: one per node with non-empty exportSettings └── <node-id>.png (colon-sanitized: 1:5216 → 1_5216.png) ``` Nodes must have **export presets configured in Figma** before archiving (select → right panel → Export +). The REST API does not expose `exportSettings` for `SECTION` nodes (figma/rest-api-spec#87) — use `FRAME`, `COMPONENT`, or `INSTANCE`. Without exports there is no oracle; a comparison against our own renderer's output is a **golden test**, not a reftest. **1. Archive + export oracle PNGs** — `figma_archive.py --export` writes `document.json`, `images/`, and `exports/` in one call. See the script header for flags, prerequisites, and limitations. ```sh python https://github.com/gridaco/grida/blob/main/.tools/figma_archive.py \ --filekey <figma-file-key> \ --archive-dir fixtures/local/refig/<name>.<filekey> \ --export ``` Requires `FIGMA_TOKEN` or `X_FIGMA_TOKEN` in the process environment — the script does NOT auto-load `.env`. Requires Python 3.10+ (uses `str | None`); on macOS default Python 3.9 use Homebrew's `python3.12`. **2. Drop the `.fig` binary in** — File → Save local copy in Figma desktop, save as `document.fig` inside the suite directory. The REST API has no `.fig` download endpoint. **3. Render with our import pipeline** — Produce per-node actual PNGs with the TS-side `@grida/io-figma` pipeline (`.fig` via fig-kiwi, or `document.json` via REST converter). The renderer consumes a node's `exportSettings` to match Figma's target size/scale per export preset; one actual PNG per oracle PNG, filenames matched by Figma node id. Writes actuals to an out-of-tree directory (e.g. `target/refig/<name>/renders/`), one PNG per node, filename matching `exports/<node-id>.png`. **4. Diff via `@grida/reftest`** — The diff/score/report step is format-agnostic: ```sh npx reftest \ --actual-dir target/refig/<name>/renders \ --expected-dir fixtures/local/refig/<name>.<filekey>/exports \ --output-dir target/reftests/<name> \ --bg white --mask alpha ``` Or the programmatic API from a TS test: ```ts import { reftest } from "@grida/reftest"; const report = await reftest({ name: "<name>", expectedDir: "fixtures/local/refig/<name>.<filekey>/exports", actualDir: "target/refig/<name>/renders",