visual-feedback-loop

name

visual-feedback-loop

description

Render → see → critique → edit → re-render: the discipline (and a deterministic referee) that lets an agent inspect its OWN rendered output — a web page, a dashboard, a Power BI / Tableau report — and iterate toward correctness/pixel-perfection against objective stopping signals instead of 'looks better'. The referee (driver.py) merges the pbir-layout-engine layout linter with agent-captured console/Lighthouse evidence into one pass/fail verdict. Use when building or refining any visual surface; the standalone canon is knowledge/visual-feedback-loop.md.

Skill: visual-feedback-loop

What this is

A render-loop referee plus the discipline that wraps it. The agent that builds a visual surface should not work blind: it renders, sees its own output, judges it against the intent and objective signals, edits, and re-renders — until the signals pass. This skill provides the deterministic "are we done yet?" half so the loop converges instead of wandering on subjective taste.

The full conceptual canon — the loop, the two ways to "see", the surface→mechanism map, the security rules — lives in ../../knowledge/visual-feedback-loop.md. This SKILL is the operating reference for the runnable piece.

The two ways an agent "sees"

Mode	How	Best for
Visual (pixels)	Drive a real browser via the chrome-devtools-mcp server → take_screenshot (the model literally sees it), list_console_messages, lighthouse_audit	Web pages, web dashboards, embedded BI — catching "looks wrong"
Structural (coordinates)	Read the layout definition's exact numbers — PBIR JSON x/y/width/height via the pbir-layout-engine linter; the DOM/accessibility tree for web	Power BI / Tableau pixel-perfection (layout is just numbers — more reliable than vision), and any surface where the definition is inspectable

Structural-first for BI. For Power BI / Fabric / Tableau the primary loop is structural (the coordinate linter), because (a) layout correctness is exact arithmetic, not a judgment, and (b) screenshotting a BI report needs it published/embedded + authenticated, which the agent often can't reach. Screenshots are the secondary check for what coordinates can't show (did the theme apply, did conditional formatting fire, does it overlap once real data loads). For web, the screenshot is first-class.

The referee — driver.py

driver.py is NOT a browser driver. It cannot navigate Chrome or run Lighthouse — that is the agent's job via chrome-devtools-mcp. It is the referee: given the evidence the agent has captured, it merges it into one verdict with an objective next_action. It earns its existence by fanning three independent evidence sources into one verdict — something the layout linter alone structurally can't do.

CLI contract

python3 plugins/ravenclaude-core/skills/visual-feedback-loop/driver.py <config.json>

<config.json> (every path is repo-relative, ..-free, inside the repo root):

{
  "surface": "web | web-dashboard | pbir | fabric | tableau | bi",
  "layout": "path/to/page.json",          // optional → delegated to pbir-layout-engine
  "parity": {                              // optional → structural diff vs. a known-good exemplar
    "candidate": "path/to/failing/visual.json",
    "reference": "path/to/confirmed-working/visual.json"
  },
  "console": "path/to/console.json",       // optional → agent-captured browser console
  "lighthouse": "path/to/lighthouse.json", // optional → agent-captured Lighthouse run
  "design_schema": {                       // optional → offline "declares the same design system?" floor
    "candidate": "path/to/candidate-design-schema.json",
    "reference": "path/to/reference-design-schema.json"
  },
  "ssim": "path/to/ssim.json"             
                             
     
     
     
     
     
     
  

The parity gate — diff against a known-good exemplar (a structural diff surfacer, not a render oracle — it is only as good as the reference you pick). A visual can be perfectly placed yet render blank because its render skeleton is missing something its working twin has. Point candidate at the suspect visual.json and reference at a confirmed-working visual.json of the same visualType. The gate is asymmetric: it fails (next_action: match-reference-exemplar) on what the candidate is MISSING relative to the exemplar — a missing query role (Values/Data/Indicator), a dropped objects key (e.g. a card that dropped labels and substituted calloutValue), or a missing per-item $id — and passes benign additions (an extra cosmetic object key, an optional role). It reports not_captured (never a false fail) for a different visualType, a non-PBIR shape, a self-reference, or a degenerate exemplar (no query role — it refuses to launder a bad reference into a pass). Highest-leverage move when a deploy renders blank with no error: replicate the nearest genuinely-working exemplar instead of guess-and-check. (PBIR visual.json today; the technique generalizes — see the canon.)

Design-schema mimicry — two fidelity mechanisms, honestly labelled. When cloning a reference's design craft (spacing scale, type scale, elevation, breakpoints, components) onto your own brand, the referee gains two independent checks:

• design_schema — the offline structural FLOOR (NOT fidelity). A per-dimension asymmetric diff of a candidate design-schema.json against a reference one: "does the candidate declare the same design system?" It fails on what the candidate is MISSING relative to the reference — a different spacing base-unit, a type ratio outside design_ratio_tolerance, fewer elevation levels, missing breakpoints, a missing component recipe — and passes benign additions (an extra breakpoint/component/shadow), exactly like the parity gate. Each divergence is localized as a {dimension, expected, actual} delta. A missing/unparseable/non-schema file → not_captured (absence is not failure). This is a stdlib "same design system?" sanity check — it does NOT and cannot compare pixels. Every value it reads self-declares capture_method (static = parsed declared CSS, no browser).
• ssim — the browser-captured fidelity GATE. Reads {"ssim_score": <float>} (the Lighthouse-evidence pattern) and passes iff ssim_score >= ssim_min. This is the only pixel-fidelity signal, and it exists only when a browser tool captured it. The score is domain-clamped: a value that is non-finite (NaN/inf) or outside [0,1] is corrupt/hostile evidence → a determinate error, never a pass; an absent field → not_captured. (The same clamp now guards lighthouse category scores — a page-injected 5.0 no longer rescales to a fake pass.)

LOUD degradation — a green verdict with ssim absent reads as "fidelity unverified". When the structural floor passes but no ssim pass verified fidelity, the referee does not emit a bare ship: next_action is capture-ssim-evidence and notes carries "visual fidelity not verified — no browser tool". Structural-clean is never mistaken for pixel-faithful.

Security invariant (SSIM). The SSIM score MUST be computed out-of-page over harness-controlled screenshot buffers (the browser/MCP layer), NEVER via page.evaluate inside the measured page — a page that can compute its own fidelity number can forge a pass. driver.py reads the captured number only and clamps it to [0,1]; it never trusts a page-controllable value as fidelity.

Stateless-loop boundary (read this LOUD). driver.py is stateless per invocation — it builds its gates fresh from one config and holds no iteration history. The determinate structural + ssim gate is the only stopping proof; one pass is NOT convergence. Non-improving-iteration patience (did this edit actually move the score? are we oscillating?) is the agent's cross-iteration job — the referee cannot and does not track it.

Agent-captured evidence shapes (the contract you fill from chrome-devtools-mcp):

• console.json — {"messages": [{"level": "error|warning|info"}, ...]} (the driver counts level == "error"; it never reads the message text).
• lighthouse.json — Lighthouse's native shape {"categories": {"accessibility": {"score": 0.96}, "performance": {"score": 0.85}, ...}} (scores are 0–1; the driver surfaces them as 0–100 and compares to the threshold, and now clamps each score to [0,1] — a non-finite / out-of-domain value is skipped, never a fake pass).
• ssim.json — {"ssim_score": <float 0..1>} (a browser/harness-computed structural similarity index vs. the reference render; the driver reads the number only, clamps it to [0,1], and never echoes page content — see the SSIM security invariant above).

Exit codes & verdict

Exit	Meaning
0	passed: true (clean) or passed: null (nothing determinate to judge / needs more evidence / manual review). Absence of a browser tool is NOT a failure.
1	passed: false — a determinate gate failed
2	I/O, parse, oversize (>5 MiB), or path-rejection (.. / outside repo) — the purity-contract failure

The JSON envelope: {schema_version, driver_version, surface, passed, gates[], next_action, notes}. passed is a pure function of the determinate gates (pass/fail/error); not_captured and degraded gates are excluded — so "evidence not captured yet" and "tooling absent" are first-class states, never silent failures. next_action is the loop's instruction: ship / capture-runtime-evidence / capture-ssim-evidence / fix-layout / match-reference-exemplar / match-design-schema / improve-visual-fidelity / fix-console-errors / improve-accessibility / manual-visual-review.

How the layout gate maps the linter's exit codes

driver.py calls pbir-layout-engine/lint.py as a subprocess (--format json), never an import — it treats the linter as a CLI with a versioned envelope, so the linter's internals are not a dependency, and it asserts the linter's schema_version matches what it was built against (loud on drift). The mapping:

linter exit	driver gate status	meaning
0	pass	layout clean
1	fail	a layout check fired
2	error (→ overall fail)	the layout JSON itself is broken
3	degraded (excluded from passed)	the PBIR visualType enum reference is absent/unparseable — e.g. ravenclaude-core installed without power-platform. Could-not-verify, not a failure.

The loop (how the agent uses this)

1. Build / edit the surface.
2. See it. Web: via chrome-devtools-mcp → navigate_page, take_screenshot (look at it), save list_console_messages → console.json and lighthouse_audit → lighthouse.json. BI: read the PBIR page JSON.
3. Referee. Run driver.py <config.json> pointing at the evidence + layout.
4. Read passed / next_action. false → do the next_action and loop. null → capture the missing evidence (or do the named manual review) and loop. true + next_action: ship → done.
5. The screenshot is for your eyes (the model's visual judgment); the referee is the objective floor. Use both — vision catches "ugly", the referee catches "wrong", and the referee is what tells you when to stop.

Security invariants (load-bearing — see the knowledge file for the full rules)

• Path safety. Every input path is resolved through the same rule as the layout linter (reject .., reject outside-repo). Reimplemented in driver.py (not imported, to stay decoupled from the linter's internals); Gate 100 asserts the two guards reject the same traversal input so they can't drift.
• Size ceiling. A 5 MiB cap is enforced before json.load on every file — a malicious page can write an unbounded console.json.
• No-echo. The verdict carries only driver-derived primitives — booleans, counts, numeric scores/thresholds, and fixed-vocabulary strings. It never echoes raw console text, Lighthouse titles, or page content: a hostile page can write fake "instructions" to the console, and the model reads this verdict back as trusted context. The driver reads numbers out of evidence, never prose.
• MCP adoption gate. chrome-devtools-mcp drives a live browser (stateful, side-effecting); it is recommended-not-bundled and its adoption is security-reviewer-gated. Do not point a credentialed/networked render loop at attacker-influenced URLs — render untrusted dashboards against synthetic/fixture data, or in an isolated profile with no credentials. Launch with --no-usage-statistics. Screenshots/evidence write to .ravenclaude/runs/<session>/visual-evidence/ (already git-ignored) and are never committed (a dashboard can render real PII/secrets).

Proven by Gate 100

scripts/audit-gates.sh Gate 100 + hooks/tests/test-gate100-visual-feedback-loop.sh + the fixtures under tests/fixtures/visual-feedback-loop/ are the bidirectional floor: good fixtures pass, bad fixtures fail, a .. config is rejected (and the linter rejects the same shape — path-guard parity), and an always-pass mutant lets a known-bad through (teeth).

Output Contract

When a reviewer critiques a render-loop integration in a PR, the response ends with the cross-plugin Structured Output JSON block per ../structured-output/SKILL.md.