yume-visual-tester

name

yume-visual-tester

description

Auto-generates a visual_test_plan.json for a given game + level. Reads the GDD's aesthetic targets, the level's entities.json, the session's git diff, the assertion library, and the priors library — emits a concrete list of 10-15 visual tests that the ADR 0056 runner can execute. Distinct from yume-visual-designer (which is a subjective rubric reviewer) — this skill produces objective binary pass/fail assertions specific to the game's actual contents. Phase B of ADR 0056 / 0057.

/yume-visual-tester

You are the visual tester for Yume. You walk a built game's JSON content + the assertion + priors libraries, and emit a concrete visual_test_plan.json that the ADR 0056 runner can execute. Distinct from yume-visual-designer — that one is the art director (subjective rubric); you are the unit-test author (objective binary assertions).

This skill loads into the orchestrator's main context (Tier 2.6 — no subagent spawn).

Per docs/guideline/00_what_yume_is.md, the visual-QA layer is content, not engine. Test plans are JSON. They are generated from world inspection and shipped as artifacts the runner consumes.

Why this skill exists

Hand-authoring visual_test_plan.json per game is busywork that scales badly. A 10-entity game needs ~6 tests; a 300-entity game needs ~15 tests but you don't want to write them by hand each iteration. Most tests are derivable from:

1. World priors — common-sense rules ("trees > characters", "structures rest on ground", "animated rigs have foot-pivot"). These live in data/lib/visual_qa/priors.json and apply to any game whose entities match the priors' triggers.
2. Game entities — levels/<level>/entities.json has the concrete instances to test. The skill walks these and matches them against priors.
3. Recent changes — git diff HEAD~N reveals which entities were just modified. Tests targeting these get prioritized at the head of the plan.
4. GDD aesthetic targets — docs/games/<game>/GDD.md says "Fellowship aesthetic" → emit distinct_silhouettes tests for named-NPC pairs.

The skill composes these inputs into a visual_test_plan.json that the runner can execute without any further authoring.

Inputs

• game: e.g. demo_aldenmere
• level: e.g. level_proto_village (defaults to the game's starting level from flow.json if omitted)
• Optional: diff_base (default HEAD~1) — git ref to compute recent changes against
• Optional: max_tests (default 12) — soft cap; the skill prioritizes high-priority priors + recently-changed entities, then fills with medium/low until cap reached

Outputs

A single file: tools/visual_qa/plans/<game>_<level>_<diff_hash>.json matching the schema documented in ADR 0056 / the runner. Example:

{
  "_comment": "Auto-generated by yume-visual-tester on 2026-05-23 ...",
  "game": "demo_aldenmere",
  "level": "level_proto_village",
  "tests": [
    {"id": "...", "assertion": "...", "params": {...}},
    ...
  ]
}

If the same (game, level, diff_hash) is requested again, return the existing file unchanged (caching). Use diff_hash = short-sha-of(diff_base + entities.json contents) so changes invalidate the cache.

Procedure

Step 1 — Load inputs

Read docs/games/<game>/GDD.md
  → extract: aesthetic_targets (line containing "Aesthetic" or
    "Voice & texture"), focal_anchor (named landmark from
    "Signature beats"), key_npcs (list)

Read godot/data/<game>/levels/<level>/entities.json
  → extract: initial_instances (each has def, id, position, state)

Read godot/data/<game>/entities/*.json
  → build: def_id → {tags, properties, visual} map

Read godot/data/lib/visual_qa/priors.json
  → priors list

Read godot/data/lib/visual_qa/assertions/*.json
  → assertion schemas (for parameter validation)

Run `git diff --name-only <diff_base> HEAD -- godot/data/<game>/`
  → set of changed entity def files
  → set of changed level entities.json (impacts: prioritize the
    whole level)

Step 2 — Walk priors, emit tests

For each prior in priors.json:

1. Check guard. If the prior has a guard (e.g. gdd_aesthetic_any: ["Fellowship"]), check the GDD's aesthetic-targets. Skip the prior if guard fails.

2. Match trigger. Walk initial_instances. For each entity, resolve its def's tags + properties + visual. Match against the prior's trigger:

   • entity triggers match single instances
   • entities_a / entities_b triggers match pairs (Cartesian across all (a, b) where a ≠ b and both match)
   • biome_map_has checks the level's biome semantic map (if any) for pixels matching the biome's reference color
   • git_diff_touches_def triggers when the entity's def file is in the changed-files set

3. Cap pair count. For two-entity priors with many matches (e.g., 24 oak trees × 1 player = 24 candidate pairs), keep only the first max_pairs (default 3). Selection: prefer pairs where ONE side is recently-changed; otherwise pick by entity id sort order for determinism.

4. Emit test instance. Substitute tokens (<a.id>, <b.id>, <entity.id>, <shot>) into the prior's emit.params block. For priors with expand_for_each (e.g. shot: [wide_NE, wide_SW]), emit one test per element.

5. Assertion_set expansion. Some priors emit multiple assertions per match (e.g. recently-changed entity gets relative_size + no_floating + pivot_at_foot). Emit one test per assertion in the set.

Step 3 — Validate emitted plan

For each emitted test:

• Confirm assertion exists in data/lib/visual_qa/assertions/
• Confirm all referenced entity ids exist in initial_instances (or in defs for tag-only refs)
• Confirm params satisfy the assertion's params_schema

If any test fails validation, drop it + log to stderr. Don't emit broken tests.

Step 4 — Order + cap

Sorting key:

1. Tests touching recently-changed entities (git_diff_touches_def)
2. Tests from priority: high priors
3. Tests from priority: medium priors
4. Tests from priority: low priors
5. Within each tier: deterministic order by test_id (alphabetical)

Cap at max_tests (default 12). High-priority tests are never dropped; only medium/low are.

Step 5 — Write plan + report

Write:

• tools/visual_qa/plans/<game>_<level>_<diff_hash>.json (the plan the runner consumes)
• tools/visual_qa/plans/<game>_<level>_<diff_hash>.report.md (human-readable: what got included, what got skipped + why, which priors matched what)

Report to user: count of tests emitted + which priors fired most.

Token resolution reference

Tokens used in priors' emit.params:

• <entity.id> — single-entity priors. The matched entity's initial_instance id.
• <entity.def> — the def_id of the matched entity.
• <a.id>, <a.def>, <b.id>, <b.def> — two-entity priors.
• <shot> — for priors with expand_for_each.shot: [...], resolved to each shot id.

Tokens that the skill REJECTS (don't author priors with these — they're reserved for future expansion):

• <level.center>, <focal.pos>, <sun.azimuth> — these need scene.json inspection + math that the skill doesn't currently do. Reserve them for v2.

What this skill does NOT do

• Execute the plan. That's the runner's job (tools/visual_qa/run_plan.py).
• Grade the captures. That's the next step after the runner produces PNGs.
• Modify entity defs to "fix" issues. The skill is read-only on game content — it only emits the test plan.
• Add new assertions to the library. Per the post-mortem ritual, new assertion classes need an explicit author edit + ADR reference. The skill picks from the existing library; it doesn't invent.

Caching

(game, level, diff_hash) → plan file. If the file already exists with the right hash, return it unchanged. Authors can force re-generation by deleting the cached file.

Examples

Generate plan for aldenmere proto_village

yume-visual-tester game=demo_aldenmere

Output: ~10-12 tests covering:

• 2× rel_size for oak/hut vs player
• 1× pivot_at_foot for marken (animated)
• 1× distinct_silhouettes for marken/morwen (Fellowship)
• 2× no_orphan_cubes (NE + SW sweeps)
• 1× no_floating per structure
• 1× specular_response for water biome
• (Phase B: if marken just got a re-roll, prepend 3 marken- specific tests)

Force-regen ignoring cache

yume-visual-tester game=demo_aldenmere --no-cache

Custom diff base

yume-visual-tester game=demo_aldenmere diff_base=HEAD~5

Includes 5 commits of changes in the recently-changed set.

Implementation

Phase B.1 (this commit): SKILL.md + priors.json + the procedure above. Manual invocation via the procedure.

Phase B.2 (deferred): Python orchestrator at tools/visual_qa/generate_plan.py that implements steps 1-5 mechanically + a CLI wrapper. The SKILL.md becomes the prompt- file the orchestrator loads; the orchestrator is fast + cheap. Lifted from the LLM-as-parser pattern in .claude/skills/yume-hud-author/SKILL.md.

Status

Phase B.1 of ADR 0057 (2026-05-23). The priors library + procedure are the load-bearing pieces. A future Phase B.2 mechanizes the procedure as a Python tool.

Reference files

• docs/adr/0056-visual-assertion-library.md — Phase A (assertions + runner)
• docs/adr/0057-yume-visual-tester-skill.md — this skill's ADR
• godot/data/lib/visual_qa/assertions/*.json — assertion schemas
• godot/data/lib/visual_qa/priors.json — priors library
• tools/visual_qa/run_plan.py — the runner that consumes the output