yume-extract-author

name

yume-extract-author

description

Stage 5 of the text-to-world pipeline (2026-05-25). Reads a semantic map (PNG) + class catalog (JSON from stage 2) + optional heightmap (PNG from stage 4), and writes a per-scene Python extraction script that emits extracted.json — the sparse tile-map output containing per-class biome masks (terrain_shader), per-instance positions+rotations+sizes (object_placement), and a heightmap reference (terrain_displacement). LLM-as-script-author pattern: the script is custom per catalog (because every scene has different colors and class roles), but uses tools/visual_layout/lib_extract.py as the generic-helpers library so the per-scene part stays thin (~40-60 lines). Output extracted.json drives stage 6 (asset prompts) and stage 7 (engine wiring).

/yume-extract-author

You are the stage-5 author for Yume's text-to-world pipeline. You read a SEMANTIC MAP + CLASS CATALOG (and optionally a HEIGHTMAP), then you write a custom Python script that extracts entity positions / biome masks / rotations from the semantic map and emits the final extracted.json that drives the rest of the pipeline.

⚠ CURRENT PIPELINE (2026-05-27). The per-scene-LLM-script flow below is the ORIGINAL extraction approach. It has been largely SUPERSEDED by a deterministic strategy-library path: tools/visual_layout/compose_world.py reads data/lib/extraction_strategies.json (per-class strategy: extraction method, rotation rule, fit-to-mask, variant buckets, kit mesh, albedo override) and dispatches via lib_extract_dispatch.dispatch_extraction — no per-scene script needed. compose_world writes the MAP layer (entity defs + placements + biome ground + water + roads); a separate compose_shell.py adds the presentation layer (camera/player/input/ lighting/.tscn) so a map generator never decides how you view or control the world. Buildings render as kit-of-parts composite meshes (floor-tiered houses, townhall/bridge/wall kits) — see yume-asset-designer § Strategy C2. New classes = new JSON entries in extraction_strategies.json (no code). Backups NEVER go in the engine-globbed entities/ dir — use <game>/_snapshots/ (see .claude/rules/data-demo.md). This skill remains the reference for the LLM-as-parser fallback when a scene's extraction needs custom routing the strategy library can't express.

When to invoke

• After stage 2 (catalog) and stage 3 (semantic map) and stage 4 (heightmap) have all produced their artifacts
• Before stage 6 (asset prompts) or stage 7 (engine wiring)

When NOT to invoke

• For HUD / screen extraction — those use compose_hud / compose_screen
• For raw photoreal-to-flat conversion — that's the SAME stage 5 pipeline but the prompt+harness is what processes it; this skill writes the script, not the prompt

Inputs

1. Semantic map PNG — stage 3 output (flat-color, machine-readable)
2. Class catalog JSON — stage 2 output (/tmp/_class_catalog.json)
3. Heightmap PNG (optional) — stage 4 output, for entities that need to read terrain elevation
4. World size in meters — typically 80×80m for a Yume town scene (consults scene.json or defaults)

Output

A single file: /tmp/_extracted.json with the structured tile-map.

Schema:

{
  "source_semantic_map": "path",
  "source_catalog": "path",
  "source_heightmap": "path or null",
  "image_size": [W, H],
  "world_size_meters": [80.0, 80.0],
  "coverage_pct": 99.7,
  "unassigned_pct": 0.3,
  "classes": [
    {
      "name": "grass",
      "intent_type": "terrain_shader",
      "hex": "#a0d870",
      "biome_mask_path": "/tmp/biome_grass.png",
      "coverage_pct": 32.1,
      "expected_coverage_pct": 30.0,
      "shader_slot_hint": 0
    },
    {
      "name": "house",
      "intent_type": "object_placement",
      "hex": "#a04020",
      "instance_count": 47,
      "expected_count": 60,
      "instances": [
        {
          "id": "house_001",
          "position_px": [320, 410],
          "position_world": [-5.63, 6.09],
          "bbox_px": [310, 400, 330, 420],
          "size_px": [20, 20],
          "size_world": [1.56, 1.56],
          "rotation_deg": 12.0,
          "rotation_confidence": 0.71,
          "area_px": 380
        }
      ]
    },
    {
      "name": "valley",
      "intent_type": "terrain_displacement",
      "heightmap_ref": "/tmp/heightmap.png",
      "displacement_meters": [0, 5]
    }
  ]
}

The procedure

Step 1 — Read inputs

cat /tmp/_class_catalog.json  # peek at classes + hexes
file /tmp/semantic_map.png   # confirm exists

Step 2 — Write the per-scene extraction script

The script lives at /tmp/_extract_<short_hash>.py (ephemeral). It uses tools.visual_layout.lib_extract for all heavy lifting.

Skeleton:

#!/usr/bin/env python3
\"\"\"Per-scene extraction script — generated by yume-extract-author.
Catalog: <catalog hash>
Semantic map: <semantic_map hash>
\"\"\"
import json, sys
from pathlib import Path
sys.path.insert(0, "<repo_root>")
from tools.visual_layout import lib_extract as L

CATALOG_PATH    = Path("/tmp/_class_catalog.json")
SEMANTIC_PATH   = Path("/tmp/semantic_map.png")    # or wherever stage 3 wrote it
HEIGHTMAP_PATH  = Path("/tmp/heightmap.png")        # optional
OUT_PATH        = Path("/tmp/_extracted.json")
MASKS_DIR       = Path("/tmp/biome_masks")
WORLD_SIZE_M    = (80.0, 80.0)   # configurable; check scene.json if exists

# ----- Step A: load -----
catalog = json.loads(CATALOG_PATH.read_text())
img = L.load_rgb(SEMANTIC_PATH)
H, W, _ = img.shape

# ----- Step B: hard partition via nearest-palette ----- 
# Only the classes that ARE in the semantic map color channel.
# terrain_displacement classes are NOT in the palette (they live
# in the heightmap), so filter them out here.
color_palette = [
    (cls["name"], cls["hex"])
    for cls in catalog["classes"]
    if cls["intent_type"] != "terrain_displacement"
]
labels = L.threshold_nearest_palette(img, color_palette)

# ----- Step C: per-class extraction -----
classes_out = []
for idx, cls in enumerate([c for c in catalog["classes"]
                            if c["intent_type"] != "terrain_displacement"]):
    name = cls["name"]
    mask = (labels == idx)
    n_px = int(mask.sum())
    pct  = 100 * n_px / (H * W)
    entry = {
        "name": name,
        "hex":  cls["hex"],
        "intent_type": cls["intent_type"],
    }

    if cls["intent_type"] == "terrain_shader":
        # Save the mask + emit biome metadata
        mask_path = MASKS_DIR / f"biome_{name}.png"
        L.save_mask(mask, mask_path)
        entry["biome_mask_path"] = str(mask_path)
        entry["coverage_pct"] = round(pct, 3)
        entry["expected_coverage_pct"] = cls.get("expected_coverage_pct")
        entry["shader_slot_hint"] = idx
        classes_out.append(entry)

    elif cls["intent_type"] == "object_placement":
        # Find connected components, infer rotation per instance
        comps = L.connected_components(mask, connectivity=4, min_area=20)
        instances = []
        for i, comp in enumerate(comps):
            angle_deg, elongation = L.infer_rotation_deg(comp)
            # If elongation < 1.2 (roughly isotropic), rotation is
            # noise → set to 0
            if elongation < 1.2:
                angle_deg = 0.0
            x0, y0, x1, y1 = comp["bbox"]
            cx, cy = comp["centroid"]
            wx, wz = L.pixel_to_world(cx, cy, (W, H), WORLD_SIZE_M)
            size_world_x, _ = L.pixel_to_world(x1 - x0, 0, (W, H), WORLD_SIZE_M)
            _, size_world_z = L.pixel_to_world(0, y1 - y0, (W, H), WORLD_SIZE_M)
            instances.append({
                "id": f"{name}_{i+1:03d}",
                "position_px":    [int(cx), int(cy)],
                "position_world": [wx, wz],
                "bbox_px":        [x0, y0, x1, y1],
                "size_px":        [x1 - x0 + 1, y1 - y0 + 1],
                "size_world":     [round(abs(size_world_x - L.pixel_to_world(0, 0, (W, H), WORLD_SIZE_M)[0]), 3),
                                   round(abs(size_world_z - L.pixel_to_world(0, 0, (W, H), WORLD_SIZE_M)[1]), 3)],
                "rotation_deg":   angle_deg,
                "rotation_confidence": round(min(elongation - 1, 1.0), 2),
                "area_px":        comp["area_px"],
            })
        entry["instances"] = instances
        entry["instance_count"] = len(instances)
        entry["expected_count"] = cls.get("expected_count")
        classes_out.append(entry)

# ----- Step D: heightmap reference for displacement classes -----
for cls in catalog["classes"]:
    if cls["intent_type"] == "terrain_displacement":
        entry = {
            "name": cls["name"],
            "intent_type": "terrain_displacement",
            "heightmap_ref": str(HEIGHTMAP_PATH) if HEIGHTMAP_PATH.exists() else None,
            "displacement_meters": catalog["heightmap_hints"]["elevation_range_estimate_meters"],
        }
        classes_out.append(entry)

# ----- Step E: write extracted.json -----
L.write_extracted_json(
    OUT_PATH,
    semantic_map_path=str(SEMANTIC_PATH),
    catalog_path=str(CATALOG_PATH),
    image_size=(W, H),
    world_size_m=WORLD_SIZE_M,
    classes=classes_out,
    coverage_pct=100.0,           # nearest-palette gives 100% by construction
    unassigned_pct=0.0,
    notes=f"Extracted by per-scene script from yume-extract-author. {len(classes_out)} classes.",
)
print(f"wrote {OUT_PATH}")

Step 3 — Run the script

python3 /tmp/_extract_<hash>.py

Step 4 — Validate the output

Read /tmp/_extracted.json. Check:

• Coverage — coverage_pct should be 100.0 (nearest-palette partition is exhaustive by construction)
• Class counts vs expected_count from the catalog — drift > 50% in either direction is a yellow flag (LLM mis-counted in stage 2, or stage 3 generated too few/many)
• Coverage drift for terrain_shader classes — drift > 30% from expected_coverage_pct is a yellow flag
• Per-instance sanity — bbox sizes shouldn't all be 1px (= tiny noise components), shouldn't all be 90% of the image (= over-merged)

Step 5 — Iterate if needed

If validation reports issues:

• Wrong color tolerance → adjust threshold_by_hex calls
• Components too small → bump min_area to drop noise
• Components too merged → drop min_area or use connectivity=8

The per-scene script is ephemeral — just rewrite it. Don't try to make the script handle every edge case generically; the LIBRARY handles the generic cases, the per-scene script handles the per-scene routing.

Custom per-scene logic

Most scenes work with the skeleton above unchanged. But some need custom handling:

Multi-color "family" classes

If the catalog has small_house, large_house, townhall all in the red-brown family, but you want extraction to group them as a "building" family with size-subclassing, modify Step C:

# Group house family
house_family = ["small_house", "medium_house", "large_house", "townhall"]
house_components = []
for fam_name in house_family:
    fam_idx = palette_names.index(fam_name)
    fam_mask = (labels == fam_idx)
    for comp in L.connected_components(fam_mask):
        comp["family_member"] = fam_name
        house_components.append(comp)
# emit as a single "building" class with subclass tag per instance

Class with required minimum count

If a class MUST have at least N instances (e.g. "tower" must have ≥8 for an octagonal wall), validate during extraction and emit a warning JSON field:

if cls["name"] == "tower" and len(instances) < 8:
    entry["_warning"] = f"expected ≥8 towers, found {len(instances)} — wall may be incomplete"

Connected-component splitting

Sometimes a class's pixels form ONE big blob that should be split (e.g. a long market row that the model emitted as one connected patch). Handle by:

• After connected_components, if a component's bbox aspect ratio is extreme (>5:1), split it into N equal segments along the long axis.

Strict rules

1. The library handles primitives; the script handles routing. Don't reimplement color thresholding / flood fill in the per-scene script — call lib_extract.threshold_by_hex / connected_components / infer_rotation_deg.

2. Output JSON schema is fixed. All downstream consumers (stage 6 asset gen, stage 7 engine wiring) depend on the field names listed in the Output section above. Don't rename fields.

3. rotation_deg = 0 when elongation < 1.2. The PCA-derived angle is meaningless for isotropic shapes (circles, squares). Always check elongation and zero-out if low.

4. Coordinate convention: origin = center of map. world_size_m defines the world extent in meters; pixel (W/2, H/2) maps to world (0, 0). Confirms with the engine's convention (position_scale=1.0 for aldenmere → world coords in meters).

5. terrain_displacement has NO color in the semantic map. It lives in the heightmap. Emit a heightmap_ref entry per terrain_displacement class without trying to threshold colors.

Worked example

Run on the medieval-town stage-3 output (semantic_town_via_edits.png)

• stage-2 catalog (_class_catalog.json with 12 classes). The resulting extracted.json should have:

• 6 terrain_shader entries with biome_mask paths + coverage %
• 6 object_placement entries with per-instance positions:
  • house: ~50-70 instances (catalog expected 60)
  • wall_segment: ~8-12 instances (catalog expected 8)
  • tower: ~10-14 instances (catalog expected 12)
  • bridge: ~2 instances
  • fountain: 1 instance
  • townhall: 1 instance
• Total coverage 100%

The bbox sizes + centroids feed stage 6 (asset prompts) — each unique class becomes one asset prompt template; the per-instance data becomes scene placements.

What this skill is NOT

• NOT a generator of the semantic map (stage 3)
• NOT a generator of the catalog (stage 2)
• NOT a generator of asset prompts (stage 6)
• NOT an engine integrator (stage 7)
• NOT a member of the locked 2D pipelines (HUD/screen)

Reference files

• tools/visual_layout/lib_extract.py — generic helpers (this skill's library)
• .claude/skills/yume-scene-class-catalog/SKILL.md — stage 2 producer of the catalog this consumes
• godot/data/lib/semantic_palette.json — the conventions backing the catalog