img2threejs

# img2threejs — Image to procedural Three.js Rebuild the object visible in a reference image as a **code-only** procedural Three.js model, gated by a staged sculpting pipeline and an AI-vision self-correction loop. This is reconstruction-by-code, **not** photogrammetry, mesh extraction, or downloaded art packs. Agent-agnostic: works under Claude Code, Codex, or OpenCode. Wherever this doc says "agent vision" or "agent browser tool", use whatever the host provides — native image reading, a browser MCP (playwright/chrome-devtools), the project preview, or a user-supplied screenshot. This file is the always-loaded router: it holds the order of operations and every hard rule as one line. The full contract behind each rule lives in the `grimoire/` or `docs/` file that rule names — read the named file at the moment you reach that stage, not before. ## Canonical shared checkout Keep one checkout of this repository and let every host enter it through a symlink, so Claude and Codex execute the same code instead of drifting apart: ```text ~/.claude/skills/img2threejs -> <your checkout> ~/.codex/skills/img2threejs -> <your checkout> ``` ## When To Use The user attaches/points to an object image and wants a procedural Three.js model, a reconstruction/animation/destruction plan, a sculpt spec, or code. Also for material studies, action-ready props, game objects, botanical/mechanical parts, and stylized reconstructions. ## Core Promise Sculpt from a photo, in order — never one-shot a mesh: 1. **Run `python3 forge/next.py --state .img2threejs/state.json [<spec>]` first**, at every start, resume, and before every correction iteration. It reports the ordered checklist, exact next command, evidence status, and bounded correction-loop status; it never replaces the spec/pass gates. Obey a hard stop; never continue from memory. 2. **Validate** the image is a suitable 3D target (`grimoire/intake/validation_rubric.md`). 3. **Assess** object class + complexity, then write a `qualityContract` before any code. 4. **Spec** it: component hierarchy, materials, lighting, pivots, sockets, action anchors. 5. **Build pass-by-pass** from blockout → structure → form → material → lighting → interaction → optimization. 6. **Verify** each pass with a screenshot compared against the reference; fail a pass if an identity-defining feature is wrong even when the global score looks fine. State explicitly when output is approximate/stylized/low-poly. A single image cannot reveal hidden sides or guarantee exact geometry — say so instead of faking confidence. ## Mandatory Local State Gate Conversation context is disposable; `.img2threejs/state.json` is the local checklist authority. Initialize once per reconstruction, then gate every step through it: ```bash python3 forge/state.py init --state .img2threejs/state.json --reference <img> --profile <generic|cs2|character|animated-character> --spec object-sculpt-spec.json python3 forge/next.py --state .img2threejs/state.json [object-sculpt-spec.json] python3 forge/state.py mark <step-id> --state .img2threejs/state.json --evidence <path> ``` - `next.py` prints the current step, pass, incomplete mandatory steps, exact next command, and `loop/max`. Exit code 3 or `status=stopped` is a hard stop: report the reason and request input. Never bypass it by reconstructing progress from chat history. - Every completed step needs evidence; mark a non-applicable step `skipped` only with `--reason` — silent omission is forbidden. Loop counts derive from `reviewHistory` actions (`refine-spec`/`refine-code`), not agent memory. Defaults: 3 corrections per pass, 6 total. - Profiles add mandatory gates without changing the core order: `cs2` requires classification, manifest, and a machine-readable CS2 review before AI review; `character` requires the character contracts and landmark evidence; `animated-character` adds all of `character` plus the nine Stage R steps (`grimoire/readiness/animation_contract.md`). Pick it whenever the rig must MOVE — on `character` the Stage R gates are absent and the build completes without ever running them, which is how animation used to ship broken. Its order is load-bearing: repair the mesh, freeze it, bind additively, then verify parity. Every profile records suitability, projection applicability, and material-evidence applicability. The state file is a resumability index, not visual evidence: renders, specs, review history, and deterministic gates remain the authoritative artifacts. ## Required Inputs - one image path / screenshot / URL / attached image (if missing or unreadable, ask) - intended use: prop, game object, hero render, playable/destructible object, animation rig (default: real-time browser prop with interactive performance) - for a CS2 request, an authoritative classification record (family/subtype and evidence refs) or an explicit request for the user/vision provider to supply one; heuristic detection alone is not enough to select a geometry adapter ## The Loop (scripts do enforcement; agent vision does judgment) Run scripts from the skill root (`forge/...`). Pure Python 3.10+ stdlib, no pip installs. Full flags: `grimoire/scripts.md`. Never let a script *score* visuals — that is the agent's job. 1. **Analyze the image first** (agent vision, before any script): work the layered observation protocol in `grimoire/intake/image_analysis.md` — identify/classify, decompose macro→meso→micro, map part relationships, name materials in PBR terms, list identity-defining features, and flag what the single view hides. Observation before inference; controlled 3D vocabulary; 3D object-space not 2D image-space. Then probe local images: `forge/stage1_intake/probe_image.py <image>` (metadata only, not a visual check). 1a. **Local Spec Search** — after image analysis, before writing or refining a spec, pull local domain evidence (anatomy/PBR/wear/geometry/runtime/physics) rather than inventing it: `python3 forge/stage2_spec/new_pre_spec_assessment.py "Name" --image <img> --out assessment.json` (auto-runs BM25, auto-picks `cs2`/`core_3d` collection, writes a `localSpecSearch` bundle that `new_sculpt_spec.py --assessment` carries into the spec). Full query-expansion recipe (bilingual terms, focused `search_specs.py` retrieval, cache rules): `grimoire/intake/local_spec_search.md`. MUST read it before retrying an incomplete or domain-specific query. 1b. **CS2 intake manifest** — for a CS2 request, create and validate `cs2-intake.json` before pre-spec authoring (admission, heuristic signal, classification, family/route resolution). MUST read `grimoire/intake/cs2_intake_contract.md` completely before creating the manifest or running pre-spec assessment. 1c. **Optional fidelity evidence adapters** — only when they improve an observed weak point; the stdlib core remains authoritative. Thin/complex masks → local SAM2; character face/pose → MediaPipe; weak front/back cues → Depth Anything V2 (`forge/stage1_intake/run_vision_adapter.py <segment|landmarks|depth> ...`; every adapter emits provenance; monocular depth is relative only). **MCP-only scene mutations never count as implementation** — write the proven change back to the spec or TypeScript, rebuild, recapture. Full adapter + MCP routing and authority boundaries: `docs/integrations/reference_fidelity_tooling.md`. 2. **Pre-Spec Assessment Gate** — classify + score complexity + write the quality contract: `forge/stage2_spec/new_pre_spec_assessment.py "Name" --image <img> --complexity <simple|moderate|complex|ultra-complex> --out assessment.json`. Rules: `grimoire/intake/quality_contract.md`. Set `objectClass.primaryDomain` (`object` | `character` | `hybrid`) and fill the seeded `detailInventory` (its `targetMinDetails` scales with complexity). **Supported CS2 knife skins and Glock-18 assets**: always pass `--cs2`, which defaults the complexity tier to `ultra-complex` (`targetMinDetails` 16, floor 9) — the finish/wear/hardware is the item, so CS2 is held to the top fidelity bar. Author procedural GEOMETRY but route the FINISH through the projection path in step 2c — a procedural finish for a patterned skin (Doppler/Gamma/Marble/Fade) reads visibly wrong against the reference. Finish routes + rulebook: `grimoire/build/cs2_finishes.md`; optional exact-texture acquisition: `grimoire/intake/cs2_texture_acquisition.md`. 2b. **Detail inventory** (do not skip for detailed subjects) — scan zones and enumerate every identity-defining small detail (gloss, bevel, fasteners, linework, contours, stains): `forge/stage1_intake/build_detail_inventory.py <image> --mode grid-3x3 --out-dir <dir> --out di.json`. Each detail MUST map to a `component.localFeatures` or `material.localOverrides` entry — never prose only. Taxonomy + 3D-term recipes: `grimoire/intake/detail_inventory.md`. 2c. **Projection-first fidelity** (characters AND reference-matched surfaces — supported CS2 skins, decals, painted patterns) — when the goal is matching a specific reference's surface, put the photo's own pixels on the mesh instead of approximating them procedurally. This is the single biggest fidelity lever; a procedural material for a patterned surface is the #1 reconstruction failure. Recipe (`grimoire/character/likeness_maximization.md` — its two levers generalize past characters): solve the camera (`stage1_intake/solve_camera_pose.py` → `referenceCamera`), **de-light** the reference (`stage1_intake/delight_albedo.py`, hard requirement — de-lighting is what makes projection safe), then project the de-lit crop and bake it into UVs (`stage3_build/bake_projected_texture.py --mesh-id <id>`). For a CS2 skin the projected de-lit crop IS the finish — no procedural Doppler material. For characters, first capture landmarks (`stage1_intake/extract_landmarks.py --out anatomy.json`), fill `preSpecAssessment.anatomy`, route `grimoire/character/reconstruction.md`. A single view cannot show hidden sides — report per-region confidence and request more views when it matters. Character sub-routes, in order — decide what parts exist before shaping any, and shape the head before the hair that sits on it: - **Parts** — `grimoire/character/structure_decomposition.md` - **Head** — `grimoire/character/head_construction.md` (what the likeness gate reads against) - **Hair** — `grimoire/character/stylized_hair_threejs.md` + parameter contract in `grimoire/character/threejs_hair_parameter_contract.json`. Lock topology only after the silhouette review passes: material tuning cannot repair wrong lock topology. 2d. **Reference-free humanoid** — a generic figure with no reference image has nothing to measure, so fill anatomy from public canon: `forge/stage2_spec/humanoid_proportions.py <spec> --style-heads 8 --in-place`. It writes `anatomy.source: "canon-table"` so canon is never mistaken for measurement, refuses to run when the spec names a reference image, and names anything the corpus does not supply rather than interpolating it. 3. Author the spec from the assessment: `forge/stage2_spec/new_sculpt_spec.py "Name" --image <img> --assessment assessment.json --manifest cs2-intake.json --out object-sculpt-spec.json`. Replace generic starter `featureReviewTargets` with the object's real identity-defining systems (≤5 critical, ≤3 important per pass); for characters add `anatomy-proportion`, `face-landmark-placement`, `pose-silhouette`, `outfit-and-palette`. Use 3D-graphics terms only (`grimoire/glossary/3d_vocabulary.md`), never "nice/smooth/shiny". Classify every component's `topologyClass`/`topologyRationale` per `grimoire/intake/surface_topology.md` before picking a `primitive` — this is what prevents a continuous organic form from being picked as a box. 4. When material fidelity matters and a source image exists, analyze each material's **finish** then extract reference PBR evidence, both per crop (verify the crop is on the part you think it is): - `forge/stage1_intake/analyze_texture.py <crop> --spec spec.json --material-id <id> --in-place` classifies the finish, extracts the gradient palette, and writes doc-grounded MeshPhysicalMaterial scalars onto the material. Recipes + Three.js texture/PBR rules: `grimoire/build/threejs_texture_reference.md`. Rule of thumb: **solid albedo for flat paint, real reference crop for patterned finishes**. - `forge/stage1_intake/extract_pbr_evidence.py <crop> --out-dir <dir> --material-id <id> --target-threshold 0.7`. Confidence < 0.7 is a stop/refine-input signal, not a pass. It is inference, not inverse rendering. - For multiple named regions: `forge/stage1_intake/material_region_analysis.py --manifest regions.json --out-dir material-evidence --out material-analysis.json`, resolve each assignment from `docs/materials/material-reference.json`, wire it in with `forge/stage2_spec/apply_material_analysis.py`. - Emit the controlled material camera/crop contract (`forge/stage4_review/material_views.py`), compare visible-footprint crops (`material_comparator.py`), apply only bounded material-scoped corrections (`material_feedback.py`), and record the blocking result (`material_gate.py`). 5. Validate, then strict-validate before generating code: `forge/stage2_spec/validate_sculpt_spec.py object-sculpt-spec.json` then `--strict-quality`. Strict blocks shallow specs (a complex object with one root, no repetition systems, no local overrides, no micro groups is NOT implementation-ready even if JSON validates). 6. **Locked build passes** — only touch the currently unlocked pass: `forge/stage3_build/orchestrate_passes.py status object-sculpt-spec.json` `forge/stage3_build/generate_threejs_factory.py object-sculpt-spec.json --out src/createObjectModel.ts` The generator is fail-closed: `strict-quality` must pass before it writes any factory, and a future `--pass-id` fails until prior passes are reviewed `continue`. If blocked, preserve the `BLOCKED` artifact and refine the subject-specific spec; do not substitute a generic template. The local state adds `--force` only for a new pass or `refine-spec`; `refine-code` edits the current artifact without regenerating it. Before overwriting, carry valid hand refinement back into the spec; generated code must not be the only copy of reconstruction decisions. 6a. **Hitting a triangle budget.** `performanceBudget.targetTriangles` selects a tessellation tier for every primitive with segment counts (low ≤6k, standard ≤60k, else hero) and caps implicit-surface sampling grids. Where a tier is not precise enough, add `geometryDescriptor.decimate: {"targetRatio": 0.4}` to that component — a quadric collapse in the generated factory, run **before** skin binding so weights are computed on surviving vertices. It keeps `position` only (normals recomputed), so it is refused on an authored/unwrapped `uvStrategy`. Offline LOD tiers: `forge/stage3_build/decimate.py <mesh.json> --ratio <r> --json`. 7. Render the current pass in a browser/preview, capture a screenshot at a review viewpoint. 7a. **Off-axis and placement gates — a single review viewpoint is not evidence about the model.** Capture a turntable, not one frame, and run all three; each catches a defect class the older gates pass by construction (a hole through a skull, a hat at hip height and a floating charm all survived eight front-only review rounds): `forge/stage4_review/turntable_gate.py --capture 0=front.png --capture 90=right.png --capture 180=rear.png --capture 270=left.png --json` `node runtime/scripts/export_mesh_geometry.mjs --url <preview> --out meshes.json` then `forge/stage4_review/self_intersection.py meshes.json --json` `forge/stage4_review/attachment_anchor.py object-sculpt-spec.json --measured measured.json --json` All three exit `0` clean / `1` gate failure / `2` error. A failure blocks `continue` even when the global fidelity score passes. Read `sampledVertexCount` / `unmeasuredAttachments` / `missingAzimuths` before believing a clean verdict: each names what the gate did not look at. 8. **Run deterministic gates before AI vision.** MUST read `grimoire/review/gates_reference.md` and `grimoire/review/self_correction.md` completely. Run `forge/stage4_review/diagnose_render.py` and record the passing Tier 1 result with `--spec object-sculpt-spec.json --pass-id <pass> --in-place`; for non-planar forms also run `forge/stage4_review/diagnose_render_multi_angle.py` with the fixed view and at least two meaningful orbit views. Then run `forge/stage3_build/orchestrate_passes.py check object-sculpt-spec.json --pass-id <pass>`. 9. Package one side-by-side sheet, then inspect it with agent vision: `forge/stage4_review/make_comparison_sheet.py --reference <img> --render <shot> --out cmp.png --json`. 10. Record the review (overall + per-layer + per-feature scores + decision): `forge/stage4_review/append_review.py object-sculpt-spec.json --pass-id <pass> --fidelity <0-1> --action <continue|refine-spec|refine-code|request-input|stop> --summary "..." --render-screenshot <shot> --comparison-image cmp.png --ai-vision-score <0-1> --layer-scores-json '{...}' --feature-reviews-json <f.json> --in-place`. For the CS2 family path, produce the versioned report first with `forge/stage4_review/cs2_review.py --manifest cs2-intake.json --metrics cs2-review-inputs.json --scene forge/tests/fixtures/knife_review_scene.json --out cs2-review.json` and attach it with `--cs2-review-json cs2-review.json --review-scene-json forge/tests/fixtures/knife_review_scene.json`. A failed family, painted-region, projection-coverage, critical-detail, or orbit gate blocks `continue` even when the global score passes. See `docs/cs2/review-gates.md`. 11. Sync pipeline state after manual review edits, record checklist evidence, then re-run the local state gate before another correction or pass: `forge/stage3_build/orchestrate_passes.py sync object-sculpt-spec.json --in-place` `python3 forge/next.py --state .img2threejs/state.json object-sculpt-spec.json`. 12. Before declaring completion, run `forge/stage4_review/check_part_coverage.py --spec object-sculpt-spec.json --manifest parts.json` and verify the action-ready hierarchy. Mark `part-coverage` and `action-ready` only with evidence. ## GLB-mediated v2 render-fidelity track (1.5 alpha)