Img2threejs

img2threejs — Image to procedural Three.js

Rebuild the object visible in a reference image as a code-only procedural Three.js model, gated by a staged sculpting pipeline and an AI-vision self-correction loop. This is reconstruction-by-code, not photogrammetry, mesh extraction, or downloaded art packs.

Use Codex-native image reading, an already-enabled browser tool, the project preview, or a user-supplied screenshot wherever this workflow calls for agent vision or a browser.

Runtime and paths

Resolve SKILL_ROOT to the directory containing this SKILL.md. When Codex exposes PLUGIN_ROOT, this skill is at ${PLUGIN_ROOT}/skills/img2threejs; otherwise use the skill's discovered absolute path. Run every bundled script by absolute path from SKILL_ROOT and keep generated specifications, renders, and models in the user's project or an approved output directory, never in the installed plugin.

Use the active Python 3.10+ executable: py -3 or python on Windows, and python3 or the configured environment's interpreter on macOS. Pass arguments as an array and do not build a shell command. In command examples, <python> means that selected executable and <skill-root> means the resolved absolute SKILL_ROOT. The harness itself uses only the standard library.

The local specification index never writes into the installed plugin. Use --cache-root <approved-path> or AI_GAME_STUDIO_CACHE_DIR; otherwise a bounded OS user-cache directory is used. Generated specifications, code, renders, and evidence still belong in the approved project/output directory.

The built-in pixel tools decode compatible PNG directly. Common JPEG/TIFF/BMP inputs may be converted through the fixed native Windows System.Drawing or macOS /usr/bin/sips route. A non-native ImageMagick fallback is allowed only when AI_GAME_STUDIO_IMAGEMAGICK_MANIFEST points to a confirmed JSON record containing its absolute executable, SHA-256, version, reviewed license, HTTPS source URL, and plan digest. Unsupported formats must stop with a conversion request; never run an executable selected only from PATH.

This plugin vendors upstream v1.4.3 under Apache-2.0. Read UPSTREAM.json and LICENSE before redistributing changes.

Core Promise

Sculpt from a photo, in order — never one-shot a mesh:

1. Run <python> "<skill-root>/forge/next.py" <spec> first. It reports the current unlocked pass, exact next command, and unmet acceptance criteria.
2. Validate the image is a suitable 3D target (grimoire/intake/validation_rubric.md).
3. Assess object class + complexity, then write a qualityContract before any code.
4. Spec it: component hierarchy, materials, lighting, pivots, sockets, action anchors.
5. Build pass-by-pass from blockout → structure → form → material → lighting → interaction → optimization.
6. Verify each pass with a screenshot compared against the reference; fail a pass if an identity-defining feature is wrong even when the global score looks fine.

State explicitly when output is approximate/stylized/low-poly. A single image cannot reveal hidden sides or guarantee exact geometry — say so instead of faking confidence.

Transparency and Process Debugging (Critical — from Bowie Knife reconstruction)

The problem: When the user cannot tell what was done or where something went wrong, they cannot debug the process. Over-claiming (reporting success when features still don't match) destroys trust and makes iterative improvement impossible.

Rule: Be transparent + don't over-claim. State exactly what changed each pass, with evidence, and name what still doesn't match:

• After each pass, explicitly list what changed: "Updated guard shape to extend left edge from -0.56 to -0.48 for handle overlap"
• Provide evidence: reference the specific values, coordinates, or parameters that changed
• Name what still doesn't match: "Handle silhouette traced but still flat plane (no Z palm-swell), procedural crosshatch not reference's exact dot-grid knurl"
• Explain why a change was made: "Extended guard left edge because handle ends at X=-0.42 and guard ended at X=-0.20, causing visual gap"
• Never claim a feature is "done" when it's only "improved" — use precise language
• When a gate passes but visual inspection shows issues, explain the limitation: "2D gate passed (fidelity 0.83) but three-quarter render shows blade reads as toy (no grind wedge) — 2D gates are blind to 3D realism"

The user needs to be able to debug the process, not just the output. If something is wrong, they should be able to trace which decision led to the error and correct it. Opaque processes force restarts; transparent processes enable refinement.

Required Inputs

• one image path / screenshot / URL / attached image (if missing or unreadable, ask)
• intended use: prop, game object, hero render, playable/destructible object, animation rig (default: real-time browser prop with interactive performance)
• for a CS2 request, an authoritative classification record (family/subtype and evidence refs) or an explicit request for the user/vision provider to supply one; heuristic detection alone is not enough to select a geometry adapter

Download a URL to an approved local input path only through a separately reviewed network step. The forge accepts local image files; it does not treat an arbitrary URL as permission to fetch or upload content.

The Loop (scripts do enforcement; agent vision does judgment)

Run scripts through their absolute paths under <skill-root>/forge/.... Pure Python 3.10+ stdlib, no pip installs. Full flags: grimoire/scripts.md. Never let a script score visuals — that is the agent's job.

1. Analyze the image first (agent vision, before any script): work the layered observation protocol in grimoire/intake/image_analysis.md — identify/classify, decompose macro→meso→micro, map part relationships, name materials in PBR terms, list identity-defining features, and flag what the single view hides. Observation before inference; controlled 3D vocabulary; 3D object-space not 2D image-space. This is generic for any subject and feeds every field below. Then probe local images: forge/stage1_intake/probe_image.py <image> (metadata only, not a visual check). 1a. Local Spec Search — after image analysis and before writing or refining a spec, local evidence is a pipeline stage, not an optional memory lookup, whenever the request needs domain-specific anatomy, PBR, wear, geometry, runtime, or physics specifications. The pre-spec command automatically runs BM25, chooses cs2 for CS2 targets and core_3d otherwise, and writes a localSpecSearch evidence bundle into the assessment: <python> "<skill-root>/forge/stage2_spec/new_pre_spec_assessment.py" "Name" --image <img> --out assessment.json. Add observed terms with repeatable --spec-query "<term>"; use --collection <collection> only when the automatic collection choice is insufficient. new_sculpt_spec.py --assessment carries that bundle into the final spec, including snippets, source_refs, and evidence_refs. For extra focused retrieval, the direct CLI remains available: <python> "<skill-root>/forge/stage1_intake/search_specs.py" "<query>" --collection <collection> --limit 3 --snippet-chars 250 --json. For CS2, include English/Vietnamese variants, for example --spec-query "safety ring vòng ngón" or search_specs.py "roughness độ nhám" --collection cs2. Expand queries with object names, component names, material/finish terms, behavior terms, and bilingual aliases; retry focused alternatives when the first result is incomplete. Build the spec from returned evidence and do not invent domain specs when local evidence exists. Search caches are local/generated only; preserve JSONL records and source provenance rather than replacing them with cache output. 1b. CS2 intake manifest — for a CS2 request, create and validate cs2-intake.json before pre-spec authoring. Run admission and probing for every source view, record the heuristic signal as non-authoritative evidence, attach the classification record, resolve the supported family, and choose route independently from exactnessTier. Missing classification, insufficient coverage, or a contradictory high-confidence class is request-input; unsupported families do not continue into spec generation.
2. Pre-Spec Assessment Gate — classify + score complexity + write the quality contract: forge/stage2_spec/new_pre_spec_assessment.py "Name" --image <img> --complexity <simple|moderate|complex|ultra-complex> --out assessment.json. Rules: grimoire/intake/quality_contract.md. Set objectClass.primaryDomain (object | character | hybrid) and fill the seeded detailInventory (its targetMinDetails scales with complexity). Supported CS2 knife skins: always pass --cs2, which defaults the complexity tier to ultra-complex (targetMinDetails 16) — the finish/wear/hardware is the item, so CS2 is held to the top fidelity bar; targetMinDetails never drops below the 9 floor even if downgraded by hand. Author procedural GEOMETRY (blade/guard/grip profiles) but make the FINISH a de-lit reference-crop PROJECTION, not a procedural finish material — projecting the photo's own pixels is what reaches reference fidelity for patterned skins (Doppler/Gamma/Marble/Fade), and is what the v1.3 baseline demos do; a procedural finish for a patterned skin reads visibly wrong against the reference. Take the projection path in step 2c (it generalizes from characters to any reference-matched surface). Procedural finish is the fallback ONLY when live view-dependent response matters more than matching this one reference. Finish routes + rulebook: grimoire/build/cs2_finishes.md; optional exact-texture acquisition: grimoire/intake/cs2_texture_acquisition.md. 2b. Detail inventory (do not skip for detailed subjects) — scan zones and enumerate every identity-defining small detail (gloss, bevel, fasteners, linework, contours, stains): forge/stage1_intake/build_detail_inventory.py <image> --mode grid-3x3 --out-dir <dir> --out di.json. Each detail MUST map to a component.localFeatures or material.localOverrides entry — never prose only. Taxonomy + 3D-term recipes: grimoire/intake/detail_inventory.md. 2c. Projection-first fidelity (characters AND reference-matched surfaces — supported CS2 knife skins, decals, painted patterns) — when the goal is matching a specific reference's surface, put the photo's own pixels on the mesh instead of approximating them procedurally. This is the single biggest fidelity lever; a procedural material for a patterned surface is the #1 reconstruction failure. Recipe (grimoire/character/likeness_maximization.md — its two levers, align-mesh+camera and project-the-photo, generalize past characters): solve the camera (stage1_intake/solve_camera_pose.py → referenceCamera), de-light the reference so it is free of baked lighting (stage1_intake/delight_albedo.py, hard requirement — this is what makes projection safe, not the flat-lit icon), then project the de-lit crop onto the mesh and bake it into UVs (stage3_build/bake_projected_texture.py --mesh-id <id>). For a CS2 skin the mesh is the procedural blade/guard/grip you author in the spec, and the projected de-lit crop IS the finish (front + back from the two views) — no procedural Doppler material. For characters, first capture landmarks (stage1_intake/extract_landmarks.py --out anatomy.json), fill preSpecAssessment.anatomy, route grimoire/character/reconstruction.md. A single view cannot show hidden sides — report per-region confidence and request more views when it matters.
3. Author the spec from the assessment: forge/stage2_spec/new_sculpt_spec.py "Name" --image <img> --assessment assessment.json --manifest cs2-intake.json --out object-sculpt-spec.json. Replace generic starter featureReviewTargets with the object's real identity-defining systems (≤5 critical, ≤3 important per pass); for characters add anatomy-proportion, face-landmark-placement, pose-silhouette, outfit-and-palette. Use 3D-graphics terms only (grimoire/glossary/3d_vocabulary.md), never "nice/smooth/shiny". Classify every component's topologyClass/topologyRationale per grimoire/intake/surface_topology.md before picking a primitive — this is what prevents a continuous organic form from being picked as a box.
4. When material fidelity matters and a source image exists, analyze each material's finish then extract reference PBR evidence, both per crop (crop the correct region — verify the crop is on the part you think it is):
   • forge/stage1_intake/analyze_texture.py <crop> --spec spec.json --material-id <id> --in-place classifies the finish (gem-metal | gemstone | painted-metal | worn-composite | brushed-steel | plastic), extracts the gradient palette, and writes doc-grounded MeshPhysicalMaterial scalars (metalness/roughness/clearcoat/transmission/ior/anisotropy/envMapIntensity) onto the material. Recipes + Three.js texture/PBR rules (colorSpace, CanvasTexture/DataTexture, height→normal) live in grimoire/build/threejs_texture_reference.md. Rule of thumb: solid albedo for flat paint, real reference crop for patterned finishes (doppler/quartz/hydro-dip/camo).
   • forge/stage1_intake/extract_pbr_evidence.py <crop> --out-dir <dir> --material-id <id> --target-threshold 0.7. Confidence < 0.7 is a stop/refine-input signal, not a pass. It is inference, not inverse rendering.
5. Validate, then strict-validate before generating code: forge/stage2_spec/validate_sculpt_spec.py object-sculpt-spec.json then --strict-quality. Strict blocks shallow specs (a complex object with one root, no repetition systems, no local overrides, no micro groups is NOT implementation-ready even if JSON validates).
6. Locked build passes — only touch the currently unlocked pass: forge/stage3_build/orchestrate_passes.py status object-sculpt-spec.json forge/stage3_build/orchestrate_passes.py check object-sculpt-spec.json --pass-id <pass> forge/stage3_build/generate_threejs_factory.py object-sculpt-spec.json --out src/createObjectModel.ts (generator is pass-gated: a future --pass-id fails until prior passes are reviewed continue).
7. Render the current pass in a browser/preview, capture a screenshot at a review viewpoint.
8. Package one side-by-side sheet, then inspect it with agent vision: forge/stage4_review/make_comparison_sheet.py --reference <img> --render <shot> --out cmp.png --json.
9. Record the review (overall + per-layer + per-feature scores + decision): forge/stage4_review/append_review.py object-sculpt-spec.json --pass-id <pass> --fidelity <0-1> --action <continue|refine-spec|refine-code|request-input|stop> --summary "..." --render-screenshot <shot> --comparison-image cmp.png --ai-vision-score <0-1> --layer-scores-json '{...}' --feature-reviews-json <f.json> --in-place. For the CS2 knife path, also attach the versioned report with --cs2-review-json cs2-review.json --review-scene-json forge/tests/fixtures/knife_review_scene.json. A failed family, painted-region, projection-coverage, critical-detail, or orbit gate blocks continue even when the global score passes. See docs/cs2/review-gates.md.
10. Sync pipeline state after manual review edits: forge/stage3_build/orchestrate_passes.py sync object-sculpt-spec.json --in-place.

CS2 image-matched rule

For a CS2 item, the target is observable agreement between the supplied image and the rendered item: silhouette, proportions, edge profile, hardware layout, coating colour, pattern placement, wear, roughness response, and camera framing. Every decision must be traceable to evidence or be labelled as an approximation.

The initial CS2 family boundary is knife only. Pistol, rifle, SMG, sniper, heavy, glove, and unknown knife subtypes must stop with unsupported-family or unsupported-subtype; they must not receive the knife component tree as a generic fallback.

Layer contract

Pass these records between layers. Do not copy an informal vision description into the next stage:

The canonical hand-off is cs2-intake.json (schemaVersion: 1). Its state is one of proceed, request-input, fallback, rejected, unsupported-family, or unsupported-subtype. Write it atomically and preserve unknown provider fields under extensions; a fallback must never erase prior evidence.

CS2 intake order

1. Admit and technically probe every view. Reject undecodable, empty, tiny, fragmented, or duplicate references before classification.
2. Record the heuristic CS2 signal only as a routing hint. detect_cs2.py is never authoritative identity evidence.
3. Require a classification record before selecting a family adapter. If classification is absent, timed out, or contradicts a high-confidence objectness result, return request-input.
4. Resolve identity in this order: explicit user metadata, uniquely resolved metadata, then the authoritative classification record. Preserve ambiguity rather than guessing.
5. Select route and exactness independently:
   • reference-projection: default for matching a specific patterned image;
   • authored-texture: only when independent texture maps are supplied or legally acquired;
   • procedural-finish: fallback when projection evidence is unavailable or live response is the stated priority. Exactness is image-only, metadata-assisted, or exact-texture; changing route must not silently upgrade or downgrade the evidence tier.
6. Select the knife adapter only after family/subtype validation. Record painted regions, unpainted substrate, visible hardware, hidden-region confidence, and every approximation in the spec.
7. For projection, solve the camera and de-light the source first. Projected pixels provide colour evidence, not automatic geometry truth; geometry still comes from the adapter and silhouette review.

Surface and review rule

For a specific CS2 reference, preserve the reference's own colour/pattern pixels whenever legal and technically possible. Procedural Doppler/Fade/Gamma/Marble patterns are not equivalent to the input image and may only be used with an explicit procedural-finish route and approximation warning. Keep albedo, roughness, metalness, normal/height, AO, mask, and wear as independent channels. Record channel source, colour space, UV orientation, dimensions, packed-channel decoding, and missing-channel derivation. A low-confidence PBR inference is a refine-input signal, not proof of exact material.

Single-view reconstruction may proceed only when visible identity features are sufficiently covered; hidden blade sides, underside, and back hardware must carry inference confidence and may trigger request-input. Review the fixed camera plus two meaningful orbit views. Report what changed, which evidence caused it, what still differs, and choose exactly one next action: continue, refine-spec, refine-code, request-input, or stop.

Gates (do not skip)

• Suitability + reference integrity: pass / conditional / reject before any planning (grimoire/intake/validation_rubric.md), AND every reference admitted via forge/stage1_intake/check_reference_admission.py (rejects empty/fragmented/tiny/duplicate/ undecodable refs with a reason). Intake understanding cross-checked by forge/stage1_intake/check_intake_correctness.py (halts on a confident class contradiction).
• Divine Eye (the harness heart) — deterministic-first, model-last: the render evaluator is forge/stage4_review/divine_eye.py — a zero-token multi-signal ensemble (IoU/scale HARD gates; proportion/symmetry-parity/pHash/SSIM/edge/blowout/flat/tonal-parity soft) with self-uncertainty (probe on signal disagreement) and deterministic routing (continue/refine-spec/refine-code/ probe). The VLM (forge/stage4_review/vlm_gate.py) is a gated, calibrated, cross-checked last layer: never consulted on a hard-gate failure, multi-sample-voted, and can rescue a soft near-threshold reject but never grant past a hard geometric failure.
• Multi-angle or it didn't happen: a non-planar form must hold from ≥2 camera angles. forge/stage4_review/diagnose_render_multi_angle.py flags degenerate-view when an orbited silhouette collapses (a flat plane faking a volume). Orbit angles use reference-free self-consistency — never scored against a reference angle the photo doesn't cover.
• CS2 knife review contract: forge/stage4_review/cs2_review.py consumes the manifest and versioned scene fixture, then blocks wrong family identity, missing projection coverage, painted-region mismatch, critical identity-detail failure, finish/material response failure, and degenerate orbit form. It records exactness tier, hidden-region confidence, per-region confidence, approximation notes, camera, environment hash, exposure, tone mapping, resolution, background, and renderer version.
• Bounded correction loop (token-burn safety): forge/stage4_review/correction_loop.py guarantees termination (success/repeated-defect/oscillation/plateau/hard-ceiling), escalating to request-input — never a silent infinite burn.
• Tier 1 (legacy, still valid): "Tier 2 (AI-vision) never runs against a render that has not passed Tier 1." Run forge/stage4_review/diagnose_render.py (silhouette IoU/proportion/symmetry/per-part color) and record it (--spec ... --in-place) before requesting a comparison sheet; orchestrate_passes.py check refuses otherwise.
• Pre-spec / strict-quality: blocks code gen until the spec is deep enough for its contract.
• Screenshot feedback: continue is allowed only with a render + comparison sheet + global AI-vision score ≥ threshold (default 0.7) AND every critical feature ≥ its own threshold. Details + per-layer scorecard: grimoire/feedback/render_capture.md.
• Action-ready: build a runtime hierarchy (pivots, sockets, colliders, destruction groups), never an inert lump; expose root.userData.sculptRuntime. grimoire/readiness/action_rigging.md.
• Assembly gate (structure, not pixels) — every model ships explodable AND clickable: this is a build requirement, not a per-project extra. Name every mesh; flag surface relief userData.explodeWithParent so it rides its shell; let a named group of anonymous meshes be one part while a named group of named parts stays a container. Explode and part-picking must share one definition of "a part" — if they disagree, both are wrong. Separate parts by SCALING the layout about the model centre, never by pushing every part the same distance (that translates the arrangement without opening any gap). Then run forge/stage4_review/check_part_coverage.py --spec <spec> --manifest <parts.json>: it FAILS on a specified component that was never built and on two components fused onto one mesh; it warns on inventoried details that never reached the spec and on meshes belonging to no named part. This is the only gate that scores STRUCTURE — every other one scores pixels, and a single fused mesh wearing a projected photo passes all of those. Its limit is honest and must be stated when reporting: it proves you built what you specified, never that you specified enough. Full contract + the two rules it took a wrong pass to learn: grimoire/build/geometry_patterns.md.
• Attachment: child appendages (branches/limbs/handles/tubes) need attachment.parentSocket, localStart, localEnd, contactType, embedDepth/overlap, gapTolerance — no mid-air parts. grimoire/readiness/joint_attachment.md.
• Material/lighting: grimoire/feedback/shading_realism.md — independent PBR channels (never alias albedo into roughness/normal/AO), macro/meso/micro frequency bands, real lights.
• Detail inventory: for moderate+ subjects strict-quality blocks code gen until the detailInventory reaches targetMinDetails and every detail maps to a real component/material entry (gloss needs low-roughness/clearcoat; fasteners need instancing/micro parts).
• Character track: when primaryDomain is character/hybrid (or --character), the spec author auto-builds a stylized humanoid template (head/neck/torso/arms + hair, glasses, headphones, face features), flattened to world space under a hidden root, with per-part character materials and character build passes (proportion-lock, feature-placement). strict-quality requires a filled anatomy block (head-units, proportions, face landmarks) and character feature targets. Suitability routing for humans: grimoire/intake/validation_rubric.md (stylized vs maximum-likeness). Stylized bust, not a face-copy; refine positions per reference.

Self-Correction

After every pass, decide exactly one: continue | refine-spec | refine-code | request-input | stop. refine-spec fixes a wrong/missing/shallow spec (re-validate, don't patch code around it); refine-code fixes geometry/material/lighting that doesn't match a sound spec. Full root-cause guide + fidelity scale: grimoire/review/self_correction.md.

Implementation Rules (brief)

TypeScript + plain Three.js unless the project uses a wrapper. Group factory createObjectNameModel(spec, options), reconstruction data kept separate from renderer objects, deterministic seeds for all procedural noise. Prefer primitives / Shape extrude / curve+tube / instancing / displacement / generated canvas textures before any external art. Full geometry & material recipes + hard-won failure patterns: grimoire/build/geometry_patterns.md.

Output

• Analysis-only: suitability verdict + scores, object extraction, macro→micro hierarchy, geometry strategy, material/lighting recipe, animation/destruction feasibility, plan + risks.
• Implementation: the above briefly, then edit code; verify with typecheck/build + a screenshot.
• Not feasible: name the blocker, ask for more views / cleaner image / accepted stylization / a narrower target. "This cannot reach the requested fidelity from this image" is a valid result.