Person-into-scene image generation
Put a SPECIFIC real person into a scene template, photorealistically, keeping their
actual identity. Built + battle-tested 2026-06-10 (Markus). Assets:
~/Documents/image-gen/ (scene-templates/, markus-references/, generated/,
MANIFEST.md, HANDOFF.md). Code lives in ~/Projects/imagegen/ (own repo since
2026-06-10, split out of phenome) — run just/uv commands from there.
ASK THE DESTINATION FIRST (before generating anything)
The output format decides scene orientation AND final crop — get it wrong and you
regenerate. Markus's images are for dating apps (Hinge/Tinder/Bumble), not
iPhone wallpaper. (Assuming "iPhone Pro dimensions" = 19.5:9 screen ratio cost a
round-trip — he called it "way too thin." That's wallpaper, not a photo.) Confirm:
- Dating-app photo (default): generate portrait
--size 1024x1536; finish
with cohere_photo.py --app safe (3:4 master that survives all three) or per-app
(bumble 3:4, tinder/hinge 4:5). Keep the subject centred WITH MARGIN — apps
re-crop algorithmically.
- Wallpaper (rare):
--app-less, use cohere_photo.py --iphone (19.5:9).
- Landscape scenes (e.g. camel-desert) LOSE most of the frame when cropped to
portrait — generate them portrait from the start, don't crop a landscape down.
See
[[feedback_image_iphone_pro_dimensions]].
The engine (plan d4712f73 — BUILT 2026-06-10): use it, don't hand-drive the stages
The slow human-in-the-loop is gone. One engine (imagegen, src/imagegen/engine.py) runs a whole
recipe from a declarative identity-profile + shot-spec YAML: gpt-image-2 generate
→ face-swap + BiRefNet matte (ONE warm Modal app, all models resident) → local finish/crop
→ VLM auto-QC rank. It surfaces the top 1–2 + flags so you read ~2 finalists, not ~40.
Three faces over the same engine — prefer them over hand-running the stages:
- CLI:
just image-run markus camel-desert (or scripts/tools/image_engine.py run --profile … --shot …)
- MCP (agent-native, preferred for agents):
image_pipeline(profile="markus", shot="camel-desert")
on the imagegen MCP server (registered in ~/Projects/imagegen/.mcp.json) — also
image_swap, image_finish, image_generate, image_engine_status.
- UI (human cull):
just image-ui — Gradio: generate → auto-ranked gallery → you pick;
plus a FREE local re-finish tab (tweak soften/grain/crop with live preview, no regen).
Recipes live in ~/Documents/image-gen/profiles/markus.yaml + shots/*.yaml. Deploy the
Modal app once per code change: just image-deploy. The four model weights coexist in one
container (numpy<2 ABI proven — just image-probe); re-validate if deps change.
2026 SOTA re-check (research-validated, keep this architecture): the generate-then-swap
decoupling is still correct — even Nano Banana Pro / FLUX / InfiniteYou reconstruct a face
like the person (5%+ drift, no persistent identity), so the face-swap stays the identity-
truth step. Restorer A/B MEASURED 2026-06-10 (see backlog doc): GPEN-BFR-1024 is the
default restore — the only model that GAINS identity vs the raw swap (+0.002 bright,
+0.017 dark face) with natural texture; GFPGAN is runner-up (−0.01/−0.018, slight wax);
CodeFormer was REJECTED as default (−0.04, codebook prior pulls toward a generic face —
the memo's prediction didn't survive measurement). Override per profile/shot YAML:
finish.restorer: gfpgan|codeformer|gpen_bfr_1024. Remaining candidates in
`/Projects/imagegen/docs/research/image-engine-upgrade-backlog.md: Nano Banana Pro as generator, DreamID / BFS diffusion head-swap (probed via scripts/modal/bfs_head_swap_probe.py`).
THE core architecture — read first (it's why this works)
Identity is IMPOSED, never GENERATED. gpt-image-2 (and every diffusion model)
cannot reproduce a specific real face from references — it averages refs into a
generic-handsome-model face. Do NOT try to fix likeness with more refs or better
prompts; that's a model ceiling and chasing it wastes rounds + money (it already
cost a full session once). Instead, two stages:
- Scene + body + pose + lighting →
gpt-image-2 (it's great at this).
- Identity → face-swap the real face in with InsightFace
inswapper
(real pixels, faithful, no AI-face look).
Same principle as the photo-cleanup tool (clean_ai_photo.py): never trust
diffusion with the face; paste/swap the real one as a post-step.
Tools (the engine + its three faces; the per-stage code below is "under the hood")
- Engine library:
imagegen (~/Projects/imagegen/src/imagegen/) — run_pipeline(profile, shot) does the whole
recipe and returns ranked variants (run.json sidecar). load_profile/load_shot,
finish_image/FinishSpec, build_prompt, judge_image are also exported.
- Unified Modal app (
image-engine): scripts/modal/image_engine_modal.py — ONE app
with buffalo_l + inswapper + GFPGAN + BiRefNet resident. swap_and_matte (engine path);
one-off CLI: ::swap (batch swap+matte), ::matte (matte only), ::smoke (single).
just image-deploy after edits. Replaces the old face-swap + birefnet apps (3→1).
- Stage CLIs (now thin wrappers over the engine — single source of logic):
scripts/tools/imagegen_scene.py (gen only, → gen.py),
scripts/tools/cohere_photo.py (finish only, → finish.py). Use the engine CLI/MCP for
the full flow; reach for these only for a single isolated stage.
- De-swirl (DEMOTED — last resort):
scripts/modal/clean_ai_photo.py. For gpt-image-2
skin artifacts, reproduce-clean (regen + swap) beats de-swirl; use it ONLY to preserve an
exact existing frame you can't regenerate. NOT in the engine hot path.
- enhance (swap-side face restore, GPEN-BFR-1024 by default) lives in the profile
(
enhance: 0.6); blend 0.3 (subtle) → 0.7 (crisp), Markus rejects plastic so don't
exceed ~0.7. finish.restorer: switches the model (gfpgan | codeformer | gpen_bfr_1024).
- Best matte model (2026): BiRefNet (MIT, hair-grade). SAM 3 (Meta, ICLR 2026) gives
instance/concept masks not alpha mattes — only for text-targeting a specific bg entity.
Workflow
# PRIMARY — one recipe, end to end (gen → swap+matte → finish → auto-QC rank):
just image-deploy # once per engine code change (cached image ⇒ ~2s)
just image-run markus camel-desert # bare names under ~/Documents/image-gen/
# → ranked gallery in generated/engine-runs/<ts>-…/final/, top pick + flags printed.
# MCP: image_pipeline(profile="markus", shot="camel-desert")
# UI: just image-ui (human cull + free local re-finish with live preview)
# A shot is a small YAML diff against the profile (shots/<name>.yaml):
# scene: camel-desert.jpg
# scene_desc: "..."; gaze: away; n: 3
# extra: "subtle Burning Man cues, faint distant mutant vehicles" # de-genericise trick
# target: {app: safe} # bumble|tinder|hinge|safe|wallpaper
# UNDER THE HOOD (only for a single isolated stage):
# gen: uv run --with openai python3 scripts/tools/imagegen_scene.py --scene … --n 4
# swap+matte: uvx --with modal modal run scripts/modal/image_engine_modal.py::swap \
# --targets "a.png,b.png" --source selfie_2026-02-10.jpg --out-dir out/
# finish: uv run python3 scripts/tools/cohere_photo.py --input swap.png --matte m.png \
# --output final.png --downscale 0.92 --soften 0.7 --grain 3 --app safe
Locked recipe (don't re-derive — all dialed across the 2026-06-10 session)
- Face refs: casual selfies, NOT studio portraits (studio → smug/"douchy"):
img_1901, selfie_2026-04-22, selfie_2026-02-10, selfie_2026-03-04.
- Swap source (the face imposed in stage 2):
selfie_2026-02-10 — proven best
this session (likeness 0.92–0.95, recent clear frontal). Default to it.
- Body refs:
photo_2017-01-22 (frame) + 24698343-...png (BULK shape — Markus
wants the fuller, broad-shouldered build, not lean).
- Bulk body-desc (pass via
--body-desc): "fit muscular male-model physique,
broad shoulders, full developed chest, strong defined arms — more muscular than
lean, athletic but natural, never gross/vascular bodybuilder; even tan, smooth
skin, no chest hair, no bulging veins."
- Gaze: usually
away — Markus prefers NOT looking into the camera (candid).
- Veins: a few natural forearm veins are FINE — not extreme, NONE on hands
(biceps a touch ok but unneeded). Do NOT prompt "no veins whatsoever" (over-smooth).
- Background context trick: to de-genericise a scene, add "subtle signs of
[place], faint distant [objects], hazy, not crowded" to
--scene-desc/--extra.
Worked well for Burning Man (mutant vehicles/art on the playa horizon).
- Face blur: inswapper's 128px face reads softer than the body — pass
--enhance 0.6 on the swap (GPEN-1024 restore, the measured default). Tune 0.3–0.7; he rejects
plastic, don't exceed ~0.7.
- Finish = SUBTLE, not bokeh (
[[feedback_image_finish_subtle_not_bokeh.md]]):
light grain + background MOSTLY IN FOCUS. cohere_photo.py --downscale 0.92 --soften 0.7 --grain 3. Heavy matte-DoF bokeh reads "too professional/staged" —
keep --soften ≤1.5. The BiRefNet matte still helps keep the subject crisp + a
GENTLE background touch; just don't crank it.
- Polaroid look (when asked):
cohere_photo.py --polaroid [--polaroid-lift 8 for night] [--border] — instant-film grade (lifted blacks, warm cast, vignette, grain).
- Reproduce-clean > de-swirl for gpt-image-2 skin artifacts (waxy/swirly chest):
REGENERATE the scene via
imagegen_scene.py (anti-artifact prompt) then swap, NOT
clean_ai_photo.py de-swirl. De-swirl is only for preserving an exact existing frame.
- Run n=3–4, cull by face-similarity (+ eye), keep best 1–2.
gpt-image-2 facts (verified 2026-06-09)
- Quality:
low|medium|high|auto — no xhigh. high ≈ $0.21/img + ref input tokens
(~$0.26 with refs).
input_fidelity is auto-high for gpt-image-2 — omit it (passing it can error).
- Person-in-scene compositing is officially supported; anchor realism with natural
lighting, no cinematic grading.
READ BEFORE YOU CLAIM (hook-enforced in ~/Projects/imagegen)
Never present an image as "fixed/clean/final" without verifying it. The verification
stack, in order of authority:
- Flash localizer (primary):
uv run python3 scripts/tools/flash_inspect.py img.png
(in ~/Projects/imagegen) — schema-constrained gemini-3-flash artifact findings with
severity score + box_2d. Run it on every candidate BEFORE presenting; when a defect
appears, BISECT stages with it (gen → swap → finish variants — it isolated the A3
color-match root cause in one pass after agent eyes missed it six times).
- Your own Read (secondary): the exact file being sent, full frame AND native-res
crops of any region you make claims about. Thumbnails/contact sheets hide discs,
washes, seams. Draw mask/patch rectangles on the image before trusting measurements
through them.
A PreToolUse hook in imagegen blocks SendUserFile for never-Read images; honor the
spirit, not just the gate. After fixing a pipeline bug, regenerate canonical-named
outputs in place (stale plain-named files are what the user opens).
Hard line
Never use the face-swap to deceive real people (dating-profile catfishing, fraud,
impersonating someone else). It's for the subject's own generated art only.
1---2name: person-into-scene3description: "Use when: photoreal Markus in scene template, 'put me in this photo', dating/lifestyle/cinematic shots, likeness fix. NOT generic image gen (/gpt-image-2)."4metadata:5 source: https://skillmd.com/skills/markusstrasser/person-into-scene6---78# Person-into-scene image generation910Put a SPECIFIC real person into a scene template, photorealistically, keeping their11actual identity. Built + battle-tested 2026-06-10 (Markus). Assets:12`~/Documents/image-gen/` (`scene-templates/`, `markus-references/`, `generated/`,13`MANIFEST.md`, `HANDOFF.md`). Code lives in `~/Projects/imagegen/` (own repo since142026-06-10, split out of phenome) — run `just`/`uv` commands from there.1516## ASK THE DESTINATION FIRST (before generating anything)1718The output format decides scene orientation AND final crop — get it wrong and you19regenerate. Markus's images are for **dating apps (Hinge/Tinder/Bumble)**, not20iPhone wallpaper. (Assuming "iPhone Pro dimensions" = 19.5:9 screen ratio cost a21round-trip — he called it "way too thin." That's wallpaper, not a photo.) Confirm:22- **Dating-app photo (default):** generate **portrait** `--size 1024x1536`; finish23 with `cohere_photo.py --app safe` (3:4 master that survives all three) or per-app24 (`bumble` 3:4, `tinder`/`hinge` 4:5). Keep the subject centred WITH MARGIN — apps25 re-crop algorithmically.26- **Wallpaper (rare):** `--app`-less, use `cohere_photo.py --iphone` (19.5:9).27- **Landscape scenes** (e.g. camel-desert) LOSE most of the frame when cropped to28 portrait — generate them portrait from the start, don't crop a landscape down.29See `[[feedback_image_iphone_pro_dimensions]]`.3031## The engine (plan d4712f73 — BUILT 2026-06-10): use it, don't hand-drive the stages3233The slow human-in-the-loop is gone. One engine (`imagegen`, `src/imagegen/engine.py`) runs a whole34recipe from a declarative **identity-profile** + **shot-spec** YAML: gpt-image-2 generate35→ face-swap + BiRefNet matte (ONE warm Modal app, all models resident) → local finish/crop36→ **VLM auto-QC rank**. It surfaces the top 1–2 + flags so you read ~2 finalists, not ~40.3738Three faces over the same engine — prefer them over hand-running the stages:39- **CLI:** `just image-run markus camel-desert` (or `scripts/tools/image_engine.py run --profile … --shot …`)40- **MCP (agent-native, preferred for agents):** `image_pipeline(profile="markus", shot="camel-desert")`41 on the `imagegen` MCP server (registered in `~/Projects/imagegen/.mcp.json`) — also42 `image_swap`, `image_finish`, `image_generate`, `image_engine_status`.43- **UI (human cull):** `just image-ui` — Gradio: generate → auto-ranked gallery → you pick;44 plus a FREE local re-finish tab (tweak soften/grain/crop with live preview, no regen).4546Recipes live in `~/Documents/image-gen/profiles/markus.yaml` + `shots/*.yaml`. Deploy the47Modal app once per code change: `just image-deploy`. The four model weights coexist in one48container (numpy<2 ABI proven — `just image-probe`); re-validate if deps change.4950**2026 SOTA re-check (research-validated, keep this architecture):** the generate-then-swap51decoupling is still correct — even Nano Banana Pro / FLUX / InfiniteYou reconstruct a face52*like* the person (~5%+ drift, no persistent identity), so the face-swap stays the identity-53truth step. Restorer A/B MEASURED 2026-06-10 (see backlog doc): **GPEN-BFR-1024 is the54default restore** — the only model that GAINS identity vs the raw swap (+0.002 bright,55+0.017 dark face) with natural texture; GFPGAN is runner-up (−0.01/−0.018, slight wax);56CodeFormer was REJECTED as default (−0.04, codebook prior pulls toward a generic face —57the memo's prediction didn't survive measurement). Override per profile/shot YAML:58`finish.restorer: gfpgan|codeformer|gpen_bfr_1024`. Remaining candidates in59`~/Projects/imagegen/docs/research/image-engine-upgrade-backlog.md`:60Nano Banana Pro as generator, DreamID / BFS diffusion head-swap (probed via61`scripts/modal/bfs_head_swap_probe.py`).6263## THE core architecture — read first (it's why this works)6465**Identity is IMPOSED, never GENERATED.** gpt-image-2 (and every diffusion model)66*cannot* reproduce a specific real face from references — it averages refs into a67generic-handsome-model face. Do NOT try to fix likeness with more refs or better68prompts; that's a model ceiling and chasing it wastes rounds + money (it already69cost a full session once). Instead, two stages:70711. **Scene + body + pose + lighting** → `gpt-image-2` (it's great at this).722. **Identity** → **face-swap the real face in** with InsightFace `inswapper`73 (real pixels, faithful, no AI-face look).7475Same principle as the photo-cleanup tool (`clean_ai_photo.py`): never trust76diffusion with the face; paste/swap the real one as a post-step.7778## Tools (the engine + its three faces; the per-stage code below is "under the hood")7980- **Engine library:** `imagegen` (`~/Projects/imagegen/src/imagegen/`) — `run_pipeline(profile, shot)` does the whole81 recipe and returns ranked variants (`run.json` sidecar). `load_profile`/`load_shot`,82 `finish_image`/`FinishSpec`, `build_prompt`, `judge_image` are also exported.83- **Unified Modal app (`image-engine`):** `scripts/modal/image_engine_modal.py` — ONE app84 with buffalo_l + inswapper + GFPGAN + BiRefNet resident. `swap_and_matte` (engine path);85 one-off CLI: `::swap` (batch swap+matte), `::matte` (matte only), `::smoke` (single).86 `just image-deploy` after edits. Replaces the old face-swap + birefnet apps (3→1).87- **Stage CLIs (now thin wrappers over the engine — single source of logic):**88 `scripts/tools/imagegen_scene.py` (gen only, → `gen.py`),89 `scripts/tools/cohere_photo.py` (finish only, → `finish.py`). Use the engine CLI/MCP for90 the full flow; reach for these only for a single isolated stage.91- **De-swirl (DEMOTED — last resort):** `scripts/modal/clean_ai_photo.py`. For gpt-image-292 skin artifacts, reproduce-clean (regen + swap) beats de-swirl; use it ONLY to preserve an93 exact existing frame you can't regenerate. NOT in the engine hot path.94- **enhance** (swap-side face restore, GPEN-BFR-1024 by default) lives in the profile95 (`enhance: 0.6`); blend 0.3 (subtle) → 0.7 (crisp), Markus rejects plastic so don't96 exceed ~0.7. `finish.restorer:` switches the model (gfpgan | codeformer | gpen_bfr_1024).97- **Best matte model (2026):** BiRefNet (MIT, hair-grade). SAM 3 (Meta, ICLR 2026) gives98 instance/concept masks not alpha mattes — only for text-targeting a specific bg entity.99100## Workflow101102```103# PRIMARY — one recipe, end to end (gen → swap+matte → finish → auto-QC rank):104just image-deploy # once per engine code change (cached image ⇒ ~2s)105just image-run markus camel-desert # bare names under ~/Documents/image-gen/106# → ranked gallery in generated/engine-runs/<ts>-…/final/, top pick + flags printed.107# MCP: image_pipeline(profile="markus", shot="camel-desert")108# UI: just image-ui (human cull + free local re-finish with live preview)109110# A shot is a small YAML diff against the profile (shots/<name>.yaml):111# scene: camel-desert.jpg112# scene_desc: "..."; gaze: away; n: 3113# extra: "subtle Burning Man cues, faint distant mutant vehicles" # de-genericise trick114# target: {app: safe} # bumble|tinder|hinge|safe|wallpaper115116# UNDER THE HOOD (only for a single isolated stage):117# gen: uv run --with openai python3 scripts/tools/imagegen_scene.py --scene … --n 4118# swap+matte: uvx --with modal modal run scripts/modal/image_engine_modal.py::swap \119# --targets "a.png,b.png" --source selfie_2026-02-10.jpg --out-dir out/120# finish: uv run python3 scripts/tools/cohere_photo.py --input swap.png --matte m.png \121# --output final.png --downscale 0.92 --soften 0.7 --grain 3 --app safe122```123124## Locked recipe (don't re-derive — all dialed across the 2026-06-10 session)125126- **Face refs: casual selfies, NOT studio portraits** (studio → smug/"douchy"):127 `img_1901`, `selfie_2026-04-22`, `selfie_2026-02-10`, `selfie_2026-03-04`.128- **Swap source (the face imposed in stage 2):** `selfie_2026-02-10` — proven best129 this session (likeness 0.92–0.95, recent clear frontal). Default to it.130- **Body refs:** `photo_2017-01-22` (frame) + `24698343-...png` (BULK shape — Markus131 wants the fuller, broad-shouldered build, not lean).132- **Bulk body-desc** (pass via `--body-desc`): "fit muscular male-model physique,133 broad shoulders, full developed chest, strong defined arms — more muscular than134 lean, athletic but natural, never gross/vascular bodybuilder; even tan, smooth135 skin, no chest hair, no bulging veins."136- **Gaze: usually `away`** — Markus prefers NOT looking into the camera (candid).137- **Veins:** a few natural forearm veins are FINE — *not extreme*, *NONE on hands*138 (biceps a touch ok but unneeded). Do NOT prompt "no veins whatsoever" (over-smooth).139- **Background context trick:** to de-genericise a scene, add "subtle signs of140 [place], faint distant [objects], hazy, not crowded" to `--scene-desc`/`--extra`.141 Worked well for Burning Man (mutant vehicles/art on the playa horizon).142- **Face blur:** inswapper's 128px face reads softer than the body — pass `--enhance143 0.6` on the swap (GPEN-1024 restore, the measured default). Tune 0.3–0.7; he rejects144 plastic, don't exceed ~0.7.145- **Finish = SUBTLE, not bokeh** (`[[feedback_image_finish_subtle_not_bokeh.md]]`):146 light grain + background MOSTLY IN FOCUS. `cohere_photo.py --downscale 0.92147 --soften 0.7 --grain 3`. Heavy matte-DoF bokeh reads "too professional/staged" —148 keep `--soften ≤1.5`. The BiRefNet matte still helps keep the subject crisp + a149 GENTLE background touch; just don't crank it.150- **Polaroid look** (when asked): `cohere_photo.py --polaroid [--polaroid-lift 8 for151 night] [--border]` — instant-film grade (lifted blacks, warm cast, vignette, grain).152- **Reproduce-clean > de-swirl** for gpt-image-2 skin artifacts (waxy/swirly chest):153 REGENERATE the scene via `imagegen_scene.py` (anti-artifact prompt) then swap, NOT154 `clean_ai_photo.py` de-swirl. De-swirl is only for preserving an exact existing frame.155- **Run n=3–4, cull by face-similarity (+ eye), keep best 1–2.**156157## gpt-image-2 facts (verified 2026-06-09)158159- Quality: `low|medium|high|auto` — **no `xhigh`**. high ≈ $0.21/img + ref input tokens160 (~$0.26 with refs).161- `input_fidelity` is **auto-high** for gpt-image-2 — omit it (passing it can error).162- Person-in-scene compositing is officially supported; anchor realism with natural163 lighting, no cinematic grading.164165## READ BEFORE YOU CLAIM (hook-enforced in ~/Projects/imagegen)166167Never present an image as "fixed/clean/final" without verifying it. The verification168stack, in order of authority:1691. **Flash localizer (primary):** `uv run python3 scripts/tools/flash_inspect.py img.png`170 (in ~/Projects/imagegen) — schema-constrained gemini-3-flash artifact findings with171 severity score + box_2d. Run it on every candidate BEFORE presenting; when a defect172 appears, BISECT stages with it (gen → swap → finish variants — it isolated the A3173 color-match root cause in one pass after agent eyes missed it six times).1742. **Your own Read (secondary):** the exact file being sent, full frame AND native-res175 crops of any region you make claims about. Thumbnails/contact sheets hide discs,176 washes, seams. Draw mask/patch rectangles on the image before trusting measurements177 through them.178A PreToolUse hook in imagegen blocks SendUserFile for never-Read images; honor the179spirit, not just the gate. After fixing a pipeline bug, regenerate canonical-named180outputs in place (stale plain-named files are what the user opens).181182## Hard line183184Never use the face-swap to deceive real people (dating-profile catfishing, fraud,185impersonating someone else). It's for the subject's own generated art only.