Add Model

Add a model to the monorepo

Two phases, five gates. Phase 1 is skippable when a pixified fork already exists. Every phase reads one reference file; read it fully before starting that phase.

Worked examples with every gotcha hit so far: references/example-liteanystereo.md (the run this skill was distilled from) and references/example-fast-foundationstereo.md (the first run through the skill: pickled NAS checkpoint, existing family).

Rules that always bind

• Pixi only. Never pip/uv. Pins copy the monorepo (python 3.12, cuda-version 13.0.*, conda pytorch-gpu, rerun-sdk at the workspace pin, timm, tyro), so the port has no env surprises.
• Do not hand-roll what exists. Before writing any helper, look in simplecv first (camera dataclasses, rescale_intri, Rig/log_rig_static, RerunTyroConfig, rerun_dataloader, Open3DFuser, log_open3d_mesh, rr.VideoCodec, scipy.spatial.transform.Rotation). The port ends with an explicit hand-roll audit (validate.md, gate 4).
• Rerun geometry rules (learned on fisheye rigs and people trackers). (a) DepthImage/EncodedDepthImage unproject only under a distortion-free Pinhole: run the model on the original fisheye frames, then log both the original view and a rectified pinhole twin per camera (cv2.fisheye.initUndistortRectifyMap with R = I so the z axis is unchanged; remap the frame and the predicted depth with the same maps — that remapped z-depth is the twin's pinhole depth) as rig sensors cam_1i (log_rig_static, never children of cam_0i); TSDF runs on the twins. (b) No per-frame Points3D clouds in catalog tools (X-Lens: 60 s at 5 fps = 3.4 GB); the viewer unprojects the twin's depth. (c) Per-id entities (world/people/<id>/…) persist as latest-at after a track ends → rr.Clear(recursive=True) at the id path when the tracker removes it. (d) People render like sam3d-body: posekit.rerun_logging helpers + person_color(id) for 2D, joints as rr.Points3D(keypoint_ids=…, class_ids=0) under an rr.AnnotationContext with the skeleton's connections (the viewer draws the edges), meshes as rr.Mesh3D(vertex_normals=…, albedo_factor=(r, g, b, 0.5)). (e) Sensors that are out of distribution for every model (robocap cam_02/03 eye cameras look at the wearer) are excluded from all runs; name the rig's cameras.
• Python conventions: beartype via PIXI_DEV_MODE claw (never @beartype), PEP 526 annotations everywhere, jaxtyping dtype+shape on every array, TypeAlias not PEP 695, tyro CLIs, 0.0 for float defaults, einops over reshape/permute chains, dataclass field docstrings, thin tools/ shims. (Claude: python-conventions, karpathy-guidelines, tdd skills; Codex has the same rules.)
• Minimal diff. Vendor only the inference subset. No speculative flexibility. Upstream code is untouched in the fork; it is owned (typed, trimmed) only in the 3-typed PR, with an equivalence test.
• Rerun output is validated with pixels, not logs (rerun-viewer-validation skill): screenshots of the demo, the app, and the catalog tool; --rr-config.headless + --rr-config.save in shells without DISPLAY.
• Process hygiene. Never pkill -f/pgrep -f a pattern that appears in the same command line (kills the tool's shell, exit 144); use pkill -x, a saved PID, fuser -k <port>/tcp, or tmux sessions. Never start ad-hoc HTTP servers for artifacts; Gradio apps go through tailscale serve --https.
• Identities. Forks live under the personal GitHub account (GH_TOKEN=$(gh auth token --user pablovela5620) gh …), monorepo PRs use the work account (GH_TOKEN=$(gh auth token --user pablo-rerun) gh pr create … — the active login is often the personal one and gh pr create then fails with "must be a collaborator"). HF mirrors under the personal HF account.
• Durable execution. Anything longer than a minute (env solves, catalog runs, Codex jobs) runs in a named tmux session with python -u, a log file, and a sentinel file on exit; background shells die with the agent session and leave half-written .rrds that look complete. Poll the sentinel, never the process name.
• Follow-ups resume the job. A finished Codex job takes corrections on its own session: codex exec resume -c 'sandbox_mode="danger-full-access"' -c 'model_reasoning_effort="xhigh"' <session-id> - < followup.md (session id in the log header; resume rejects --sandbox). New tmux session, own log and sentinel; the prompt names the branch, the commit message and what to re-run.
• Feedback loop. Every delegated run (Codex or subagent) ends its report with a "skill discrepancies" list: anything unclear, missing, contradictory, or guessed. Fold each item into this skill before the next phase.

Running as Codex (sandbox)

• The sandbox may hide the CUDA virtual package (CONDA_OVERRIDE_CUDA=13.0 for solves) and make the default pixi cache read-only (PIXI_CACHE_DIR=/tmp/<job>-pixi-cache). Neither proves the GPU works: run the GPU gate outside the sandbox or report exactly which command the reviewer must run — never fake numbers.
• Work in a dedicated git worktree (see port.md "Setup"); never edit the reviewer's checkout.

Phase 0 — Scope (write the answers into the fork's NOTES.md later)

1. License. MIT/Apache: mirror freely. NVIDIA Source Code License / research-only: allowed to vendor and mirror with the license text; flag non-commercial in the vendored __init__.py docstring, the fork's NOTES.md, and (own mirrors only) the HF card. Weights can carry a different license than the code (e.g. NVIDIA Open Model Agreement vs Source Code License) — record both. Body models are separately licensed assets: SMPL and SMPL-X are not drop-in for each other (24/6890 vs 55/10475), the smplx transfer_model converts poses not model files, chumpy pickles load without chumpy through a pickle.Unpickler.find_class shim, and the file is hosted privately (SMPL forbids redistribution) with its SHA-256 in the loader test.
2. Weights. Is there an official HF repo? Search the Hub (curl 'https://huggingface.co/api/models?search=<name>') before mirroring anything. Pin the HF revision SHA (curl .../api/models/<repo> → sha) and record the file's SHA-256 in the loader test. Is the file a state_dict or a pickled module (torch.load(..., weights_only=False))? Pickled modules need the unpickler remap recipe in port.md.
3. Inference subset. List the modules the demo actually imports. Everything else (training, export, ONNX/TRT, dataset loaders, visualisation utils) stays out.
4. Hard deps. Custom CUDA/Triton kernels (keep only with a pure-torch fallback + parity test), xformers, flash-attn, open3d. Check what the model package imports vs what the README says. When an upstream pin blocks the monorepo (rfdetr 1.5.0 → transformers<5), check newer releases' requires_dist on PyPI and the attributes the upstream wrapper touches before adding a second env lane (rfdetr ≥1.6 needs transformers>=5.1 and kept RFDETRNano(pretrain_weights=), .model.model/.postprocess/.resolution, optimize_for_inference).
5. Reference number. Does upstream ship an eval script and a per-dataset number? If not, the monorepo's ETH3D playground_1l sample (HF pablovela5620/monoprior-example, stereo/eth3d) is the fallback: record the result as a baseline, not a reproduction. Non-stereo families define their own numbers: metric depth → abs-rel/δ1 and a median-scale-aligned variant (X-Lens on ETH3D: metric abs-rel 0.76 vs scale-aligned 0.15 — the miss was global scale, which EPE alone hides); trackers → counts, per-stage ms, reprojection error of lifted joints, and a replay fixture of the exact model inputs/outputs recorded by the fork's demo. The fork owns the fixture layout (NOTES.md table of keys/dtypes/shapes); the port's loader reads that file and never invents its own contract. Report multi-view timing as ms per frameset with view count and resolution.
6. Upstream entry points. Name the demo script (for demo-upstream), its preprocessing (scaling, padding divisor, normalisation, AMP dtype), the forward signature (iters, test_mode, output shape), and the minimum input size the network accepts (fixes the fast test's tiny pair).
7. Contract. Which existing predictor family does it join (monopriors.models.stereo_depth, relative_depth, normals, ...)? Joining an existing Base*Predictor through its per-model config dataclasses, defaults dict, and Tyro subcommand-union registry is the goal; a new family needs its contract designed first (grill the user).
8. Which PRs. 1-vendor, 2-predictor, 3-typed are always required. 4-app/5-catalog only when the family has no app/catalog tool yet — an existing tool gains the new model through the registry. The app PR is optional when the user scopes it out; renumber (4-catalog).

Probe commands that answer 1–5 in minutes (run them, do not ask the user for these facts):

git clone -q --depth 1 https://github.com/<org>/<repo> /tmp/<model>-probe && cd /tmp/<model>-probe && git rev-parse HEAD
head -40 LICENSE*; grep -n -iE "commercial|research" LICENSE* | head           # code license
grep -rhoE "^(import|from) [a-zA-Z_0-9.]+" --include=*.py <model-dir> | sort | uniq -c | sort -rn   # real deps
curl -s 'https://huggingface.co/api/models?search=<name>' | python3 -c 'import sys,json;[print(m["id"]) for m in json.load(sys.stdin)]'
curl -s https://huggingface.co/api/models/<repo>/tree/main | python3 -c 'import sys,json;[print(f["path"],f.get("size")) for f in json.load(sys.stdin)]'
curl -s https://huggingface.co/api/models/<repo> | python3 -c 'import sys,json;print(json.load(sys.stdin)["sha"])'
python - <<'PY'   # is the .pth a state_dict or a pickled module, and which classes does it reference?
import zipfile,re; z=zipfile.ZipFile('<file>.pth'); d=z.read([n for n in z.namelist() if n.endswith('data.pkl')][0])
print(sorted(set(re.findall(rb'[A-Za-z_][A-Za-z_0-9]*\.[A-Za-z_0-9.]+', d)))[:60])
PY

When forward() reads an attribute the checkpoint lacks, A/B each candidate value against the reference sample's ground truth (FFS normalize: True 0.48 % vs False 45 % bad1) — never pick by reading code alone.

Grill the user on anything ambiguous above (batch the questions, give recommendations) before phase 1.