Port a model into LibreYOLO
This skill is executable. It assumes you've been asked to port a model and your job is to ship it. Reference material (architectural patterns, ABC contracts, validators) is in the back half; the front half is the path you follow to write code.
0. Read this first — orientation
You have an upstream model. Six questions get you pointed at the right scaffold:
- Tier: checkpoint-driven (a
BaseModelfactory family — everything below), or prompt-driven (promptable segmentation, open-vocab detection, VLM)? Prompt-driven models join a sibling factory (LibreSAM/LibreOpenVocab/LibreVLM), not theBaseModelregistry — see §4.1. - Architecture: per-anchor head with NMS (YOLO-grid), set-prediction with Hungarian matching (DETR), or one-to-one head with top-K and no NMS (NMS-free YOLO-grid)?
- Tasks shipped: detect / pose / segment (this skill's templates), or classify / semantic / depth / restore / point / gaze (clone the merged exemplar family — §10 lists them)?
- Backbone source: standard PyTorch, vendored separately-licensed (e.g. DINOv3), or loaded from an optional dependency (e.g. transformers — RF-DETR's DINOv2 backbone)?
- License: code (upstream
LICENSEfile) and weights (HF model card YAML). Both must be permissive (MIT / Apache-2.0 / BSD) for the family to live in core. - Implementation and evidence target: inference-only, training with recorded validation evidence, or training with e2e parity?
The answers route you through §1 (license gate) → §3 (pick scaffold) → §4 (per-family ledger entry to clone) → §5 (commit sequence) → §6 (paste-ready templates). Reference sections fill in the contract details.
Two concrete starts:
- RTMO (one-stage pose) → YOLO-grid pose, Apache-2.0 (MMPose-based). Closest scaffolds: YOLOX for the CSPDarknet backbone, RTMDet (
models/rtmdet/) as the worked example of the mm-series route — mm-series upstream and key remap inmodels/rtmdet/convert.py. (RTMDet itself used to be the hypothetical example here; it has since shipped following exactly this skill.) - A random model on HF → Run §1 (license check) first. If permissive, find architecture in §4 ledger; if no row matches, fall back to §3 decision tree.
1. License check — both code and weights
Two absolute rules. No exceptions, no "just to check", no exploratory read.
- Never read AGPL code — ever. Do not open, clone,
curl,cat, or otherwise view the source of an AGPL-licensed project, not even to "understand the architecture" or "check a shape". Once you determine a repo is AGPL (or dual-licensed with AGPL as the open option), stop and port from a permissive source, the paper, or a clean-room spec instead. Reading it contaminates the port. This holds even when the AGPL repo is the most convenient reference.- Never point at another repo's assets — especially an AGPL repo's — for datasets or weights. Do not reference, hotlink, download-at-runtime, or document a URL into a third-party (and above all an AGPL) repository for dataset files, label files, or checkpoints. Ship dataset config that resolves to a permissively-licensed source we control or that the user supplies; if the only copy lives in an AGPL/non-redistributable repo, treat it as unavailable and surface it as a licensing decision.
If either rule would block the port, that is the correct outcome — raise it, don't route around it.
LibreYOLO core is MIT and stays MIT-compatible. Check both before writing any code.
curl -sL https://raw.githubusercontent.com/<org>/<repo>/<branch>/LICENSE | head -3
| Upstream license | Code in core? | Rehost weights under LibreYOLO/? |
Action |
|---|---|---|---|
| MIT / Apache-2.0 / BSD | ✅ | ✅ | normal integration; ship NOTICE per libreyolo-upload-hf-model skill |
| GPL-3.0 / AGPL-3.0 | ⚠️ plugin only | ❌ | libreyolo-<family> separate package |
| Custom / non-redistributable | case-by-case | ❌ usually | link to upstream CDN like YOLO-NAS / Deci |
For weights, check the HF model card YAML at the top:
license: apache-2.0 # ← permissive
license: gpl-3.0 # ← copyleft
license: cc-by-nc-4.0 # ← non-commercial: still hostable — redistributable is
# the only bar for weights. Ship the license verbatim
# and lead the card with a non-commercial banner
# (SegFormer precedent); users are responsible for
# complying with the weight license.
Already-shipped reference cases:
- D-FINE / DEIM / RF-DETR / EC — Apache-2.0 → clean.
- DEIMv2 — Apache-2.0 family code, but s/m/l/x sizes vendor DINOv3 (Meta's custom non-OSI license). Not "Apache-2.0 clean" wholesale.
- YOLOv9 — MIT, via
MultimediaTechLab/YOLO(Kin-Yiu Wong & Hao-Tang Tsui). NotWongKinYiu/yolov9(GPL-3.0). When upstream has multiple forks, pick the permissive one. - YOLO-NAS — Apache-2.0 code + Deci CDN weights (non-redistributable) → weights linked, not rehosted.
- YOLO-World — GPL-3.0 → flagged plugin-only in #108, never merged.
- L2CS-Net — portable code but redistribution-restricted weights → inference-only family, weights not rehosted, and the family is skiplisted in runtime auto-conversion (
_SKIP_FAMILIESinmodels/autoconvert.py).
If a license problem surfaces mid-port, never "fix" it by rewriting or renaming the offending code to obscure where it came from. Surface it as a licensing decision — plugin, link-out, or drop.
Vendored sub-components with separate licenses
If your family vendors a separately-licensed architectural sub-component (DEIMv2 vendors DINOv3 for s/m/l/x backbones; Meta's custom non-OSI license), the family's own license isn't sufficient. Required:
LICENSE.mdnext to the vendored code (e.g.libreyolo/models/deimv2/engine/backbone/dinov3/LICENSE.md).- A NOTICE entry citing the sub-component, its upstream, and its license.
- A docstring annotation on the family's model class explaining which size variants depend on the differently-licensed sub-component.
- A clear rule for the user: do the constraints of the sub-component flow through to the produced weights? Document the answer.
Don't quietly bundle a non-OSI sub-component without these three artifacts.
2. Pick the implementation and evidence target
Pick one and document scope explicitly:
- Inference-only. No trainer is wired, or
train()rejects the task. The user canmodel.predict(...)but notmodel.train(...). YOLO-NAS pose, L2CS, and CLIP are examples. - Training with limited evidence. Trainer exists and
train()accepts the task. Record the exact completed checks and the missing convergence evidence; do not turn a user acknowledgement into the capability contract. - Training with RF1 evidence.
test_rf1_trainingpasses; row inMODEL_CATALOG; recipe gaps documented in the family docstring. YOLOX, YOLOv9, YOLO-NAS detect, D-FINE, DEIM, DEIMv2, RT-DETR, RF-DETR sit here, plus the classify families (MobileNetV4 / ConvNeXt / EfficientNetV2 / ResNet) on the shared classify path.
Inference-only is a legitimate ship state. Don't gate the port on a working trainer.
Which target to pick belongs to the PRD: libreyolo-write-model-prd section 4
checks the task/data shape and the available fine-tuning recipe. If the PRD is
silent on implementation scope or evidence, apply the same checks and record
the answer in the PR description.
3. Pick your scaffold
Use this decision tree to pick the family you'll clone as your starting point.
3.1 By architecture
| Upstream looks like | Closest scaffold | What to keep | What to swap |
|---|---|---|---|
| YOLO-grid + SimOTA / TaskAlignedAssigner | YOLOX (models/yolox/, 6 files, ~288 LoC model.py) |
mosaic+mixup pipeline, label assignment idiom, BGR preprocess if YOLOX-style | head architecture, backbone, optionally loss |
| YOLO-grid + GFL/DFL + ESNet/light backbone | PicoDet (models/picodet/, 6 files, ~272 LoC model.py) |
shared GFL head pattern, RGB+ImageNet norm, training implementation and evidence notes | backbone parser, neck, possibly DFL reg_max |
| YOLO-grid + ELAN/RepNCSPELAN, complex conversion | YOLOv9 (models/yolo9/, 7 files) |
aux-head-skip pattern, heavy structural converter | head modules |
| NMS-free YOLO-grid (one-to-one head, top-K) | YOLOv9-E2E (models/yolo9_e2e/, ~271 LoC model.py) |
NMS-free postprocess pattern, parent-child sibling pattern | inherit from your detect parent rather than BaseModel |
| DETR — light, mostly metadata-wrap conversion | D-FINE (models/dfine/, 16 files) |
encoder/decoder/MS-deform-attn modules, FlatCosineScheduler, deploy() wrapper | matcher, loss weights |
| DETR — sibling of an existing port (different loss/matcher) | DEIM (models/deim/) |
inherit from D-FINE, override only what differs | loss, matcher |
| DETR — vendors a separately-licensed backbone | DEIMv2 (models/deimv2/) |
safetensors handling, DINOv3 vendoring + LICENSE.md | backbone-specific code |
| DETR — multi-task (detect + pose + segment) | EC (models/ec/, ~431 LoC model.py) |
task-dispatch in _init_model/_postprocess, is_pose_state_dict/is_seg_state_dict discriminators |
architecture |
| DETR — multi-task native port, backbone from an optional dep | RF-DETR (models/rfdetr/, 18 files) |
DINOv2-via-transformers backbone + explicit state-dict transfer (backbone.py), lazy registration, family-local config |
task heads, matcher |
| YOLO-grid pretrained on ImageNet w/ heavy backbone | YOLO-NAS (models/yolonas/) |
in-process EMA unwrap (no separate converter), CDN-not-HF weight URL | head + backbone |
| Classify-only backbone (CNN / hybrid) | MobileNetV4 / ConvNeXt / EfficientNetV2 / ResNet (models/<family>/) |
shared BaseTrainer classify path, {"probs": ...} postprocess, timm-parity test pattern |
the backbone itself |
| Semantic segmentation | PIDNet (CNN) / EoMT (ViT) | SemanticValidator wiring, semantic mask contract |
architecture |
| Dense prediction (depth / restore) | Depth Anything V2 (models/depth_anything/) / NAFNet (models/nafnet/) |
native-resolution inference path, dedicated validator | architecture |
| Zero-shot / text-conditioned | CLIP (models/clip/) for classify; the models/openvocab/ tier for detection (§4.1) |
text-tower handling, set_classes API |
towers |
3.2 By task scope
| You're shipping | Multi-task pattern to follow |
|---|---|
| detect only | any single-task family above; declare SUPPORTED_TASKS = ("detect",) |
| detect + pose | YOLO-NAS (asymmetric sizes — pose has n, detect doesn't) |
| detect + segment | RF-DETR (native multi-task port; multi-task training via RFDETR_SEG_TRAINERS) |
| detect + pose + segment | EC (3-way dispatch in _init_model, three converters via --task flag) |
| classify only | MobileNetV4 / ConvNeXt / EfficientNetV2 / ResNet — shared BaseTrainer classify path |
| semantic / depth / restore / point / gaze only | PIDNet + EoMT / Depth Anything V2 / NAFNet / FOMO / L2CS — single-task BaseModel families, each with a dedicated validator |
| open-vocab detect, promptable seg, VLM | not BaseModel families — sibling factories LibreOpenVocab / LibreSAM / LibreVLM (§4.1) |
3.3 By non-PyTorch upstream
- Paddle / TensorFlow / safetensors: PicoDet ports a community PyTorch reimplementation (Bo's). For Paddle direct, you'd write a heavier conversion script that handles framework-specific buffer cleanup. For safetensors, see DEIMv2's
weights/convert_deimv2_weights.py:19-43— dispatch onPath(input).suffix == ".safetensors", build a fresh native model,safetensors.torch.load_model(model, path, strict=True).
4. Per-family ledger
Dense reference. Find the family closest to your port and clone its directory as your starting scaffold. Each row tells you what's already solved.
| Family | Pattern | Sizes | Tasks | Training state (per task) | Files | Weight conversion | Notable |
|---|---|---|---|---|---|---|---|
| YOLOX | YOLO-grid (NMS) | n/t/s/m/l/x | detect | trainable; RF1 covered | 6 | none (in-process unwrap) | BGR 0–255 inference; mosaic+mixup; closest to upstream of any family |
| YOLOv9 | YOLO-grid (NMS) | t/s/m/c | detect | trainable; RF1 covered | 7 | heavy structural | RGB 0–1; aux head dropped; xyxy normalized targets in loss; from-scratch recipe gap (3 param groups at same LR, no backbone-LR split) |
| YOLOv9-E2E | NMS-free YOLO-grid | t/s/m/c | detect | trainable; RF1 covered | 5 | reuses YOLOv9 converter (different model_family at wrap) |
Inherits from LibreYOLO9. Postprocess does top-K only — del iou_thres. In the _is_nms_free_family() allowlist (was a miss for a while — since fixed) |
| YOLO-NAS | YOLO-grid (NMS) | n*/s/m/l (n* pose-only) | detect, pose | detect trainable; pose has no trainer | 7 | none (in-process EMA unwrap) | Weights from Deci CDN (license, not LibreYOLO HF). Only family with asymmetric-per-task INPUT_SIZES |
| PicoDet | YOLO-grid (NMS) | s/m/l | detect | trainable; RF1 gap documented | 6 | light structural (mmcv key remap + EMA drop) | GFL+DFL loss; ESNet backbone; per-size INPUT_SIZES (320/416/640) |
| D-FINE | DETR | n/s/m/l/x | detect | trainable; RF1 covered | 16 | metadata-wrap (~50 LoC) | Per-group LR via lr_mult in _setup_optimizer + _train_epoch override; FlatCosineScheduler (added by this family); min_lr_ratio=0.05; backbone-LR multiplier 0.5× |
| DEIM | DETR (D-FINE sibling) | n/s/m/l/x | detect | trainable; RF1 covered | small | metadata-wrap | Architecturally identical to D-FINE; min_lr_ratio=0.5; tie-break in factory (models/__init__.py, search "Ambiguous D-FINE/DEIM") |
| DEIMv2 | DETR | atto/femto/pico/n/s/m/l/x | detect | trainable; RF1 covered | small | metadata-wrap + safetensors handling | s/m/l/x vendor DINOv3 (separate license). Per-size min_lr_ratio overrides (n = 1.0, others 0.5). Documents warmup_iters epoch-override scaling |
| EC | DETR | s/m/l/x | detect, pose, segment | trainable; evidence recorded per task | many | metadata-wrap multi-task (--task flag) |
3-way _init_model dispatch; is_pose_state_dict / is_seg_state_dict discriminators; pose forces nc=1, names={0:"person"}; mask head DETR-style (transformer cross-attention, not YOLO proto) |
| RT-DETR | DETR | r18/r34/r50/r50m/r101/l/x | detect | trainable; RF1 covered | medium | light structural | Multi-char size codes (r50m vs r50) — overrides detect_size_from_filename with length-descending sort. Per-group LR via lr_ratio + _scale_lr override — better template than D-FINE for new DETR ports. Pretrained-backbone-download fix prototyped in bf16a2b but not in current code |
| RF-DETR | DETR (native port) | n/s/m/l | detect, segment, pose, obb | trainers implemented per task; evidence recorded separately | 18 | bespoke autoconvert recognizer (needs the full checkpoint for size detection + COCO class remap) | No longer a PyPI-package wrapper — full native port. DINOv2 backbone loaded via transformers AutoBackbone with an explicit state-dict transfer (models/rfdetr/backbone.py) because from_pretrained silently no-ops on the windowed subclass (landmine #37). Lazy-registered behind the transformers dep (_ensure_rfdetr). Family-local config (models/rfdetr/config.py). Multi-task training via compile-time RFDETR_TRAINERS vs RFDETR_SEG_TRAINERS selection |
File-count signal: 6–7 files = single-task YOLO-grid scaffold. 16+ = a DETR family with non-trivial loss + matcher + transforms.
Families added since the table above was first written
Compact rows — clone these directly for the newer archetypes:
| Family | Pattern | Tasks | Training | Notable |
|---|---|---|---|---|
RTMDet (models/rtmdet/) |
YOLO-grid (NMS) | detect | trainable; evidence gaps documented | CSPNeXt; mm-series upstream; runtime auto-convert remap in models/rtmdet/convert.py |
| RT-DETRv2 | DETR (child of RT-DETR) | detect | inherits RT-DETR | registered after v1 so metadata-less checkpoints default to v1 |
| RT-DETRv4 | DETR (child of D-FINE) | detect | trainable | must register before D-FINE (more-specific can_load); own models/rtdetrv4/convert.py |
| FOMO | boxless point head | point | trainable | point task; PointValidator; per-size imgsz from family CONFIGS |
| L2CS | CNN gaze | gaze | inference-only | redistribution-restricted weights → autoconvert skiplist, no HF rehost |
| Depth Anything V2 | ViT dense | depth | inference-only | DepthValidator; check per-size weight licenses (upstream b/l are CC-BY-NC) |
| NAFNet | encoder-decoder restore | restore | wired (see docstring) | native-resolution path (no letterbox); RestoreValidator |
| EoMT | ViT semantic | semantic | see docstring | SemanticValidator; unique query/mask keys make can_load trivial |
| PIDNet | CNN semantic | semantic | see docstring | 1024-px default input; fusion-key can_load |
| MobileNetV4 / ConvNeXt / EfficientNetV2 / ResNet | classify backbone | classify | trainable | shared BaseTrainer classify path; parity vs timm is bit-identical (max_abs_diff == 0) |
| CLIP | dual-tower zero-shot | classify | inference-only | pure-torch towers (no open_clip at runtime); set_classes API; clip_validator.py |
| DINOv2 | ViT | semantic, classify | via RFDETRConfig |
lazy-registered together with RF-DETR (transformers dep) |
YOLO9-P2 (models/yolo9_p2/) |
YOLO-grid (child of YOLO9) | detect | trainable | stride-4 P2 head for small objects; sizes t/s; the WEIGHT_VARIANTS = ("visdrone",) precedent for dataset-variant weight suffixes |
Darknet lineage: YOLO2/3/4 (models/darknet/ + thin models/yolo{2,3,4}/) |
anchor-grid CNN | detect | inference-only | one shared DarknetFamily (cfg parser + blocks + anchor decode) serves all three; public-domain upstream; converter weights/convert_darknet_weights.py, parity via weights/parity_darknet.py |
Darknet lineage: YOLO1 (models/darknet/ + thin models/yolo1/) |
dense FC-head CNN | detect | inference-only | shares the DarknetFamily engine but the FC head ([connected]/[local]/[detection]) does NOT fit the anchor decode: v1-specific decode_detection (7x7x30, VOC-20, fixed 448, square-stretch preprocess). OpenCV can't oracle it, so faithfulness = byte-exact reader + dog/bicycle/car golden. b weights on HF; tiny t weights lost upstream (code-ready, BYO .weights) |
YOLO7 (models/yolo7/) |
anchor-grid CNN | detect | training evidence recorded in the RF1 skip map + infer | MIT upstream (same repo as the YOLO9 source); own v7.yaml + net; converter weights/convert_yolo7_weights.py, parity via weights/parity_yolo7.py; training via SimOTA loss (loss.py) adapted from Apache-2.0 YOLOX |
BiRefNet (models/birefnet/) |
Swin v1 + bilateral-reference decoder | matte | inference-only (v1) | MIT upstream; matte task (ADR 0010); MatteValidator (MAE + S-measure); family-local Swin v1 (original lineage, NOT the timm models/swin/ tower, see NOTICE); ASPP deformable conv exports to ONNX DeformConv (opset 19) via a registered symbolic; converter weights/convert_birefnet_weights.py, parity via weights/parity_birefnet.py (max_abs_diff == 0) |
Faster R-CNN (models/faster_rcnn/) |
two-stage RPN + RoIAlign + class-wise NMS | detect | inference-only | native port from torchvision v0.26.0 (BSD-3-Clause), sizes n/s/m/l; official state keys load strictly and all four variants have exact eager parity. COCO-91 sparse ids map to contiguous COCO-80. ONNX is batch-1/fixed-square and emits final already-NMSed boxes/scores/labels. Weight mirrors carry BSD-3-Clause on a disclosed implied basis plus torchvision's pretrained-model caveat; weights/upload_faster_rcnn_hf.py enforces the five-file contract |
FCN (models/fcn/) |
dilated ResNet + primary/auxiliary FCN heads | semantic | inference-only | native port from torchvision v0.26.0 (BSD-3-Clause), sizes r50/r101 at 520; this is not the original VGG FCN-8s graph. Both heads have exact eager parity. Semantic predict/val and ONNX/TorchScript/OpenVINO/TensorRT are validated. Weight mirrors carry BSD-3-Clause on a disclosed implied basis plus torchvision's pretrained-model caveat; weights/upload_fcn_hf.py enforces the five-file contract |
4.1 Sibling factories (not BaseModel families)
Prompt-driven models do not join the checkpoint-driven BaseModel registry.
They live in sibling factories with their own contracts:
LibreOpenVocab(models/openvocab/) — text-conditioned open-vocabulary detectors returning standard detectionResults: Grounding DINO (models/grounding_dino/), OWLv2 (models/owlv2/), and OMDet-Turbo, which runs throughtransformerswith no vendored model source. Shared towers live inmodels/bert/(text) andmodels/swin/(vision).LibreSAM— promptable segmentation: SAM-1 and SAM-2 (models/sam/), MobileSAM (models/mobilesam/).LibreVLM(models/vlm/) — vision-language models.
If your port is prompt-driven, clone one of these factories instead of a
BaseModel family. The license gate (§1), parity discipline (§12), and HF-upload rules
(commit 10) still apply in full; the BaseModel ABC contract (§8) does not.
5. Walking the port — commit sequence
Each numbered commit is a self-contained PR-able unit. Don't combine. Don't write the trainer before the inference parity test passes (see §12).
Commit 1 — Skeleton + factory recognition
Create libreyolo/models/<family>/{__init__.py, model.py, nn.py, utils.py}
using template §6.1. Implement:
Libre<FAMILY>class withFAMILY,FILENAME_PREFIX,INPUT_SIZES,SUPPORTED_TASKS,DEFAULT_TASK,TRAIN_CONFIG = None(we'll wire later).can_load(state_dict)— pick a key unique to your architecture. Never"backbone"or"weight".detect_size(state_dict)— infer size from a shape signature.detect_nb_classes(state_dict)— read nc from the head._init_model,_get_available_layers,_forward,_postprocess(stub OK),_preprocess(use shared letterbox).
Add from .<family>.model import Libre<FAMILY> to libreyolo/models/__init__.py in the right registry order (most distinctive markers first). Add Libre<FAMILY> to libreyolo/__init__.py exports.
Enroll the family in the model registry: add one "<family>": "<group>"
line to MODEL_GROUPS in libreyolo/models/registry.py. Group semantics are
in docs/nomenclature.md ("Model groups"). Take the coverage group from the
PRD; it mirrors the implemented surface and never decides capability.
tests/unit/test_model_registry.py fails until the family is enrolled.
Verify: python -c "from libreyolo import Libre<FAMILY>; m = Libre<FAMILY>(size='s'); print(m.task, m.family)" runs.
Commit 2 — nn.py + forward smoke
Port the model architecture into libreyolo/models/<family>/nn.py. Mirror
upstream attribute names where possible — it makes conversion a metadata-wrap
(see Commit 3) and keeps the parity diff readable.
Verify: model builds at all sizes, model(torch.zeros(1, 3, 640, 640)) returns the expected tensor shape (or dict for DETR).
Commit 3 — Conversion script + atomic write
Create weights/convert_<family>_weights.py using template §6.4 (single-task)
or §6.5 (multi-task). Use wrap_libreyolo_checkpoint(...) with task,
supported_tasks, default_task populated even for single-task ports —
free disambiguation later.
Write atomically: tmp = output.with_suffix(".tmp"); save_checkpoint(wrapped, tmp); tmp.rename(output). Print a missing/unexpected-key diff after loading the wrapped dict into a fresh model.
Verify: python weights/convert_<family>_weights.py upstream/x.pth weights/Libre<FAMILY>s.pt --size s runs and produces a file. LibreYOLO("weights/Libre<FAMILY>s.pt") loads without errors.
Commit 3b — Runtime auto-conversion recognizer
libreyolo/models/autoconvert.py makes LibreYOLO("<upstream_file>.pth")
work directly: the factory unwraps the common upstream layouts (ema.module /
ema / net / model / state_dict / plain), asks every registered family
via BaseModel.convert_upstream_state_dict whether it recognizes the tensors,
wraps the winner in v1.0 metadata (size / task / nc read from the tensors), and
writes <source>-<Prefix><size>[-task].pt beside the source.
- The
BaseModeldefault (models/base/model.py::convert_upstream_state_dict) claims any layout yourcan_loadaccepts — if upstream keys already match your native port, auto-conversion is free and this commit is just tests. - If upstream naming differs, add
models/<family>/convert.pywithis_upstream_state_dict()+convert_upstream()(shared by the runtime auto-converter and your offlineweights/convert_<family>_weights.py), and override the hook on the model class (template §6.10). Precedents:models/{yolo9,rtmdet,rtdetr,rtdetrv4,picodet}/convert.py. - Special cases: non-redistributable weights are skiplisted
(
_SKIP_FAMILIES— L2CS); RF-DETR registers a bespoke recognizer inautoconvert.pybecause it needs the full checkpoint (size detection + COCO class remap), not just the tensor dict. - Arbitration when several families claim one file: subclass beats base, then
registry order (= import order in
libreyolo/models/__init__.py). The filename is consulted only for the D-FINE/DEIM tensor tie.
Verify: LibreYOLO("<downloaded upstream>.pth") loads your family end to
end, AND your recognizer returns None for every sibling family's upstream
checkpoints — a greedy recognizer steals other families' files (landmine #36).
Commit 4 — Inference parity proof
This is the gate. Before any postprocess work, prove the model produces identical outputs to upstream on identical inputs.
Use template §6.8 (cross-load script). Save it as
tests/unit/test_<family>_parity.py or as a one-off under
weights/. The test:
- Imports the upstream model class and yours side by side.
- Loads upstream weights into both.
- Feeds the same
torch.zeros(...)(or a fixed seed) through both. - Asserts
max_abs_diff == 0ineval()mode on layers present in both.
Verify: parity script passes for every size of your family. Don't proceed until it does.
Commit 5 — _postprocess returning the canonical dict
Implement _postprocess in models/<family>/utils.py. Return:
{"boxes": (N, 4), "scores": (N,), "classes": (N,)} # detect
# + "masks": (N, H, W) for segment
# + "keypoints": (N, K, 3) for pose (xy + visibility)
If your model emits keypoints as (N, K, 2), append a column of ones for
visibility — Keypoints.has_visible requires column 3
(libreyolo/postprocess/ec.py::postprocess_pose for the precedent).
Verify: tests/unit/test_<family>_postprocess.py smoke-tests shape contracts on synthetic input.
Commit 6 — End-to-end inference
LibreYOLO("Libre<FAMILY>s.pt").predict("test.jpg") returns a Results
object with the right slots populated, and Results._select(idx) slices
boxes ↔ masks ↔ keypoints in lockstep. Drawing dispatches on slot presence
(if result.keypoints is not None: draw_keypoints(...)), no task field
needed.
Add <Family>ValPreprocessor to libreyolo/validation/preprocessors.py (or
inherit existing). Set uses_letterbox, custom_normalization,
wants_unresized_image properties to match your training transform.
Verify: model.predict("test.jpg") runs end-to-end. model.val(data="coco128.yaml") runs.
Verify in libreyolo ui (required): the UI dropdown is built from
get_all_cli_names(), so a registered family appears automatically, but the
result card only works if save=True writes an annotated image: the server
calls model(img, conf=..., save=True) and shows the saved file plus a
task-aware summary from _summarize_result in libreyolo/ui/server.py.
Launch libreyolo ui, drop a test image, run your smallest size, and confirm:
(1) inference does not require extra kwargs the UI cannot supply (prompts,
auxiliary detectors); (2) an annotated image renders, not the unchanged
source; (3) the summary is task-appropriate, not a fallback "0 objects".
If the port introduces a new Results slot, extend _summarize_result (and
Results.plot if the slot has no drawing path) in the same PR.
Commit 7 — ONNX export
For YOLO-grid: 1 output "output", opset 13. For DETR: 2 outputs
["pred_logits", "pred_boxes"], opset ≥ 16 (deformable attention needs
grid_sample).
DETR families: add yourself to BaseBackend._is_nms_free_family() at
libreyolo/backends/base.py:211. NMS-free YOLO-grid (yolo9_e2e-style)
must also be added or exported backends will wrongly apply NMS. NCNN
doesn't work for DETR — block early with NotImplementedError.
Verify: model.export(format="onnx") produces a working graph.
OnnxBackend("Libre<FAMILY>s.onnx").predict(...) matches PyTorch outputs.
Commit 8 — Trainer (skip if shipping inference-only)
Decide implementation scope and evidence (§2). Inference-only: TRAIN_CONFIG = None, override
train() to raise NotImplementedError, and stop here. Otherwise:
Create models/<family>/trainer.py using template §6.6 (YOLO-grid) or §6.7
(DETR with per-group LR via lr_ratio). Append <Family>Config(TrainConfig)
to libreyolo/training/config.py (or use a family-local config — RF-DETR /
RT-DETR / YOLOv9-E2E do this).
For limited-evidence training, expose the same train() interface and document
the completed checks and known limits separately from the callable API.
For non-detect tasks: override best_metric_key explicitly. Default is
"metrics/mAP50-95" (bbox). Set "metrics/mAP50-95(M)" for segment,
"metrics/keypoints_mAP50-95" for pose.
Verify: model.train(data="coco128.yaml", epochs=3) runs and produces a
new .pt.
Commit 9 — e2e catalog row
Append your family's sizes to MODEL_CATALOG in tests/e2e/conftest.py:417. Run:
pytest tests/e2e/test_val_coco128.py -k <family>
pytest tests/e2e/test_rf1_training.py -k <family>
DETR families: skip the last_loss < first_loss assertion in
test_rf1_training (DETR loss is too noisy on small datasets — RF-DETR and
D-FINE both exempt themselves).
Verify: both e2e tests pass for every size.
Commit 10 — HuggingFace upload
Only for redistributable weights — re-check §1. Permissive-licensed
weights are rehosted under LibreYOLO/; anything else follows the YOLO-NAS
link-out or L2CS no-rehost path and this commit becomes documentation of
where the user gets the weights instead.
Follow the libreyolo-upload-hf-model skill. Cross-check your filename
against the whitelist there before uploading. The 5-file contract:
.gitattributes, README.md, LICENSE, NOTICE, Libre<FAMILY><size>[-<task>].pt.
Multi-task families: one HF repo per task variant
(LibreYOLO/Libre<FAMILY>s for detect, LibreYOLO/Libre<FAMILY>s-pose for
pose, LibreYOLO/Libre<FAMILY>s-seg for segment).
Verify: on a fresh machine / cleared cache (no weights/Libre<FAMILY>s.pt
staged), LibreYOLO("Libre<FAMILY>s.pt") auto-downloads from the new HF repo
and loads. There is no from_pretrained API; the bare canonical filename is
the download trigger (BaseModel.get_download_url() builds the
huggingface.co/LibreYOLO/<name>/resolve/main/<name>.pt URL).
6. Paste-ready templates
Copy these, fill in the # TODO markers. Mirror upstream attribute names
in nn.py whenever possible — it makes conversion a metadata-wrap.
6.1 Family directory layout
libreyolo/models/<family>/
├── __init__.py # exports Libre<FAMILY>
├── model.py # BaseModel subclass (template 6.2 or 6.3)
├── nn.py # the actual nn.Module — port-specific, mirror upstream names
├── utils.py # postprocess, preprocess_numpy
├── loss.py # only if you ship a trainer
└── trainer.py # BaseTrainer subclass (template 6.6 or 6.7) — optional
__init__.py:
"""Libre<FAMILY> family: <one-line architecture summary>."""
from .model import Libre<FAMILY>
__all__ = ["Libre<FAMILY>"]
6.2 model.py — single-task detect-only
"""Libre<FAMILY>: BaseModel subclass wiring <FAMILY> into the LibreYOLO factory."""
from __future__ import annotations
from typing import Any, Optional
import torch
import torch.nn as nn
from ...training.config import <FAMILY>Config # delete if TRAIN_CONFIG=None
from ...validation.preprocessors import <FAMILY>ValPreprocessor
from ..base import BaseModel
from .nn import Libre<FAMILY>Model
from .utils import postprocess as _postprocess
from .utils import preprocess_numpy as _preprocess_numpy
class Libre<FAMILY>(BaseModel):
"""<one-line architecture summary>."""
FAMILY = "<family>" # TODO: short lower-case ID
FILENAME_PREFIX = "Libre<FAMILY>" # TODO: PascalCase, no -det suffix
INPUT_SIZES = {"s": 640, "m": 640, "l": 640} # TODO: confirm with upstream
SUPPORTED_TASKS = ("detect",)
DEFAULT_TASK = "detect"
TRAIN_CONFIG = <FAMILY>Config # or None for inference-only
val_preprocessor_class = <FAMILY>ValPreprocessor
@classmethod
def can_load(cls, weights_dict: dict) -> bool:
# TODO: pick a key UNIQUE to your architecture.
# Never "backbone", "weight", or anything generic.
# Cross-check against every existing family's state dict in tests.
return any("<unique_token>" in k for k in weights_dict)
@classmethod
def detect_size(cls, weights_dict: dict) -> Optional[str]:
# TODO: read a shape signature that disambiguates sizes.
# Common pattern: a head conv's out_channels.
key = "<head.cls_pred.weight>"
if key not in weights_dict:
return None
out_ch = int(weights_dict[key].shape[0])
return {<channels>: "s", ...}.get(out_ch)
@classmethod
def detect_nb_classes(cls, weights_dict: dict) -> Optional[int]:
# TODO: read nc from a head weight shape, accounting for reg channels.
key = "<head.cls_pred.weight>"
if key not in weights_dict:
return None
return int(weights_dict[key].shape[0]) # adjust if reg channels are mixed in
def _init_model(self) -> nn.Module:
return Libre<FAMILY>Model(size=self.size, nc=self.nb_classes)
def _get_available_layers(self) -> dict[str, nn.Module]:
return {
"backbone": self.model.backbone,
"neck": self.model.neck,
"head": self.model.head,
}
@staticmethod
def _get_preprocess_numpy():
return _preprocess_numpy
def _preprocess(self, image, *, color_format=None, **kwargs):
# TODO: pick the shared letterbox helper or implement a family-local one.
# See models/yolox/model.py or models/picodet/model.py for precedents.
...
def _forward(self, x: torch.Tensor) -> Any:
return self.model(x)
def _postprocess(self, raw, conf_thres: float, iou_thres: float, **kwargs):
return _postprocess(raw, conf_thres, iou_thres, **kwargs)
6.3 model.py — multi-task (detect + pose [+ segment])
Add per-task class vars and dispatch. EC is the 3-task reference (models/ec/model.py).
class Libre<FAMILY>(BaseModel):
FAMILY = "<family>"
FILENAME_PREFIX = "Libre<FAMILY>"
SUPPORTED_TASKS = ("detect", "pose", "segment") # subset as needed
DEFAULT_TASK = "detect"
INPUT_SIZES = {"s": 640, "m": 640, "l": 640}
POSE_INPUT_SIZES = {"s": 640, "m": 640, "l": 640} # may add asymmetric sizes (e.g. "n")
SEG_INPUT_SIZES = {"s": 640, "m": 640, "l": 640}
TASK_INPUT_SIZES = {
"detect": INPUT_SIZES,
"pose": POSE_INPUT_SIZES,
"segment": SEG_INPUT_SIZES,
}
# State-dict discriminators — cross-test against sibling families!
_POSE_HEAD_KEY = "<unique pose key>"
_SEG_HEAD_KEY = "<unique seg key>"
@classmethod
def is_pose_state_dict(cls, sd) -> bool:
return cls._POSE_HEAD_KEY in sd
@classmethod
def is_seg_state_dict(cls, sd) -> bool:
return any(k.startswith(cls._SEG_HEAD_KEY) for k in sd)
@classmethod
def detect_task_from_state_dict(cls, sd) -> Optional[str]:
if cls.is_pose_state_dict(sd): return "pose"
if cls.is_seg_state_dict(sd): return "segment"
return None # falls back to detect via DEFAULT_TASK
def _init_model(self) -> nn.Module:
if self.task == "pose": return Libre<FAMILY>PoseModel(size=self.size, nc=1)
if self.task == "segment": return Libre<FAMILY>SegModel(size=self.size, nc=self.nb_classes)
return Libre<FAMILY>Model(size=self.size, nc=self.nb_classes)
def _postprocess(self, raw, conf_thres, iou_thres, **kwargs):
if self.task == "pose": return _postprocess_pose(raw, conf_thres, iou_thres, **kwargs)
if self.task == "segment": return _postprocess_seg(raw, conf_thres, iou_thres, **kwargs)
return _postprocess(raw, conf_thres, iou_thres, **kwargs)
6.4 Conversion script — single-task metadata-wrap
"""Convert upstream <FAMILY> weights to LibreYOLO format.
Usage:
python weights/convert_<family>_weights.py upstream/<file>.pth weights/Libre<FAMILY>s.pt --size s
"""
from __future__ import annotations
import argparse
from pathlib import Path
from _conversion_utils import (
add_repo_root_to_path,
extract_state_dict,
load_checkpoint,
save_checkpoint,
wrap_libreyolo_checkpoint,
)
def convert(input_path: str, output_path: str, size: str, nc: int = 80) -> None:
raw = load_checkpoint(input_path)
state_dict = extract_state_dict(raw, prefer_ema=True)
print(f"Extracted {len(state_dict)} parameter entries from {input_path}")
# OPTIONAL — strip an upstream prefix if upstream wraps:
# state_dict = strip_state_dict_prefix(state_dict, "model.")
# OPTIONAL — print missing/unexpected diff after dry-load into your model:
# add_repo_root_to_path(); from libreyolo import Libre<FAMILY>
# m = Libre<FAMILY>(size=size); res = m.model.load_state_dict(state_dict, strict=False)
# print("missing:", res.missing_keys); print("unexpected:", res.unexpected_keys)
wrapped = wrap_libreyolo_checkpoint(
state_dict,
model_family="<family>",
size=size,
nc=nc,
task="detect",
supported_tasks=("detect",),
default_task="detect",
)
out = Path(output_path)
tmp = out.with_suffix(out.suffix + ".tmp")
save_checkpoint(wrapped, tmp)
tmp.rename(out) # atomic
print(f"Wrote {out}")
if __name__ == "__main__":
p = argparse.ArgumentParser()
p.add_argument("input")
p.add_argument("output")
p.add_argument("--size", required=True, choices=["s", "m", "l"])
p.add_argument("--nc", type=int, default=80)
args = p.parse_args()
convert(args.input, args.output, args.size, args.nc)
6.5 Conversion script — multi-task with --task flag
EC pattern. One CLI invocation per task variant.
"""Convert upstream <FAMILY> weights to LibreYOLO format (multi-task)."""
from __future__ import annotations
import argparse
from pathlib import Path
from _conversion_utils import (
extract_state_dict, load_checkpoint, save_checkpoint, wrap_libreyolo_checkpoint,
)
_SUPPORTED = ("detect", "pose", "segment")
_DEFAULT = "detect"
def convert(input_path, output_path, size, task, nc):
raw = load_checkpoint(input_path)
state_dict = extract_state_dict(raw, prefer_ema=True)
# Per-task override
…(truncated)