Run Rf100vl Benchmark

Run the RF100-VL benchmark

RF100-VL: 100 real-world detection datasets from Roboflow Universe (164,149 images, 564 classes, 7 domains), paper arXiv 2505.20612 (NeurIPS 2025 Datasets and Benchmarks). Each dataset ships fixed train/valid/test splits in COCO JSON. There is no official runner; the harness referenced below is the runner.

Exact install/launch commands: harness docs/rf100vl-operator-runbook.md. Vast rental mechanics (2FA/TOTP, launch, destroy): skills/launch-serverless-gpu-job (Vast section) or the personal vast-launch skill. This skill is the protocol plus the operational knowledge those two do not cover.

Protocol (locked decisions)

Never report toolkit-native trapezoidal mAP; it inflates up to 2.7 AP on RF100-VL versus pycocotools (paper, App B). LibreYOLO validation is pycocotools-based already; the 500 cap is the opt-in eval_max_det kwarg (model.val(data=..., split="test", eval_max_det=500)). Defaults for normal users are unchanged (AP at maxDets 100) and test-locked.

Scoring can cost more than training, and it scales with box density, not image count. RF100-VL's gwhd2021 has 43.6 boxes/image against a median of 4.9; at conf 0.001 and maxDets 500, stock pycocotools spent 496 s/epoch scoring against 112 s training — 4.4×, and 6.9 of that dataset's 9.6 h. faster-coco-eval removes it: verified across all 100 test splits at max_det=500, 1381/1400 metric values bit-identical, max deviation 2.22e-16 (one float64 ULP), headline mean AP delta exactly 0, wall 131.4 s → 8.4 s (15.6× overall, 56× on gwhd2021). The dense outliers to expect are gwhd2021, recode-waste, uavdet-small. Keep stock pycocotools as the reference implementation for audit; do not silently swap backends without recording which one ran.

Validation frequency is protocol-mandated every epoch (gdino val_interval=1; rt-detr / d-fine / lw-detr call evaluate() unconditionally, lw-detr twice). The knob that exists is validation COST (image cache, hoisted validator, graphed val forward), not frequency. LibreYOLO's own TrainConfig.eval_interval defaults to 10 — a short smoke never validates unless you pass eval_interval=1.

Where the work happens

Repo LibreYOLO/vision-analysis-benchmark (harness branch rf100vl-harness).

# per-dataset fine-tuning: one worker per lane, subprocess children,
# atomic status files, resume, timeouts, dense-dataset OOM fallback
va-bench rf100vl-train --data-dir ./rf100-vl --weights-root ./rf100vl-weights \
  --gpus 0,1,2,3,4,5,6,7 --jobs-per-gpu 3

# per-dataset test-split eval
va-bench rf100vl --all --data-dir ./rf100-vl --weights-root ./rf100vl-weights

# both plus checks and rendering as ONE resumable command
va-bench rf100vl-campaign --model yolov9t --data-dir ./rf100-vl \
  --weights-root ./rf100vl-weights --gpus 0,1,2,3,4,5,6,7 --jobs-per-gpu 3

--jobs-per-gpu is the biggest throughput lever, but throughput is not the makespan. Size it against the LONGEST dataset, not just VRAM. Measured 2026-08: going 8 → 16 lanes on one box made every lane 1.74× slower (uniform across all 16 datasets, so it is contention, not any one model) and therefore lengthened the serial long pole, finishing later overall. When one dataset sets the makespan, packing deeper is strictly harmful. Pack when total work dominates; do not pack when a single dataset does.

Pin OMP_NUM_THREADS (and MKL_/OPENBLAS_) whenever you pack. Left unset, torch takes 64 intra-op + 128 inter-op threads per process: 16 lanes gave ~124 threads per dataloader worker, 11,890 threads on 128 cores, and 67.6% system time with 6% idle — two thirds of the CPU spent on scheduling, not work. Pinning to 2 took system time to 7.4%.

CUDA graphs improve packing because they remove host launch contention between lanes sharing a GPU. They also add persistent static buffers, so measure VRAM with graphs in the configuration you will actually run.

Supporting verbs: rf100vl-preflight, rf100vl-dash (pass --data-dir or queued datasets show no image counts and the ETA is size-blind), rf100vl-report.

• Dataset, fast path. Pull LibreYOLO/rf100-vl: 100 per-dataset tars + lock files. Prefer max_workers=32 and stay logged in (HF_TOKEN) for the authenticated rate limit. Do not cargo-cult huggingface_hub[hf_transfer] / HF_HUB_ENABLE_HF_TRANSFER=1 on hub 1.x.
• Dataset, canonical path. --download wraps the rf100vl pip package (ROBOFLOW_API_KEY); use it to rebuild/verify the HF copy. Licensing is in va_bench/data/rf100vl_licenses.json.
• Weights: best.pt at <weights_root>/<dataset>/<weight_file>.
• A capability guard aborts on builds without eval_max_det/amp_dtype. cuda_graph / cache are reported, not required — missing them runs the protocol correctly, just slower.
• Flag lists: harness README / --help win over this skill.

Decisions BEFORE dataset one (one-way doors)

The run signature hashes the recipe. Any recipe change (cuda_graph, cache, epochs, imgsz, …) means a fresh campaign — banked checkpoints under the old recipe cannot be resumed. Enabling cuda_graph mid-campaign once orphaned 973 banked epochs.

1. Recipe final? (va_bench/recipes/rf100vl/*.json)
2. LibreYOLO commit pinned and actually installed? pip install --upgrade on a git URL silently no-ops when the version string is unchanged and reports success. Use pip install -q --force-reinstall --no-deps "git+...", then prove the installed TrainConfig has the fields the recipe sets.
3. Vast TOTP seed saved (~/.config/vastai/vast_totp_seed)? A 2FA session expiring mid-campaign once forced stopping a billing box through an unverifiable path.
4. Shakedown done on the current stack? (below)

Compute: vast.ai (default)

Account setup, 2FA, launch, exec, pull, destroy: follow skills/launch-serverless-gpu-job (Vast section). RF100-VL specifics:

• Image, tested end to end: vastai/pytorch:2.11.0-cu128-cuda-12.9-mini-py312-2026-06-15. Vast's own image takes their key-injection path; plain pytorch/pytorch sometimes leaves sshd rejecting keys. cu128 is required for 5090 (sm_120). Interpreter: /venv/main/bin/python, not bare python.
• Accept or reject in 60 seconds with harness deploy/vast/accept-box.sh (GPUs, kernels, matmul, HF, PyPI, disk ≥ 120 GB free). Destroy duds immediately. loading is essentially unbilled (meter starts at running); a wedged 15-minute pull destroyed after measured $0.02. An older estimate of "$1 for the host search" overstated this by ~10x — the real cost is operator attention. Never nurse a doubtful host because destroying it "feels wasteful."
• Disk allocate ~120 GB without image cache, ~250 GB with cache: "disk". The offer filter "machine has ≥ 300 GB free" and the --disk you rent are different; disk bills on allocated GB. One campaign needs roughly 70 GB (image + pip + 49 GB dataset + weights); disk-cache .npy sidecars add ~108 GB at 416px and ~159 GB at 640px (see the disk projection above — 640px does not fit 250 GB without purging as you go).
• Workload is CPU / host-bound, priced GPU-centric. Measured [pre-cache]: 46 ms GPU vs 507 ms CPU per step; 8 cores/lane still ~94% CPU-saturated. pick_box.py's old MIN_CORES_PER_LANE = 3.0 was far too low. Weight core count and single-core clock heavily (a 2.6 GHz EPYC 7K62 lost to a consumer Ryzen). Size cores for epoch 1 (cache fill + all lanes cold), not the steady-state average. Prefer high-clock CPUs even if the GPU $/hr is slightly higher.
• Primary target: one 8× RTX 5090 interruptible box with strong CPU; pack with --jobs-per-gpu after a VRAM/lane re-measure. Fallback: several 1–4× 5090/4090 boxes.
• Offer filter: verified, reliability > 0.99, ≥ 8 vCPU per lane after packing, machine disk ≥ 300 GB free (so 120–250 allocated fits), ≥ 500 Mbps down, download < 0.01 USD/GB, host driver CUDA 12.8+, distinct egress IP from known-bad NATs. Bid 20–30% above minimum.
• Interruptible: outbid pauses the box; disk persists and bills; destroy deletes. Harness resumes at dataset (status files) and epoch (last.pt) level. Sync weights/results off-box at milestones; always pull before destroy.
• Local-first: one dataset, then rf20vl pilot, then rent. When the stack changes (new LibreYOLO commit, recipe, packing, image), shake out on ONE cheap GPU first (~$1).
• Select on $/GPU-hour, not $/hour. Measured 2026-08: 8× RTX 5060 Ti (16 GB, 128 cores) at $0.79/hr is $0.099/GPU-h; an 8× 5090 box is $0.467/GPU-h for perhaps 2.5–3× the throughput. On a host-bound workload the cheap many-core box usually wins. Check VRAM separately (see sizing above) — 16 GB is the constraint that actually rules boxes out.

Running several boxes at once

Each box is fully independent: the harness has no cross-box coordination, so "multi-node" is really N single-node campaigns plus your attention. That makes the failure modes operational, not algorithmic.

• One chain script per box (wait → campaign → upload → stop), launched detached with setsid nohup ... < /dev/null &. Put a gate at the top that asserts the stack invariant you care about and aborts before spending a night — e.g. build the model and assert BN eps, or assert the eval backend.
• One monitor per box, and make its filter cover failure, not just progress: a monitor that only greps the happy path is silent through a crashloop, which looks identical to "still running".
• Crash recovery is genuinely good and does not need you: dataset+epoch level resume, atomic status files, per-dataset timeouts, OOM → grad-accum fallback. ~300 dataset-runs completed with 0 failures unattended.
• Silent wrongness is what needs you, and it scales badly with box count. Every serious problem in the 2026-08 campaigns was silent: wrong BN eps, uploads skipping checkpoints, disk projections, a bad ETA. Before fanning out, make sure each of those has a loud check; otherwise N boxes produce N results you must hand-verify anyway.

What healthy looks like (do not "fix" this)

• Launch/host-bound: [pre-cache] healthy meant GPUs at 9–35% util and ~170 W of 575 W with everything fine. Low GPU numbers are the signature of this workload, not a fault. The runbook once claimed 60–100% util as healthy; that reading burned an expensive detour.
• Re-baselined 2026-08 with disk cache + faster eval: dataload fell to 0.2 ms of a 350 ms step, so the pre-cache "507 ms CPU per step" figure no longer describes the loop. Util now runs ~39% mean exclusive and 74–94% when packed. Both bands are healthy; read util next to power, and treat a change against your own shakedown as the signal rather than any absolute number.
• nvidia-smi util is time-with-a-kernel-resident on the CARD, not die occupancy per job. With 3 lanes sharing a GPU the row describes the card.
• Datasets are heterogeneous: 92 to 8,791 train images, 0.31 to 12+ MP. Epoch times span ~40×. Longest-first scheduling; one dataset sets the makespan.
• Epoch 1 costs 1.3–2.1× a steady epoch [pre-cache]; with post-resize cache the ratio widens (epoch 1 fills the cache). Every ETA in the first hour is garbage — do not make money decisions off it. A "16.4 h" ETA was once an artifact of averaging epoch 1 into a two-epoch mean.

Sizing a box: measure VRAM, never predict it

Do not derive VRAM from parameter count or GFLOPs. Measured 2026-08 at 640px, batch 16: yolox-s (8.97M) = 6.2 GB/lane, yolov9s (~9M) = 10.8 GB/lane — same size, 1.7× apart. Activation memory, cuDNN workspace autotuning, allocator behaviour and CUDA-graph static buffers all dominate, and none of them follow parameter count.

Measure instead. It costs about a minute per model:

libreyolo profile run <data.yaml> --weights <W> --size <s> \
  --imgsz 640 --batch 16 --steps 20 --device 0   # prints "peak VRAM"

Run the probe matrix on the target GPU before committing a campaign, and take ~20% headroom. The registry's params_millions/gflops fields are 0.0 for most non-flagship specs, so they cannot stand in for this.

Being wrong is survivable but not free: the harness falls back to grad-accum (keeping effective batch 16) and restarts that dataset from epoch 0, so a bad guess costs throughput and wasted epochs, not correctness.

Disk: project it, do not eyeball it

Two campaigns nearly died on this. Post-resize cache measured ~953 KB/image at 640px and ~660 KB at 416px, so:

cache_GB ≈ total_images × bytes_per_image
RF100-VL (163,151 images): ~159 GB at 640, ~108 GB at 416

That is far above the "105 GB of sidecars" this skill used to quote. Add the dataset (49 GB) and checkpoints (measured 108 MB each for a ~9M-param model, so ~30-56 GB across 100 datasets) and 640px does not fit a 250 GB disk.

Fix: purge each dataset's cache when it completes. The cache is reused across that dataset's 100 epochs and is dead weight afterwards, so purging on state == done bounds it to the active working set (~54 GB) instead of the full ~159 GB. Never delete anything but *.r<W>x<H>.npy.

When something looks slow: profile, do not theorize

libreyolo profile answers "where is the time going" in under a minute. An hour of py-spy (blocked by container caps), ps aggregation, and log-timestamp forensics produced a confidently wrong answer that libreyolo profile run corrected in 52 seconds. The campaign should print a Profile hint next to the Monitor hint; if it does not, still run:

libreyolo profile run ...      # one profiled training epoch
libreyolo profile phases ...   # train / validation / save split
libreyolo profile what-if ...  # projected gain from a fix

High self-CPU on an op like aten::max with near-zero self-CUDA usually marks a GPU→CPU sync absorbing device wait, not a CPU bottleneck in that op. Total self-CUDA ≪ total self-CPU ⇒ launch-bound (CUDA graphs).

First check that the GPU counters attached at all. On a rented box where CUPTI fails to load, the profiler still prints a full report, but every device-side number is zero and the VERDICT line is then an artifact rather than a measurement. Measured 2026-08 on an 8x RTX 5060 Ti box:

REAL step 362.3 ms = dataload 0.1 ms + compute 362.3 ms  ->  44.2 img/s
GPU util 0%  (0 ms GPU-busy / 362 ms step)  |  0 kernels/step @ ~0us
>> VERDICT: HOST/LAUNCH-BOUND - GPU only ~0% busy

That verdict was wrong: enabling CUDA graphs on the same model and batch changed the epoch time by nothing. 0 kernels/step is the tell. When you see it, ignore the verdict and the "self-CUDA vs self-CPU" rule above, both of which are derived from the dead counters, and read the per-phase wall table instead, which is host-side timing and stays valid:

phase         gpu_ms  wall_ms  kernels     ops
to_device        0.0      5.9        0       8
forward          0.0    127.9        0    4562
backward         0.0    222.9        0    5093
optimizer        0.0      6.3        0    2800

CUDA graphs and cache: prefer them, then verify both

Keep cuda_graph: true and cache: "disk" as the default recipe protocol. Both are perf-only and covered by the recipe hash. Two things to verify before trusting either, because each has cost a campaign.

Capture is gated by family and by commit. At an older pinned LibreYOLO only yolo9 and rfdetr implemented the training capture hook; every other family took the eager fallback, so setting the flag did nothing but log a warning. Current dev covers far more families (docs/training_cuda_graphs.md holds the table and the per-family speedups). Check that table for the commit you actually pinned, not for dev, and confirm the run log says cuda_graph: captured training forward/backward at input shape (...). Absent that line, the flag is decorative.

The speedup is batch-dependent, and can be zero. The documented gains are measured at batch 8, where launch overhead is a large share of the step. At the protocol's batch 16 the same model can show nothing: yolonas-s measured 83.4 / 78.9 s per epoch eager against 83.6 / 78.4 s graphed, with capture confirmed in the log. yolo9 still gains at batch 16. Measure before assuming.

Capture inflates VRAM, and libreyolo profile run will not show it because it does not capture graphs. yolo9-m at 640 and batch 16 profiled at 15721 MB, which looks survivable on a 16 GB card. The real graphed run OOM'd on all 100 datasets, with the failure naming the cause:

13.61 GiB allocated in private pools (e.g., CUDA Graphs)

Disabling capture on that box dropped the same run to 15207 MB and it trained fine at full protocol batch. So size the lane with a 2-epoch smoke campaign, which exercises the real path including capture, EMA and validation:

va-bench rf100vl-campaign --model <M> --recipe <R> \
  --datasets <one-dataset> --gpus 0 --jobs-per-gpu 1 --smoke-epochs 2

For a NMS-family model that cannot hold capture on the target card, dropping cuda_graph is the cheap deviation: the harness treats graph replay as outside the run signature and bit-identical on loss. Dropping physical batch is the expensive one, because it changes BatchNorm statistics even though nbs keeps the effective batch at 16.

Contribute the speedup back

The campaign is a load test of LibreYOLO training, so treat what the profiler finds as a LibreYOLO bug list rather than a campaign workaround list. The scoring cost above is the model: it was found by profiling a campaign, fixed in the library as a default, and now every user benefits. When a fix belongs in the library, send it there and pin the campaign to the commit that carries it, rather than carrying a private patch on the box.

Shakedown (before any full campaign)

1. Rent one cheap high-clock-CPU GPU.
2. Install exactly as the campaign would (--force-reinstall --no-deps); prove recipe fields exist on the installed TrainConfig.
3. Run 2–3 representative datasets (tiny ~100 imgs, huge ~8k, large-image 12 MP class), 10–15 epochs, eval_interval=1, plus one full completion.
4. Kill-and-resume between checkpoints (not mid-write).
5. Record VRAM/lane, cores/lane at epoch 1 and steady state, epoch-1 ratio, validation share, healthy util band. Size the real box from those numbers.

Stopping and resuming

• Ctrl-C mid-checkpoint can corrupt last.pt and poison resume for that dataset. Stop gracefully: tmux send-keys -t bench C-c, wait for the orchestrator to exit, then vastai stop instance if keeping the box.
• A stopped box bills disk only; restart needs those exact GPUs still free. Decide stop-vs-destroy from disk $/day, re-stage cost (~35 min, nearly free bandwidth), and whether banked checkpoints are usable (recipe change ⇒ they are not).
• Stopped is not restartable on demand. 2026-08 a restart returned resources_unavailable, state change queued and intended_status reverted to stopped; the GPUs came back only hours later. Treat a stop as "possibly forever". Sync every box-local file you cannot regenerate BEFORE stopping — recipes, chain scripts, any hand-written config. sync-artifacts does not upload the recipe, and a run whose recipe_sha256 points at a file that exists nowhere is not reproducible.
• Instance ops need a live 2FA session (~7 days) and ssh<N>.vast.ai DNS can fail transiently. Keep the dashboard tunnel up: an established tunnel survives a DNS outage and is the only health channel left when ssh will not resolve.
• Destroy ends spend. Always pull/sync first.

Decisions to take per campaign

1. Model list and sizes. Flagships deep (yolo9 and rfdetr); other families start with their one or two smallest variants.
2. Pilot first: rf20vl end to end. Graduate to 100 only if all 20 complete, kill-and-resume reproduces within noise, and AP ordering is sane.
3. Budget and waves: price from live offers (deploy/vast/pick_box.py), publish after flagships land, append families as they finish.
4. Recipe deviations: live in the recipe JSON, hash-recorded, disclosed. No silent knobs. Finalize before dataset one.

Artifacts and publishing

Keep, per model: per-dataset eval JSONs, raw predictions (COCO detections), per-run stats.json (recipe sha, best epoch, seed, dataset version, wall time), the recipe JSON, the dataset version lock, and the final submission JSON. Predictions are on by default for publishable runs.

• Upload with va-bench sync-artifacts to a LibreYOLO HF dataset repo (create the repo by hand; fine-grained write token scoped to that repo). Pass --eval-dir so predictions are collected. Stock-pycocotools rescore of a saved dump reproduced harness AP to four decimals with no harness code imported.
• sync-artifacts is silent about what it does not upload. It ships the paths it knows (eval/, stats/, submissions/, and runs/*/weights when the weights root has the .runs layout) and ignores everything else, while printing a cheerful uploaded N, skipped 0. It does not ship the recipe JSON, nor arbitrary files placed in the weights root, nor checkpoints from a flat <root>/<dataset>/<file> layout. Verify after every publish by listing the repo (HfApi().list_repo_files), not by reading the uploader's log — and note the tree API paginates at 1000 entries, so a raw count there silently truncates.
• Leaderboard submission: vision-analysis via submit-benchmark-results (see benchmark-on-visionanalysis).
• Never hand-edit result JSONs. Regenerate them.

Always check: does selection agree with the test score?

The cheapest possible guard, and the one whose absence cost the most. Compare each run's valid_mAP50_95 (from stats.json) against the published test AP. For a healthy run they track within noise; a large one-sided gap means the two paths disagree about the model, not that the model generalises badly.

2026-08: yolox-nano reported valid 0.5663 and test 0.1620 on ball while yolox-tiny agreed to 0.002 on the same datasets. The cause was BatchNorm eps (models trained at 1e-5, evaluated at 1e-3, ruinous for the depthwise nano only), fixed in libreyolo #700. A whole 100-dataset campaign published a headline 0.3601 that should have been 0.4853 and nothing flagged it.

The corollary: a 16-point gap between adjacent sizes of one family is not a result, it is a bug report. Adjacent sizes land ~3 points apart.

Traps

• Stock pycocotools summarize() breaks with a non-default maxDets list (headline AP becomes -1). Never call it with a modified list.
• Package-cleaned data only. Raw Universe exports keep the dummy class and score near zero.
• Dataset named -grccs: --datasets=-grccs (space form is eaten by argparse).
• Never resume after changing physical batch, accumulation, or recipe fields covered by the run signature (including cuda_graph / cache).
• Dense datasets can OOM rfdetr; harness falls back to grad-accum and restarts that dataset from epoch 0. Expected.
• ec family: AdamW, no mosaic (mosaic triggers a degenerate-box assertion).
• Largest datasets take hours; raise timeouts for slow families, never remove.
• Differences under 0.5 mAP on the 100-dataset mean are noise. Replicate.
• ROBOFLOW_API_KEY and vast credentials: env/local config only. Never commit.

Published numbers to beat (fully supervised, AP 0.50:0.95)

RF-DETR N/S/M/L/XL/2XL: 57.7 / 60.2 / 61.2 / 62.2 / 62.9 / 63.2. LW-DETR T to X: 57.1 to 62.1. D-FINE N to X: 58.2 to 62.2. YOLO11 N to X: 55.3 to 56.5. YOLO26 N to X: 52.0 to 60.0. Sources: the rf-detr repo README (develop) and paper v4 tables. No YOLO9-lineage numbers exist anywhere yet.