Profile & speed up LibreYOLO (training + inference)
libreyolo profile answers one question fast: where does the time go, and is
the GPU starved? It is a measurement tool built to be driven by an agent
in a loop — every subcommand takes --json, and both entry points write a
self-contained profile.json you can copy and compare.
The profiler only measures. It never changes your model. You read the
verdict, you change one thing (a config value or code), then you re-measure.
Don't wait for the tool to auto-tune — that's your job.
The CLI is self-describing. Never guess a flag — run
libreyolo profile --help and libreyolo profile <sub> --help. Run
libreyolo profile get <profile.json> with no field to list the exact metric
names for the installed version.
The loop
run|infer → summary → (kernels|ops|phases) → change ONE thing → compare → repeat
Both run (training) and infer (inference) emit the same profile.json, so
summary/get/phases/kernels/ops/compare/what-if work on either.
A. Training throughput — profile run
libreyolo profile run <data> --weights LibreYOLO9t.pt --size t --repeat 3 --json
<data> is a dataset yaml/name (e.g. coco128). --batch -1 auto-fits ~70% VRAM.profile run stops right after the profile window (it passes
profile_then_stop=True). In contrast, model.train(..., profile=True)
profiles the window and then keeps training, so it is safe on a real or
resumed run.- Always
--repeat 3+ — a single run lies when launch-bound; --repeat
gives mean ± stdev and is what makes a later compare significant. The
aggregate is runs/profile/profile_repeat.json — use that path (not a
per-trial prof_0/profile.json sibling).
Read the verdict, then pull the matching lever (then re-measure):
| Verdict |
Meaning |
You change |
| dataloader |
GPU waits on input (~≥20% of step) |
more workers, cache='ram'/'disk', lighter aug, larger batch |
| host / launch |
GPU only partly busy — fed too slowly |
larger micro-batch (amortizes launches), fewer per-step .item()/syncs, CUDA graphs, op fusion |
| compute |
GPU saturated (~≥80% busy) |
already GPU-bound — AMP/bf16, or accept it / change model size |
| memory-pressure |
VRAM thrash (util reads >100%) |
lower batch, reduce activation memory — util/img/s here are unreliable |
Highest-value, lowest-effort win: host/launch-bound → raise the micro-batch.
(yolov9t on an RTX 5070 Ti → micro-batch 36 = +142% img/s, painless.)
B. Inference latency — profile infer
libreyolo profile infer <image-or-dir> --weights LibreYOLO9t.pt --size t --batch 1 --json
- Source defaults to a bundled sample image, so
libreyolo profile infer --weights X
works standalone. --half uses fp16 (CUDA). --runs/--warmup size the window.
- Reports latency p50/p90/p99, throughput (img/s at
--batch), and a
stage split: preprocess / forward / postprocess(NMS).
--conf, --iou, --max-det change how much NMS work happens — the knobs to
vary when NMS-bound.
| Verdict |
Meaning |
You change |
| nms / postprocess |
NMS dominates the step |
lower --max-det, higher --conf, fewer classes, or an end-to-end (NMS-free) model |
| preprocess |
CPU resize/letterbox dominates |
larger --batch, faster decode, GPU preprocessing |
| compute |
forward dominates (GPU busy, or CPU) |
bigger batch, --half, a smaller model, or export (ONNX/TensorRT) |
| host / launch |
forward dominates but GPU underused (CUDA) |
larger --batch, or a bigger model to fill the GPU |
Drill (shared, when the verdict isn't enough)
libreyolo profile phases <profile.json> # stage/phase split
libreyolo profile kernels <profile.json> --top 20 # worst GPU kernels (--category --grep --tensorcore --phase)
libreyolo profile ops <profile.json> --top 20 # aten ops by host time
libreyolo profile get <profile.json> latency_p50_ms # one metric, tight loops
Low tensorcore_pct on a compute-bound fp32 run → AMP/bf16/--half helps. Big
layout / copy share → channels_last. Many tiny elementwise kernels → fusion.
Change one thing, then prove it helped
Change one lever, re-run with the same --repeat/--runs, then:
libreyolo profile compare <before.json> <after.json>
compare reports the delta and a significance call. "single run — use
--repeat N" means the delta is noise; repeat before trusting it.
Gotchas
- The tool won't change your config — you do, then re-measure.
- Always
--repeat for training (one run is noise, esp. launch-bound).
- Compare the aggregate, not a per-trial sibling.
- Under memory-pressure, trust the verdict, not raw util (thrash inflates it).
- This measures speed, not accuracy — validate mAP with
libreyolo val after
changing batch/LR/aug.
1---2name: libreyolo-profiling3description: Diagnose and fix slow LibreYOLO with the `libreyolo profile` CLI — both TRAINING throughput (`profile run`) and INFERENCE latency (`profile infer`). Use whenever training feels slow, GPU utilization is low, images/sec is disappointing, inference/predict latency is too high, a run is dataloader- / host-launch- / NMS- / preprocess-bound, or someone wants to optimize step time, batch size, throughput, or p50/p90/p99 latency. Teaches the profile → diagnose → change → compare loop an agent runs to push speed to the max. This is for SPEED, not accuracy (mAP).4---56# Profile & speed up LibreYOLO (training + inference)78`libreyolo profile` answers one question fast: **where does the time go, and is9the GPU starved?** It is a **measurement tool** built to be driven by an agent10in a loop — every subcommand takes `--json`, and both entry points write a11self-contained `profile.json` you can copy and `compare`.1213**The profiler only measures. It never changes your model.** You read the14verdict, *you* change one thing (a config value or code), then you re-measure.15Don't wait for the tool to auto-tune — that's your job.1617The CLI is **self-describing**. Never guess a flag — run18`libreyolo profile --help` and `libreyolo profile <sub> --help`. Run19`libreyolo profile get <profile.json>` with no field to list the exact metric20names for the installed version.2122## The loop2324```25run|infer → summary → (kernels|ops|phases) → change ONE thing → compare → repeat26```2728Both `run` (training) and `infer` (inference) emit the same `profile.json`, so29`summary/get/phases/kernels/ops/compare/what-if` work on either.3031## A. Training throughput — `profile run`3233```bash34libreyolo profile run <data> --weights LibreYOLO9t.pt --size t --repeat 3 --json35```3637- `<data>` is a dataset yaml/name (e.g. `coco128`). `--batch -1` auto-fits ~70% VRAM.38- `profile run` stops right after the profile window (it passes39 `profile_then_stop=True`). In contrast, `model.train(..., profile=True)`40 profiles the window and then **keeps training**, so it is safe on a real or41 resumed run.42- **Always `--repeat 3`+** — a single run *lies* when launch-bound; `--repeat`43 gives mean ± stdev and is what makes a later `compare` significant. The44 aggregate is `runs/profile/profile_repeat.json` — use **that** path (not a45 per-trial `prof_0/profile.json` sibling).4647Read the **verdict**, then pull the matching lever (then re-measure):4849| Verdict | Meaning | You change |50|---|---|---|51| **dataloader** | GPU waits on input (~≥20% of step) | more `workers`, `cache='ram'`/`'disk'`, lighter aug, larger batch |52| **host / launch** | GPU only partly busy — fed too slowly | **larger micro-batch** (amortizes launches), fewer per-step `.item()`/syncs, CUDA graphs, op fusion |53| **compute** | GPU saturated (~≥80% busy) | already GPU-bound — AMP/bf16, or accept it / change model size |54| **memory-pressure** | VRAM thrash (util reads >100%) | **lower batch**, reduce activation memory — util/img/s here are *unreliable* |5556Highest-value, lowest-effort win: **host/launch-bound → raise the micro-batch.**57(yolov9t on an RTX 5070 Ti → micro-batch 36 = **+142% img/s**, painless.)5859## B. Inference latency — `profile infer`6061```bash62libreyolo profile infer <image-or-dir> --weights LibreYOLO9t.pt --size t --batch 1 --json63```6465- Source defaults to a bundled sample image, so `libreyolo profile infer --weights X`66 works standalone. `--half` uses fp16 (CUDA). `--runs`/`--warmup` size the window.67- Reports **latency p50/p90/p99**, **throughput** (img/s at `--batch`), and a68 **stage split**: preprocess / forward / postprocess(**NMS**).69- `--conf`, `--iou`, `--max-det` change how much NMS work happens — the knobs to70 vary when NMS-bound.7172| Verdict | Meaning | You change |73|---|---|---|74| **nms / postprocess** | NMS dominates the step | lower `--max-det`, higher `--conf`, fewer `classes`, or an end-to-end (NMS-free) model |75| **preprocess** | CPU resize/letterbox dominates | larger `--batch`, faster decode, GPU preprocessing |76| **compute** | forward dominates (GPU busy, or CPU) | bigger batch, `--half`, a smaller model, or export (ONNX/TensorRT) |77| **host / launch** | forward dominates but GPU underused (CUDA) | larger `--batch`, or a bigger model to fill the GPU |7879## Drill (shared, when the verdict isn't enough)8081```bash82libreyolo profile phases <profile.json> # stage/phase split83libreyolo profile kernels <profile.json> --top 20 # worst GPU kernels (--category --grep --tensorcore --phase)84libreyolo profile ops <profile.json> --top 20 # aten ops by host time85libreyolo profile get <profile.json> latency_p50_ms # one metric, tight loops86```8788Low `tensorcore_pct` on a compute-bound fp32 run → AMP/bf16/`--half` helps. Big89`layout / copy` share → channels_last. Many tiny elementwise kernels → fusion.9091## Change one thing, then prove it helped9293Change **one** lever, re-run with the same `--repeat`/`--runs`, then:9495```bash96libreyolo profile compare <before.json> <after.json>97```9899`compare` reports the delta **and a significance call**. "single run — use100--repeat N" means the delta is noise; repeat before trusting it.101102## Gotchas103104- **The tool won't change your config — you do, then re-measure.**105- **Always `--repeat` for training** (one run is noise, esp. launch-bound).106- **Compare the aggregate**, not a per-trial sibling.107- **Under memory-pressure, trust the verdict, not raw util** (thrash inflates it).108- **This measures speed, not accuracy** — validate mAP with `libreyolo val` after109 changing batch/LR/aug.
libreyolo