Benchmark a LibreYOLO model for visionanalysis.org
This is a pointer skill. The benchmark and publish logic live in two separate
repos, each with its own authoritative skill. Do not run benchmarks from inside
libreyolo; switch to the right repo and follow its skill.
The two repos
| Repo | Role | Authoritative skill (open on GitHub) |
|---|---|---|
vision-analysis-benchmark (the harness) |
Runs models, emits va.submission.v1 JSON |
generate-benchmark-results |
vision-analysis (the website) |
Validates JSON, rebuilds the dataset, deploys | submit-benchmark-results |
Direct links to the authoritative skills (read these — this signpost only orients):
- Harness: https://github.com/LibreYOLO/vision-analysis-benchmark/blob/main/skills/generate-benchmark-results/SKILL.md
- Website: https://github.com/LibreYOLO/vision-analysis/blob/main/skills/submit-benchmark-results/SKILL.md
Local checkouts on this machine:
C:\Users\Usuario\Documents\GitHub\vision-analysis-benchmark and
C:\Users\Usuario\Documents\GitHub\vision-analysis.
GitHub: LibreYOLO/vision-analysis-benchmark, LibreYOLO/vision-analysis.
What to know before you start
The dataset, layout gotchas, protocol config, supported runtimes, and the
libreyolo_commit rule all live in the harness skill's "Dataset & protocol"
section (generate-benchmark-results). Read that there; this signpost does not
copy it (so it cannot drift). One thing worth knowing up front: the canonical
eval set is the HF dataset LibreYOLO/coco-val2017-mini500, not full COCO.
Reproducibility: each emitted submission (harness >= 2.1.0) carries a repro
block recording the exact command, harness + libreyolo commits, a verifiable
image-id fingerprint, and weights hash / export manifest. When someone asks how
a published number was produced, that block is the answer. See the harness
skill's "Reproducing a published result" section for the step-by-step.
Flow
- Go to
vision-analysis-benchmark, followgenerate-benchmark-results. This emits oneva.submission.v1JSON per (model x runtime x hardware) run. - Hand the emitted JSON(s) to
vision-analysis, followsubmit-benchmark-resultsto validate, rebuildgenerated/verified-results.v1.json, and open the PR / deploy.
The two skills above are authoritative. If anything here disagrees with them, they win - update this signpost rather than diverging.