Yolo Master Agent

YOLO-Master Agent Skill

Use This Skill

Use this skill for any repository task that should drive the YOLO-Master stack end-to-end:

• train, val, predict, track, export, benchmark, tune
• model inspection and task detection
• LoRA save/load/merge and yolo.lora.diagnose
• PEFT comparison via yolo.eval.peft_compare
• Sparse SAHI comparison via yolo.eval.sparse_sahi_compare
• MoE diagnose/prune
• multimodal visual inference with OpenAI VLM/LLM cooperation
• multimodal batch evaluation over a dataset or image folder
• solutions workflows
• launchers for Gradio / Streamlit
• end-to-end orchestration via yolo.pipeline.experiment

Execution Rule

First make sure the local Ultralytics framework is installed and the yolo CLI is available. Prefer the CLI over raw Python API for supported commands, and use the bundled dispatcher when you want a deterministic, structured run:

python -m pip install -e .
yolo version
python agent/scripts/run_yolo_master_skill.py --json '{"skill":"yolo.train","inputs":{"model":"yolo11n.pt","data":"coco8.yaml"},"params":{"epochs":1,"imgsz":32}}'

On Apple Silicon hosts with PyTorch MPS support, the dispatcher now defaults heavy compute modes such as train, val, benchmark, predict, and track to device=mps when no explicit device is provided. Override with runtime.device or params.device if needed.

If the CLI run is auto-selected onto MPS/CUDA and fails for a device-level runtime reason, the dispatcher will retry once on CPU and return a structured recovery record with the full attempt trail.

When you need fast coverage across many skills or requests, use the AutoTrain-style validator first:

python agent/scripts/validate_yolo_master_skill.py --suite quick --pretty --summary-only

quick is the default agent loop. It combines fast-smoke, dry-run, and contract so agents can iterate without waiting on real model inspection or CLI cold-start probes. Use all only when you explicitly want the slower full non-manual regression pass. The case pack now lives in assets/autotrain_cases/ as skill-grouped JSON files. It includes multimodal dry-run/contract probes plus dry-run coverage for yolo.pipeline.experiment, yolo.lora.diagnose, and yolo.eval.peft_compare.

For quick regression checks, prefer the tiered suites:

python agent/scripts/validate_yolo_master_skill.py --suite quick --pretty --summary-only
python agent/scripts/validate_yolo_master_skill.py --suite fast-smoke --pretty --summary-only
python agent/scripts/validate_yolo_master_skill.py --suite cli-smoke --pretty --summary-only
python agent/scripts/validate_yolo_master_skill.py --suite deep-smoke --pretty --summary-only
python agent/scripts/validate_yolo_master_skill.py --suite extended --pretty --summary-only
python agent/scripts/validate_yolo_master_skill.py --suite dry-run --pretty --summary-only
python agent/scripts/validate_yolo_master_skill.py --suite contract --pretty --summary-only
python agent/scripts/validate_yolo_master_skill.py --suite all --pretty --summary-only

Workflow

1. Inspect the request and normalize paths.
2. Install/refresh the local Ultralytics package first when the yolo CLI is missing.
3. Prefer policy.dry_run=true while validating or evolving the skill surface.
4. On Apple Silicon, let the dispatcher pick mps by default for train/val/eval unless the request already sets device.
5. Let the dispatcher auto-complete safe runtime defaults such as workers=0 on macOS train/val paths when the request leaves them unset.
6. Use yolo CLI for supported tasks; fall back to Python API only when the CLI does not cover the action.
7. For predict and track, accept source in either inputs.source or params.source; the dispatcher will normalize it before CLI emission.
8. Pass all task-specific options through params unchanged.
9. Return structured artifacts, metrics, evaluation summaries, environment reports, and next actions.
10. For long jobs, set policy.async=true on train/tune/pipeline requests to submit a subprocess job and return job_id, status, stdout/stderr, manifest, and progress paths.

Multimodal Inference

yolo.multimodal.infer is an optional enhancement layer for visual reasoning. It does not replace yolo.predict: it runs YOLO first, condenses detections into reasoning evidence, then calls the OpenAI Responses API with input_text plus input_image, and optionally runs a second LLM refinement pass.

Environment variables:

• OPENAI_API_KEY
• OPENAI_BASE_URL optional
• OPENAI_API_MODE optional, auto, responses, or chat.completions
• OPENAI_VLM_MODEL optional
• OPENAI_LLM_MODEL optional
• structured_output=true asks the VLM/LLM to return a strict JSON verdict that can be parsed into verdict
• prompt_template=vlm_coco_multitask asks the VLM to output caption, global classification, COCO-style object proposals, rough segmentation proxies, YOLO cross-checks, and fusion hints
• prompt_template=vlm_open_world_detection asks the VLM to preserve open-world objects, optional COCO mappings, rough segmentation proxies, captioning, and fusion hints for novel categories
• prompt_template=vlm_open_world_detection_compact is the compact open-world schema tuned for providers that tend to truncate long JSON, especially qwen-vl-plus
• prompt_template=vlm_open_world_detect_classify_compact focuses on a few grounded object proposals plus scene-level classes
• prompt_template=vlm_open_world_caption_misses_compact focuses on scene captioning plus the most important likely misses
• max_output_tokens defaults to 3500 in COCO multitask template mode; avoid lowering it below this unless you also reduce schema fields
• use_marked_image=true draws numbered YOLO boxes onto a lightweight marked copy before VLM inspection
• visual_search_mode=auto lets the VLM request crop-and-zoom follow-ups through visual_search.needs_zoom and search_regions
• fusion_mode=preview converts parsed VLM/LLM hints into metric-safe keep/suppress/add/relabel/adjust proposals plus COCO-style prediction records; use fusion_mode=off to disable it
• fusion_policy=add_only is now the default. It only allows filtered high-confidence VLM additions that look like genuine misses; use balanced or aggressive only when you explicitly want VLM-driven suppress/adjust/relabel actions
• fusion_policy=open_world_assist is the opt-in exploratory path. It keeps the normal metric preview for COCO-mappable outputs, but also preserves unmapped open-world objects in multimodal.fusion.open_world_predictions_preview
• open-world normalization now tries to anchor novel labels against the bundled LVIS 1203 and V3Det 13204 taxonomies, and batch reports expose taxonomy.best, taxonomy.candidates, dataset hit counts, and unmatched totals
• taxonomy matching is now intentionally conservative by default:
  • open_world_taxonomy_min_score=40
  • open_world_taxonomy_require_exact_for_generic=true This prevents weak generic matches such as grass -> bear grass from being treated as confirmed taxonomy anchors unless you explicitly loosen the policy
• open-world report aggregation now separates:
  • enhancement_stats: labels allowed to enter open-world enhancement statistics
  • reasoning_only: labels preserved for agent reasoning but filtered from aggregate enhancement stats Default filters are:
  • open_world_filter_unmatched_taxonomy=true
  • open_world_filter_generic_labels=true
• open_world_assist_profile gives the agent a higher-level mode switch for open-world runs without forcing you to set every threshold by hand:
  • strict default for evaluation-oriented runs: stronger taxonomy gate, generic labels filtered, unmatched labels kept as reasoning-only
  • balanced: keeps generic filtering, but allows unmatched taxonomy labels to remain in enhancement stats
  • exploratory: lowest taxonomy gate, generic/unmatched labels stay in enhancement stats for broad discovery passes Explicit param values still win over the profile defaults
• core implementation now lives in runtime/; scripts/ contains only thin executable wrappers that delegate into runtime/cli/
• opt-in hooks now exist for:
  • IoU-based open-world relabeling via open_world_iou_relabel_enabled
  • WordNet hypernym fallback when taxonomy matching misses
  • cross-profile verified-list merging for prompt ensemble style arbitration
• When vlm_model looks like qwen-vl-* and prompt_template=vlm_open_world_detection, the dispatcher now auto-switches to a compact task profile unless you explicitly request a different template path:
  • default: vlm_open_world_detect_classify_compact
  • caption/miss emphasis: vlm_open_world_caption_misses_compact
  • override with compact_open_world_profile=detect_classify|caption_misses

Behavior:

• thinking_with_image=true attaches the image to the VLM request
• enable_llm_refine=true or OPENAI_LLM_MODEL enables the refinement pass
• missing OPENAI_API_KEY returns a structured blocked result
• Provider-aware defaults can be externalized under runtime/multimodal/providers/*.yaml; built-in configs currently include openai and dashscope.
• DashScope/OpenAI-compatible chat endpoints can use params.provider="dashscope" plus DASHSCOPE_API_KEY, or set OPENAI_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1 with params.openai_api_mode="chat.completions".
• the manifest now preserves the multimodal block, including parsed verdicts, visual-search crop passes, fusion preview, and artifact paths when available
• every response envelope includes usage.tokens and cost_estimate; cost is null when provider pricing is not configured.

Example:

python agent/scripts/run_yolo_master_skill.py --json '{"skill":"yolo.multimodal.infer","inputs":{"model":"yolo11n.pt","source":"ultralytics/assets/bus.jpg","prompt":"What matters most in this image?"},"params":{"thinking_with_image":true,"vlm_model":"gpt-4.1-mini","llm_model":"gpt-4.1-mini","max_reasoning_items":3,"max_reasoning_boxes":20},"policy":{"dry_run":true}}' --pretty

COCO multitask VLM prompt:

python agent/scripts/run_yolo_master_skill.py --json '{"skill":"yolo.multimodal.infer","inputs":{"model":"yolo11n.pt","source":"ultralytics/assets/bus.jpg","prompt":"Detect, classify, segment roughly, caption, and propose metric-safe fusion changes."},"params":{"thinking_with_image":true,"structured_output":true,"prompt_template":"vlm_coco_multitask","use_marked_image":true,"visual_search_mode":"auto","fusion_mode":"preview","vlm_model":"qwen-vl-plus","llm_model":"qwen-plus","openai_api_mode":"chat.completions"}}' --pretty

Open-world VLM prompt:

python agent/scripts/run_yolo_master_skill.py --json '{"skill":"yolo.multimodal.infer","inputs":{"model":"yolo11n.pt","source":"ultralytics/assets/bus.jpg","prompt":"Find visible objects, including novel categories outside COCO, and preserve them for downstream reasoning."},"params":{"thinking_with_image":true,"structured_output":true,"prompt_template":"vlm_open_world_detection","use_marked_image":true,"visual_search_mode":"auto","fusion_mode":"preview","fusion_policy":"open_world_assist","vlm_model":"qwen-vl-plus","llm_model":"qwen-plus","openai_api_mode":"chat.completions"}}' --pretty

Multimodal Batch Evaluation

Use yolo.multimodal.evaluate when the agent needs to evaluate a real image sample or dataset split with YOLO first, then VLM/LLM cross-checks.

• inputs.data selects a dataset YAML such as coco128.yaml; params.split defaults to val
• inputs.source may point to a local image folder, image file, or image-list text file
• params.limit, offset, stride, shuffle, and seed control sampling; limit=0 means all resolved images
• params.run_yolo_val=true also runs a YOLO-only validation baseline when inputs.data is available
• Ground-truth labels are read for reporting when available; they are not added to the VLM prompt unless include_ground_truth_in_prompt=true
• params.prompt_template="vlm_coco_multitask" enables VLM-side detection/classification/rough segmentation/caption output for downstream fusion experiments
• params.prompt_template="vlm_open_world_detection" enables a less conservative open-world path where novel categories are preserved even when they cannot be mapped into COCO metrics
• params.use_marked_image=true and params.visual_search_mode=auto enable Set-of-Mark-style box grounding and crop/zoom follow-up calls
• params.fusion_mode="preview" writes conservative fused prediction previews and, for batch evaluation, a fusion-preview-coco-predictions.json artifact for downstream COCO scoring
• Fusion is policy- and confidence-guarded by default: add_only blocks suppress/relabel/adjust, and even in broader policies high-confidence YOLO boxes are protected while box adjustments must stay close to the original box
• In open_world_assist, unmapped novel objects are emitted to open_world_predictions_preview instead of being dropped; only COCO-mappable predictions participate in metric preview and guardrail selection
• open_world_assist now defaults to an add-first posture: it prefers preserving novel objects and no longer enables suppress/adjust/relabel by default
• When YOLO-format labels are available, evaluation.metric_preview compares YOLO-only vs fused predictions on the sampled images and writes fusion-metric-preview.json; treat it as a fast same-sample guardrail, not an official benchmark
• metric_guardrail writes metric-guarded-coco-predictions.json: it keeps fused predictions only when there is a material change and same-sample map50_95 shows a positive delta without recall regression; otherwise it falls back to YOLO-only predictions

Example:

python agent/scripts/run_yolo_master_skill.py --json '{"skill":"yolo.multimodal.evaluate","runtime":{"prefer_cli":true,"prefer_mps":true},"inputs":{"model":"yolo11n.pt","data":"coco128.yaml","prompt":"Cross-check detector outputs and summarize obvious false positives, misses, duplicates, and uncertainty."},"params":{"limit":5,"split":"val","imgsz":640,"batch":1,"thinking_with_image":true,"prompt_template":"vlm_coco_multitask","use_marked_image":true,"visual_search_mode":"auto","fusion_mode":"preview","vlm_model":"qwen-vl-plus","llm_model":"qwen-plus","openai_base_url":"https://dashscope.aliyuncs.com/compatible-mode/v1","openai_api_mode":"chat.completions"},"policy":{"dry_run":false}}' --pretty

AutoTrain Loop

Use the bundled validator and case pack to keep this skill honest:

• case pack: assets/autotrain_cases/ grouped by skill, with assets/autotrain_cases.json retained for compatibility
• report: logs/autotrain-report.json
• bootstrap: python -m pip install -e .
• dispatcher supports policy.dry_run=true for cheap coverage before real runs
• yolo CLI is the preferred execution surface for supported actions
• quick is the default iteration suite: fast-smoke + dry-run + contract
• the validator enables a short-lived runtime cache for Torch/MPS detection so repeated subprocess cases do not re-import the stack
• fast-smoke protects bootstrap and planning paths with tight timing budgets
• cli-smoke validates real yolo CLI cold-start execution
• deep-smoke holds heavyweight real-model inspection and local .pt inference checks
• all runs every non-manual case, including slower cli-smoke and deep-smoke
• extended-cli carries slower real CLI validation probes such as mini-dataset yolo train and yolo val on mps, and is marked manual_only
• contract verifies failure-path behavior and manifest emission
• contract now also includes in-process recovery probes so auto device fallback semantics stay covered without adding test-only hooks to the dispatcher
• contract includes single-image and batch multimodal stub probes so OpenAI-compatible request shaping, structured verdict parsing, and aggregation stay covered
• CLI failures now carry categorized hints so the agent can recover instead of stopping at a raw traceback.
• Built-in dataset YAML names such as coco128.yaml are auto-resolved against the local repository before execution.
• doctor returns environment, device selection source, and agent-facing recommendations.
• CLI train/val/predict/benchmark/export responses now carry environment metadata, and auto-selected runs can include a recovery trail when a device fallback occurs.

Pipeline And PEFT Tools

Use yolo.pipeline.experiment for end-to-end train/val/export/benchmark flows. It accepts either stage keys such as train, val, export, benchmark, or an explicit params.stages list, and can include inspect, lora_diagnose, moe_diagnose, and peft_compare. Real runs write progress.jsonl next to the manifest for file-tail progress monitoring.

For packaged MoE, MoA, MoT, and Latent MoE configurations, inputs.profile accepts the stable identifier printed by yolo mixtures. Do not also pass inputs.model; profile resolution supplies the exact YAML path and task:

python agent/scripts/run_yolo_master_skill.py --json '{"skill":"yolo.pipeline.experiment","inputs":{"profile":"26/yolo26-master-mot-n","data":"coco8.yaml"},"params":{"train":{"epochs":1,"imgsz":32},"val":{"imgsz":32},"export":{"format":"onnx"}},"policy":{"dry_run":true}}' --pretty

Every skill_manifest.json is schema-versioned and preserves the redacted normalized request plus handler-specific results such as pipeline stages, selected checkpoints, diagnostics, and plans. Profile-driven manifests also record the catalog metadata and model YAML SHA-256. API keys, access tokens, passwords, secrets, authorization values, and common inline credential assignments are written as <redacted>; raw stdout/stderr tails are bounded.

Use yolo.release.audit to build a read-only release bundle from a completed Agent manifest:

python agent/scripts/run_yolo_master_skill.py --json '{"skill":"yolo.release.audit","inputs":{"manifest":"runs/agent/experiment/skill_manifest.json"},"params":{"output":"runs/agent/experiment/release_bundle.json"},"policy":{"dry_run":false}}' --pretty

The audit verifies the source manifest checksum, resolves only explicit artifact references, records SHA-256 and size for each evidence file, joins model identity with governance/export YAML, and returns publishable, experimental, or refused. It never copies checkpoints or constructs a model. Inspect decision.missing for incomplete evidence and decision.hard_failures for integrity, path-safety, source-stage, schema, or identity failures.

Legacy manifests without schema_version are classified as compatibility.kind=legacy_unversioned and always remain refused until a new versioned manifest is emitted. The compatibility path consumes only explicit environment.references.model.resolved, job.save_dir, and artifacts[*].path values; it never scans a run directory. For a local or manual-CI threshold gate, use:

python scripts/audit_release_manifest.py runs/agent/experiment/skill_manifest.json \
  --output runs/agent/experiment/release_bundle.json --fail-on experimental

--fail-on refused returns success for experimental audits; --fail-on experimental returns non-zero for both experimental and refused.

Use yolo.lora.diagnose to inspect active or loaded adapters:

python agent/scripts/run_yolo_master_skill.py --json '{"skill":"yolo.lora.diagnose","inputs":{"model":"yolo11n.pt"},"params":{"path":"runs/train/exp/weights/lora_adapter_best","svd_max_layers":20,"spectrum_max_layers":12},"policy":{"dry_run":true}}' --pretty

Use yolo.eval.peft_compare to compare Full-SFT and PEFT variants with the same base train/val settings:

python agent/scripts/run_yolo_master_skill.py --json '{"skill":"yolo.eval.peft_compare","inputs":{"model":"yolo11n.pt","data":"coco8.yaml"},"params":{"train":{"epochs":1,"imgsz":32,"batch":1},"variants":[{"name":"full_sft","train":{"lora_r":0}},{"name":"lora_r8","train":{"lora_type":"lora","lora_r":8,"lora_alpha":16}}]},"policy":{"dry_run":true}}' --pretty

Manual Probes

Use these when you want stronger confidence than the default smoke suites without pulling slow jobs into routine validation.

Environment doctor and adaptive install probe:

python agent/scripts/run_yolo_master_skill.py --json '{"skill":"yolo.system","action":"doctor","params":{"ensure_cli":true}}' --pretty

Real CLI training and validation probes on the bundled mini dataset:

python agent/scripts/validate_yolo_master_skill.py --suite extended --pretty --summary-only

Equivalent direct CLI train command:

yolo train model=scripts/peft_validation/yolo11n.pt data=agent/assets/mini-detect/mini_detect.yaml imgsz=64 epochs=1 batch=1 device=mps workers=0 plots=False verbose=False patience=1 project=runs/agent name=train-mini-mps-manual

Equivalent direct CLI val command:

yolo val model=scripts/peft_validation/yolo11n.pt data=agent/assets/mini-detect/mini_detect.yaml imgsz=16 batch=1 device=mps workers=0 plots=False verbose=False project=runs/agent name=val-mini-mps-manual

Structured dispatcher example with automatic MPS selection:

python agent/scripts/run_yolo_master_skill.py --json '{"skill":"yolo.train","runtime":{"prefer_cli":true,"prefer_mps":true},"inputs":{"model":"scripts/peft_validation/yolo11n.pt","data":"agent/assets/mini-detect/mini_detect.yaml"},"params":{"epochs":1,"imgsz":64,"batch":1,"workers":0,"plots":false,"verbose":false,"patience":1},"artifacts":{},"policy":{"dry_run":false}}' --pretty

Regenerate the taxonomy-enriched small-batch open-world report from an existing real-run log:

python agent/scripts/regenerate_open_world_report.py \
  --input agent/logs/qwen-open-world-small-batch.json \
  --json-out agent/logs/qwen-open-world-small-batch-report.json \
  --md-out agent/logs/qwen-open-world-small-batch-report.md

References

• Read README.md for the concise directory map and maintenance boundaries.
• Read references/skill-architecture.md for the full architecture map, skill registry, request/response contract, and execution logic.
• Read references/thinking-with-image.md when improving VLM visual reasoning, marked-image prompting, crop/zoom search, or COCO metric fusion.
• Read the open-world taxonomy assets in assets/open-world-taxonomy for reusable class references, including LVIS 1203 and V3Det 13204 category lists plus source metadata.

Guardrails

• Do not hardcode new CLI strings when a Python API exists.
• Keep params as the pass-through layer for new Ultralytics arguments.
• Prefer yolo CLI for supported commands; use Python API only as fallback.
• On Apple Silicon, prefer mps for training and validation unless the request explicitly overrides the device.
• Consume evaluation in addition to metrics when judging train/val runs.
• Use yolo.system doctor before long runs when the agent needs to confirm install state, selected device, and local repo activation.
• Prefer the recovery field over raw stderr when a run auto-falls back from MPS/CUDA to CPU.
• Keep slow real training out of default validator suites; use the manual probe path instead.
• Treat UI launchers and research scripts as launcher-style skills, not plain sync functions.