Hugging Face Vision Trainer

Vision Model Training on Hugging Face Jobs

Train object detection, image classification, and SAM/SAM2 segmentation models on managed cloud GPUs. No local GPU setup required—results are automatically saved to the Hugging Face Hub.

Detailed Guide

Read the detailed guide before executing this skill. It retains the complete procedure and reference material. Treat its safety, prerequisites, and validation requirements as mandatory. For focused work, load the relevant sections; for end-to-end work, read the guide completely.

When to Use This Skill

Use this skill when users want to:

• Fine-tune object detection models (D-FINE, RT-DETR v2, DETR, YOLOS) on cloud GPUs or local
• Fine-tune image classification models (timm: MobileNetV3, MobileViT, ResNet, ViT/DINOv3, or any Transformers classifier) on cloud GPUs or local
• Fine-tune SAM or SAM2 models for segmentation / image matting using bbox or point prompts
• Train bounding-box detectors on custom datasets
• Train image classifiers on custom datasets
• Train segmentation models on custom mask datasets with prompts
• Run vision training jobs on Hugging Face Jobs infrastructure
• Ensure trained vision models are permanently saved to the Hub

Prerequisites Checklist

Before starting any training job, verify:

Account & Authentication

• Hugging Face Account with Pro, Team, or Enterprise plan (Jobs require paid plan)
• Authenticated login: Check with hf_whoami() (tool) or hf auth whoami (terminal)
• Token has write permissions
• MUST pass token in job secrets — see directive #3 below for syntax (MCP tool vs Python API)

Dataset Requirements — Object Detection

• Dataset must exist on Hub
• Annotations must use the objects column with bbox, category (and optionally area) sub-fields
• Bboxes can be in xywh (COCO) or xyxy (Pascal VOC) format — auto-detected and converted
• Categories can be integers or strings — strings are auto-remapped to integer IDs
• image_id column is optional — generated automatically if missing
• ALWAYS validate unknown datasets before GPU training (see Dataset Validation section)

Dataset Requirements — Image Classification

• Dataset must exist on Hub
• Must have an image column (PIL images) and a label column (integer class IDs or strings)
• The label column can be ClassLabel type (with names) or plain integers/strings — strings are auto-remapped
• Common column names auto-detected: label, labels, class, fine_label
• ALWAYS validate unknown datasets before GPU training (see Dataset Validation section)

Dataset Requirements — SAM/SAM2 Segmentation

• Dataset must exist on Hub
• Must have an image column (PIL images) and a mask column (binary ground-truth segmentation mask)
• Must have a prompt — either:
  • A prompt column with JSON containing {"bbox": [x0,y0,x1,y1]} or {"point": [x,y]}
  • OR a dedicated bbox column with [x0,y0,x1,y1] values
  • OR a dedicated point column with [x,y] or [[x,y],...] values
• Bboxes should be in xyxy format (absolute pixel coordinates)
• Example dataset: merve/MicroMat-mini (image matting with bbox prompts)
• ALWAYS validate unknown datasets before GPU training (see Dataset Validation section)

Critical Settings

• Timeout must exceed expected training time — Default 30min is TOO SHORT. See directive #6 for recommended values.
• Hub push must be enabled — push_to_hub=True, hub_model_id="username/model-name", token in secrets

Limitations

• Use this skill only when the task clearly matches its upstream product or API scope.
• Verify commands, API behavior, pricing, quotas, credentials, and deployment effects against current official documentation before making changes.
• Do not treat generated examples as a substitute for environment-specific tests, security review, or user approval for destructive or costly actions.