jakerains SAM 3 (facebookresearch/sam3)
Overview
Build, integrate, fine-tune, and evaluate Meta SAM 3 with reproducible commands and minimal setup friction.
Quick Routing
| User intent | Action |
|---|---|
| Install SAM 3 and run first inference | Follow setup in references/setup-and-inference.md |
| Add SAM 3 to an existing Python app | Generate starter code with scripts/create_inference_starter.py and adapt API calls |
| Verify environment before setup/inference | Run scripts/sam3_preflight_check.py |
| Fine-tune on custom data | Use references/training-and-eval.md training flow and config guidance |
| Run SA-Co benchmarks or eval custom predictions | Use eval commands in references/training-and-eval.md and upstream scripts/eval/* docs |
| Debug runtime failures | Run the troubleshooting checklist in references/setup-and-inference.md |
Core Workflow
- Confirm objective and modality.
- Set up environment and checkpoint access.
- Run a smoke test.
- Execute the task path: inference, training, or evaluation.
- Return reproducible commands and file paths.
1) Confirm objective and modality
- Identify whether the user needs image inference, video inference, fine-tuning, or benchmark evaluation.
- Confirm whether CUDA is available and which GPU memory budget applies.
- Confirm whether Hugging Face access to
facebook/sam3is already approved.
2) Set up environment and checkpoint access
Use a clean environment:
conda create -n sam3 python=3.12 -y
conda activate sam3
pip install torch==2.7.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
git clone https://github.com/facebookresearch/sam3.git
cd sam3
pip install -e .
Authenticate with Hugging Face before first model load:
hf auth login
Optionally run a preflight check before model download:
python scripts/sam3_preflight_check.py --strict
For full setup and verification commands, read references/setup-and-inference.md.
3) Run a smoke test
From this skill folder, generate a starter script:
python scripts/create_inference_starter.py --mode image --output ./sam3_smoke.py
Edit placeholders and run from a SAM3 checkout.
4) Execute the task path
- Image and video inference:
Use
references/setup-and-inference.mdto applySam3Processorandbuild_sam3_video_predictorpatterns. - Fine-tuning:
Use
references/training-and-eval.mdand start from a provided config insam3/train/configs. - Benchmark evaluation:
Use
references/training-and-eval.mdplus upstream dataset notes underscripts/eval/gold,scripts/eval/silver, andscripts/eval/veval.
5) Return reproducible output
- Report exact commands run and any config overrides.
- Include checkpoint source and authentication assumptions.
- Include prompt text, frame index, and confidence threshold when reporting inference outputs.
Guardrails
- Do not assume checkpoint access is granted; verify login and permission first.
- Prefer official
sam3.model_builderand predictor APIs over custom re-implementations. - Keep generated scripts editable and avoid machine-specific absolute paths.
- If running on CPU, explicitly note expected performance limits before large jobs.
Resources
- Setup and inference guide:
references/setup-and-inference.md - Training and evaluation guide:
references/training-and-eval.md - Starter generator:
scripts/create_inference_starter.py - Preflight checker:
scripts/sam3_preflight_check.py