VSS Generate Video Calibration
Run AutoMagicCalib over one of three input sources and drive the calibration through the microservice REST API. For normal single-run calibration, input-resolution work differs per source and shared work from verify_project onward lives in this file. Pair selected input-mode reference with Shared Calibration Tail; explicit tuning uses its own routed workflow below.
When to Use This Skill
Run or tune AutoMagicCalib end-to-end on local files, RTSP streams, or the bundled sample dataset and (when needed) deploy the AMC microservice.
Do not use for non-AMC camera calibration or runtime analytics.
Workflow
Resolve tuning intent before input mode, then follow the applicable routing table and workflow below. Each section that ends in workflow, quick start, or flow is intended to be executed top-to-bottom. Detailed reference material lives in references/; load only the references needed for the selected intent and input mode.
Examples
Worked end-to-end examples are kept under evals/ (each *.json manifest contains a runnable scenario) and inline in the per-workflow curl blocks below. Run a Tier-3 evaluation with nv-base validate <this-skill-dir> --agent-eval to replay them.
Limitations
- Requires the matching VSS profile / microservice to be deployed and reachable from the caller.
- NGC-hosted models and NIMs may be subject to rate-limits, GPU memory requirements, and license restrictions.
- Concurrency, GPU memory, and storage limits depend on the host hardware and the profile's compose file.
Shared helper references are loaded only when needed:
- Read
references/common-steps.mdwhen a mode reference needs the sharedcreate_project, video-upload, or handoff snippets. - Read
references/calibration-tail.mdwhen you need the reusable Python implementation of the stage-linear-media → verify → VGGT/post-process → AMC/post-process → compare-results tail. - Read
references/tuning.mdfirst when the user explicitly asks to tune, optimize, or find AMC settings. It owns tuning project names and the multi-attempt lifecycle; do not run the ordinary shared calibration tail for that request. - Read
references/calibration-parameters.mdonly when tuning evidence requires a numeric AMC configuration change.
Intent Routing
- Normal calibration — follow the existing input-mode workflow and shared calibration tail. Run one AMC configuration, then offer bounded tuning after reporting a completed or tunable failed result. Do not start repeated attempts without an explicit tuning request or acceptance of that offer.
- Explicit tuning — read
references/tuning.mdbefore the relevant input-mode reference. Use the input-mode reference only for source discovery, ordering, capture, and upload semantics; the tuning reference replaces its project-naming and shared-tail handoff instructions.
Input Routing
Match the user's request to a mode, then load that mode's reference for input collection, mode-specific API calls, and the full Python script.
| User says / has | Mode | Reference |
|---|---|---|
| "launch AMC" / "deploy auto-calibration" / "set up auto-magic-calib" / "start AMC microservice" | deploy |
references/deploy-auto-calibration-service.md |
| "calibrate my videos" / "calibrate from video files" / local MP4 files | videos |
references/videos.md |
| "calibrate RTSP streams" / "calibrate from live cameras" / live RTSP URLs | rtsp |
references/rtsp.md |
| "test sample dataset" / "verify AMC install" / "launch and test" | sample-dataset |
references/sample-dataset.md |
Disambiguation rule: if the user is asking to launch / deploy / set up AMC (no calibration verb) → deploy. If they provide RTSP URLs → rtsp. If they mention local files / a videos directory → videos. If they ask to verify install or test the bundled sample → sample-dataset. Combined intents (e.g. "launch AMC and calibrate my videos") → walk deploy first, then the calibration mode. When ambiguous, ask via AskUserQuestion.
Prerequisites (shared across calibration modes)
- Platform preflight from
references/deploy-auto-calibration-service.mdStep 0 passes before any AMC deploy or calibration API work. The calibration host needs Ubuntu 24.04 onx86_64, NVIDIA Driver 590 or newer, NVIDIA GPU access, and NVENC hardware encoder support. If the preflight fails, stop immediately, tell the user which requirement was not met, and ask them to provide an existingcalibration.json, run calibration on a supportedx86_64dGPU host, or transfer generated calibration artifacts. Do not continue AMC setup, VIOS probing, capture, upload, or calibration automatically. DGX Spark isaarch64, so use existing/generated artifacts for this flow. - AMC microservice + UI running. If not, walk
references/deploy-auto-calibration-service.mdfirst. - Microservice reachable at
http://<HOST_IP>:${VSS_AUTO_CALIBRATION_HOST_PORT:-8010}/v1/ready→{"code":0,...}. - Projects directory writable by the container user. If you didn't just deploy (so Step 5 of the deploy reference hasn't run), confirm the write test in
references/deploy-auto-calibration-service.md§ Step 5 — otherwise the firstcreate_projectreturns[Errno 13] Permission denied. - Python 3 with
requestsinstalled.
Mode-specific prerequisites (VIOS for rtsp, sample zip for sample-dataset) live in the respective references. The platform preflight applies even when an AMC service is already running.
Shared Calibration Tail
The shared sequence is stage-linear-media → verify → VGGT (when ready) → post-process → AMC → post-process → results. After the mode-specific reference has uploaded videos / automatically ingested RTSP clips / uploaded the bundled sample, run this tail. Use references/calibration-tail.md for the shared Python snippet.
For a new normal or sample run using defaults, prepare its configuration before invoking the tail: fetch GET /v1/config/defaults, retain the confirmed dataset layout_px_per_m and explicitly requested overrides, then apply the complete configuration with POST /v1/config/<project_id>. Do this only while the backend is idle and no other client is changing settings; confirm GET /v1/config matches before calibration. Current service-wide settings may belong to a previous tuning attempt, not the defaults. Do not reset an existing project being resumed or overwrite user-selected settings from a file or the UI.
AMC UI sequence: Step 1 Project Setup, Step 2 Video Configuration, Step 3 Parameters, Step 4 Rectification, Step 5 Manual Alignment, Step 6 Execute, Step 7 Results.
Step A: Stage Linear Media
AMC v3.3.0 cannot calibrate raw media. After the mode-specific workflow has uploaded videos or completed RTSP ingest, explicitly choose one path before verification:
- Already-linear/pinhole media — call
POST /v1/linear_media/<project_id>and requirerectification_state == "COMPLETED". - Distorted media — open AMC UI Step 4: Rectification; select Auto or Manual; review the estimate; then click Generate Rectified Videos. Auto supports
simple_divisional(default),simple_radial, andradial; Manual supports per-cameramodel,k1, andk2forradial.READY_FOR_REVIEWis not complete: requirerectification_state == "COMPLETED"before continuing. Use Videos Are Rectified only for confirmed already-linear media. Re-rectification invalidates verification, calibration, and post-processing outputs.
For REST-only rectification, use the running AMC service contract at the service-root /openapi.yaml (without /v1); the interactive API reference is at /docs:
- Auto —
POST /v1/rectification/<project_id>starts frame-0 estimation. PollGET /v1/rectification/<project_id>untilREADY_FOR_REVIEW; thenGET /v1/rectification/<project_id>/cameras, review everyauto_estimate, and commit all camera parameters withPOST /v1/rectification/<project_id>/manualusing{"cameras":{"cam_00":{"model":"...","k1":0.0,"k2":0.0},...}}. This explicit commit generates full rectified videos. - Manual —
POST /v1/rectification/<project_id>/manual/start; optionally preview each adjustment throughPOST /v1/rectification/<project_id>/preview/<camera_id>; then commit a complete per-cameracamerasmap toPOST /v1/rectification/<project_id>/manual. - Poll
GET /v1/rectification/<project_id>untilCOMPLETED. Stop onERROR; do not verify or calibrate fromREADY_FOR_REVIEW.
Rectification produces rectified.mp4 and rectified.jpg. External alignment files normally use coord_space=original; use rectified only for points created on AMC rectified media. Never call /v1/calibrate/<project_id> before the linear-media or rectification state is complete.
Step B: Verify Project
POST /v1/verify_project/<project_id>
Response: {"project_state": "READY"} — must be READY before calibrating. If not READY, re-check that videos + alignment + layout are present (either via API or via UI manual alignment).
Step C: Independent VGGT Calibration
After verification and before AMC, inspect vggt_state. VGGT is available only for multi-camera projects. Start VGGT by default from READY, resume and wait from RUNNING, and always post-process a COMPLETED multi-camera result, including one completed before the current invocation. MODEL_MISSING or ERROR is reported as an AMC-only fallback unless the error indicates a shared input or infrastructure problem. Check amc_state, vggt_state, and postprocess_state independently; wait for each workload to finish before starting another.
POST /v1/vggt/calibrate/<project_id>
GET /v1/get_project_info/<project_id> # poll vggt_state
POST /v1/postprocess/<project_id> # multi-camera only, after VGGT
GET /v1/get_project_info/<project_id> # require postprocess_state == COMPLETED
GET /v1/vggt_results/<project_id>/evaluation_statistics # VGGT metrics when GT exists
Step D: Start AMC Calibration
Confirm the plan before calibrating. Whether the settings file and detector were auto-detected or asked, present a short summary and confirm via AskUserQuestion before the POST /calibrate. The resolved values are the defaults, so confirming is one click — but the user can switch the detector or skip an auto-detected settings file. Summarize:
- Detector —
resnetortransformer(the value to be sent). - Calibration settings — the file being applied (path), or default parameters (with the option to tune them in the UI first — see below).
- Optional overrides — ground-truth zip and focal lengths, if any.
The sample-dataset install-check run uses a fixed resnet and can proceed without this confirmation.
POST /v1/calibrate/<project_id>
Content-Type: application/json
{"detector_type": "resnet"} # or "transformer"
detector_type is a separate /calibrate parameter — not consumed by /v1/config/<id>. If the user provided a calibration settings file, parse it for "detector" / "detector_type" and use that value. If the file doesn't specify one, the default (resnet) is the value shown in the confirmation above — the user can switch it there before calibrating. If there's no settings file at all, ask the user via AskUserQuestion:
resnet— default, fast.transformer— slower, better under heavy occlusion.
UI Step 3 (Parameters) does NOT cover detector choice; never assume the user picked one in the UI.
Also when there's no settings file, ask whether to review the calibration parameters before this single run (AskUserQuestion):
- Proceed with the default parameters — well-suited to typical warehouse scenes; recommended unless the user has specific tuning in mind.
- Review or adjust parameters in the UI first — open the project, go to Step 3: Parameters, change values, and click Save; then continue.
In Step 3, set layout_px_per_m directly or measure a known two-point distance. Re-run post-processing after a scale or alignment change.
Wait for the user's choice — and, if they choose to tune, for them to confirm they've Saved — before calling /calibrate.
Step E: Poll for AMC Completion
GET /v1/get_project_info/<project_id>
Poll every 10 s. Use project_info.amc_state for AMC completion; aggregate project_state is not a pipeline-success signal.
| State | Meaning |
|---|---|
RUNNING |
AMC calibration in progress |
COMPLETED |
Finished |
ERROR |
Failed — pull log via GET /v1/amc/calibrate/<id>/log |
When calibration starts, surface the project ID, the UI URL (http://<HOST_IP>:${VSS_AUTO_CALIBRATION_UI_HOST_PORT:-5000}), and the log endpoint so the user can watch progress while the run proceeds. During RUNNING, emit a progress line at least once a minute with elapsed time so a long run doesn't look stalled. On ERROR, fetch and show the last lines of GET /v1/amc/calibrate/<id>/log before stopping. Live logs can also be streamed via GET /v1/calibrate/<project_id>/log/<type>/stream.
Typical time: 10–60 min (your-own videos), 10–30 min (bundled sample). A six-camera transformer run can exceed one hour; keep polling and inspect logs/UI instead of treating 60 minutes as failure.
Step F: AMC Post-process and Results
For multi-camera projects, run layout post-processing after AMC calibration. VGGT, when available, runs first and is post-processed before AMC. Single-camera AMC uses its single-view output path; do not require shared multi-camera post-processing.
POST /v1/postprocess/<project_id>
GET /v1/get_project_info/<project_id> # poll postprocess_state until COMPLETED
Do not report a multi-camera project as successful until postprocess_state == "COMPLETED"; raw AMC results may exist even when post-processing fails.
GET /v1/get_project_info/<project_id> # project state
GET /v1/result/<project_id>/evaluation_statistics # only if GT uploaded
GET /v1/result/<project_id>/overlay_image # visual overlay (PNG)
GET /v1/amc/calibrate/<project_id>/log # calibration log
Evaluation response includes Average L2 distance(m) and Average reprojection error 0(px). Evaluation metrics are produced only when a ground-truth GT.zip was uploaded — a missing evaluation_statistics result is normal otherwise and is not the end of result reporting. When VGGT also completed, compare both methods' metrics and Results-page overlays, then select the more accurate calibration for export.
VGGT visualizations are separate from AMC overlays. A VGGT-only result may have no trajectory overlay because those images require AMC tracklet artifacts; review available virtual-object overlays, camera parameters, and exports instead. If AMC fails, report the completed VGGT result separately rather than treating both methods as failed.
After COMPLETED, always give the user a way to review the result for that exact project, regardless of whether metrics exist:
- UI —
http://<HOST_IP>:${VSS_AUTO_CALIBRATION_UI_HOST_PORT:-5000}; open the project, then the Results page to view the overlay. - Overlay image on disk —
${VSS_APPS_DIR}/services/auto-calibration/projects/project_<id>/output/multi_view_results/BA_output/results_ba_scaled_world/overlay_img_*.png(single-camera projects useoutput/single_view_results/cam_00/verification_map_overlay.png). - Project files —
${VSS_APPS_DIR}/services/auto-calibration/projects/project_<id>/.
Optional Tuning Handoff
After reporting a normal run, offer bounded tuning when the user wants further optimization or the failure logs identify a potentially tunable AMC stage. Do not offer numeric tuning for missing or invalid inputs, synchronization, alignment, layout scale, rectification, service, GPU, or storage failures.
If the user accepts, read references/tuning.md. Reuse the normal project as attempt 1 when its inputs and baseline are valid, preserve its existing AMC and VGGT results, and do not rename it.
Settings File and Detector Pattern
Optional across all three modes. Before using a JSON settings file, retrieve GET /v1/config/defaults and inspect the service-root /openapi.yaml (without /v1), or /docs, from the running AMC version. Parse the file, reject known-invalid legacy skip rather than silently translating it to skip_frame, then submit the JSON unchanged in meaning. Do not treat /config/defaults as a complete allow-list: the running API is the authoritative schema validator.
POST /v1/config/<project_id>
Content-Type: application/json
<parsed JSON object; submit as application/json>
The file replaces what the user would otherwise tune in UI Step 3 (parameters, bundle-adjustment, and evaluation knobs). Rectification is UI Step 4 and follows Step A. After a successful POST, also parse the file for "detector" / "detector_type" — if it's "resnet" or "transformer", use that value for the /calibrate call in Step D (detector is a separate API parameter, not consumed by /config).
Non-2xx is surfaced — do not silently fall back. Skip this call entirely if the user chose the UI-fallback path.
UI Fallback Pattern
When alignment / layout files aren't on disk, direct the user to the appropriate AMC UI step. Complete the required UI actions in step order:
- Settings missing → "Open UI project
<project_id>, go to Step 3: Parameters, tune via the settings dialog (or accept defaults), click Save." Also: before the/calibratecall, ask the user viaAskUserQuestionwhether to use theresnetortransformerdetector — Step 3 doesn't cover detector choice. - Layout missing → "Open UI project
<project_id>, go to Step 2: Video Configuration, uploadlayout.pngonly (do NOT re-upload videos — they're already attached via API/RTSP), click Save." - Layout scale unknown → "In Step 3: Parameters, enter the known
layout_px_per_mor measure a known two-point distance on the layout, then Save." Do not accept an unconfirmed default scale. - Alignment missing → Complete Step 4: Rectification first, then open Step 5: Manual Alignment, mark at least four correspondence sets on the rectified camera views and layout, and Save. Alternatively, upload an external alignment file: UI file upload expects original-frame coordinates; use the API with
coord_space=rectifiedfor a file already in rectified coordinates.
Wait for user confirmation. For alignment/layout, verify on disk before continuing:
# Project state lives under $VSS_APPS_DIR/services/auto-calibration/projects
# (the path bind-mounted into the MS container in
# deploy/docker/services/auto-calibration/ms/compose.yml).
HOST_PROJECTS="${VSS_APPS_DIR}/services/auto-calibration/projects"
ls "$HOST_PROJECTS/project_<project_id>/manual_adjustment/"
# Expected: alignment_data.json, layout.png
Success Criteria
amc_state == "COMPLETED"for a successful AMC run; reportvggt_stateindependently. A failed method does not invalidate the other method's completed result.- For each completed multi-camera method, require successful post-processing and inspect its available artifacts before recommending an export.
- If manual alignment was used:
${VSS_APPS_DIR}/services/auto-calibration/projects/project_<id>/manual_adjustment/containsalignment_data.json+layout.png. - If GT was uploaded: fetch both available methods' evaluation statistics and compare their metrics plus overlays before selecting the result to export. Typical thresholds are
Average L2 distance(m)< 1.5 andAverage reprojection error 0(px)< 5 for your data or < 10 for the bundled sample. - Report failed or unavailable methods and missing artifacts explicitly. Without GT, label quality assessment heuristic and require visual review.
Key Output Files
Under ${VSS_APPS_DIR}/services/auto-calibration/projects/project_<project_id>/:
project_<project_id>/
├── manual_adjustment/
│ ├── alignment_data.json
│ └── layout.png
├── output/
│ ├── single_view_results/cam_XX/
│ │ ├── camInfo_hyper_XX.yaml
│ │ └── trajDump_Stream_0_3d.txt
│ ├── multi_view_results/BA_output/results_ba/
│ │ ├── initial/camInfo_XX.yaml
│ │ └── refined/camInfo_XX.yaml # ← final calibration
│ └── multi_view_results/BA_output/results_ba_scaled_world/
│ └── overlay_img_XX.png # ← visual overlay for review
└── calibration.log
Troubleshooting
Mode-specific issues live in each reference's own troubleshooting table.
| Issue | Fix |
|---|---|
verify_project state not READY |
Confirm videos uploaded/ingested and alignment + layout are present (either via API or via UI manual alignment). Mode-specific upload steps in the reference. |
| Manual alignment files missing after UI step | User didn't click Save; also verify ${VSS_APPS_DIR}/services/auto-calibration/projects/project_<id>/manual_adjustment/ exists. |
Calibration remains RUNNING longer than expected |
Inspect GET /v1/amc/calibrate/<id>/log, progress, and host resources. Runtime varies with inputs and settings; elapsed time alone does not establish failure or insufficient tracklets. See Custom Dataset. |
Immediate ERROR state |
Check video readability, synchronization, overlapping fields of view, and camera order; upload order defines indices. |
| Low L2 but high reprojection | Provide explicit focal_length override during input upload (see videos / rtsp references). |
VGGT INIT, never READY |
Confirm multi-camera input, completed media preparation, and successful verification. For MODEL_MISSING, follow references/deploy-auto-calibration-service.md Step 2. |
| Upload timeout | Large videos — bump timeout=300 to e.g. 600 in the per-mode Python script. |
| Port scan finds no backend | Backend not running — walk references/deploy-auto-calibration-service.md first. |
| REST call returns connection refused | Target microservice is not running. Probe /docs or /health; deploy or repair it with vss-build-vision-ai or the applicable vss-deploy-* skill. |
| NGC pull returns HTTP 401/403 | NGC_CLI_API_KEY is missing or expired. Re-authenticate with docker login nvcr.io, then retry without printing the key. |
| Container OOM or model load failure | GPU memory is insufficient for selected profile. Choose smaller variant or free GPU memory before retrying. |
For Downstream Skills: MV3DT Export
Exports describe rectified camera geometry. Downstream video must use the corresponding rectified media or matching distortion remapping; see Custom Dataset.
Downstream consumers (e.g. a Multi-View 3D Tracking skill owned by another team) fetch the MV3DT-format calibration output directly from the microservice. This skill returns the project_id; the downstream skill calls:
GET /v1/result/{project_id}/mv3dt_result?result_type=amc
# Response: application/zip — mv3dt_output.zip containing transforms.yml
For independent VGGT output (only available if VGGT ran to COMPLETED, see Step C):
GET /v1/result/{project_id}/mv3dt_result?result_type=vggt
# Response: application/zip — vggt_mv3dt_output.zip
Downstream skill flow:
- Call this skill with the user's inputs; capture the printed
project_id. - Wait for the skill to return (it polls until
COMPLETEDinternally). GET /v1/result/{project_id}/mv3dt_result?result_type=amc— save the ZIP locally.- If independent VGGT calibration also ran, optionally fetch
?result_type=vggtfor the VGGT MV3DT result.
Related Skills
vss-manage-video-io-storage— VIOS API skill; only thertspcalibration mode depends on VIOS being reachable.
See Custom Dataset for input requirements and Calibration Workflow for the UI-driven alternative to this API flow.