Sop Build

Full SOP Pipeline — Build, Evaluate, Deploy, Test

Orchestrates the complete SOP pipeline from asset download through deployment validation. Each phase must succeed before proceeding to the next.

Overview

• Set up the entire SOP pipeline from scratch on a new machine
• Re-run the full pipeline after upstream changes
• Rebuild and redeploy after ds-sop-skills or vss-sop-build changes

Prerequisites

Usage

Phase Pre — Prerequisites Check & Auto-Install

Run this first, before any other phase. It checks secret key files, GPU drivers, Docker, NVIDIA Container Toolkit, and NGC configuration, and can automatically install or configure missing prerequisites (NVIDIA Driver 580, CUDA Toolkit 13, Docker, Docker Compose, NVIDIA Container Toolkit, NGC CLI, and NGC CLI configuration).

cd <bp-repo>
# Run checks and auto-fix/install any missing components
./agentic/vss-sop-skills/vss-sop-deploy/scripts/preflight_check.sh --bp-repo . --fix

To run the checks only, without modifying your system:

cd <bp-repo>
./agentic/vss-sop-skills/vss-sop-deploy/scripts/preflight_check.sh --bp-repo .

Verify: the script reports all prerequisites satisfied (secret keys, NVIDIA driver, Docker, NVIDIA Container Toolkit, NGC CLI & config). If any manual check fails, resolve it per the printed advice before proceeding. See vss-sop-deploy Phase 0 for reference docs.

Phase 0 — Verify Models & Download Assets

For optimal accuracy, you must retrain/fine-tune the models, which can be done using the SOP Training Blueprint. After training, move model and config to /opt/models/... and /opt/sop/... directories.

cd <bp-repo>
./agentic/vss-sop-skills/vss-sop-deploy/scripts/download_assets.sh --bp-repo .

This performs:

1. Checks that the required trained model and config files exist under /opt/models and /opt/sop. If not, prompts the user to retrain/fine-tune the models using the SOP Training Blueprint.
2. Downloads and re-encodes the sample RTSP video to H.264 at 30 FPS.
3. Sets up the ds-sop cache directory and data log directories.

Verify: /opt/models/vlm/checkpoint/ contains VLM weights, /opt/models/gbed_models/ddm/checkpoint.pth.tar exists, /opt/sop/configs/actions.json and /opt/sop/configs/vlm_prompts.txt exist.

Phase 1 — Generate SOP Microservice

Generate the DeepStream SOP microservice with RTSP streaming output into ../ds_sop_microservice. Skip if ../ds_sop_microservice/ already exists with the expected layout.

The generation prompt must include the RTSP streaming output feature request (../ds-sop-skills § 18), which the VSS-SOP blueprint requires:

Please follow instructions in ../ds-sop-skills/example_sop_prompt.md to generate a SOP microservice with rtsp streaming output feature ($18) in folder ../ds_sop_microservice

This follows ../ds-sop-skills/deepstream-sop/SKILL.md to generate:

• FastAPI microservice with /v1/chat/completions SSE endpoint
• DeepStream GEBD pipeline with DDM-Net via Triton CAPI
• VLM inference (Cosmos Reason) for action classification
• SOP sequence checker with compliance monitoring
• Kafka messaging (NvProto + JSON schema)
• Docker build file and deploy compose
• RTSP streaming output (RTSPStreamingServer, tee1-tapped encode branch, SW_ENCODER/ENABLE_RTSP_OUTPUT/RTSP_PORT)

Verify: ../ds_sop_microservice/ contains docker/Docker.build, deploy/compose.yaml, nvds_action_detector/{api_server,ds_3d_action_pipeline,ds_sop_process,vllm_inference}.py.

Phase 2 — Evaluate SOP Microservice

Evaluate the generated microservice to catch issues before the full blueprint build.

Please follow ../ds-sop-skills/eval_sop_prompt.md to evaluate ../ds_sop_microservice with rtsp streaming output feature ($18)

Use the following environment settings:

VLLM_MODEL_PATH="/opt/models/vlm/checkpoint/"
DDM_MODEL_PATH="/opt/models/gbed_models/ddm/checkpoint.pth.tar"
MODEL_ROOT_DIR=/opt/models
ACTION_CONFIG_PATH=/opt/sop/configs/actions.json
VLM_PROMPT_PATH=/opt/sop/configs/vlm_prompts.txt
HOST_CACHE=$HOME/.cache/ds-sop
TEST_VIDEO_PATH=./sop-resources/sop-server-fan-installation-data_v1.0-260213/server_fan/raw/Install_1_h264_30fps.mp4
USER_ID=0
GROUP_ID=0

The Pylon SDK binary is at ../ds_sop_microservice/binaries/pylon-25.10.2_linux-x86_64_setup.tar.gz.

The evaluation follows ../ds-sop-skills/deepstream-sop/references/skill_12_evaluation_workflow.md and performs:

1. Static validation of generated source.
2. Docker image build.
3. Service launch with deploy/.env.
4. API unit tests.
5. File-video API evaluation.
6. Latency, camera, or Kafka evaluation as applicable.

Fix any issues found during evaluation before proceeding. Record commands, results, timings, and fixes in the evaluation report.

Phase 3 — Build VSS-SOP Blueprint (vss-sop-build)

Run the vss-sop-build skill from the blueprint repo root. This scaffolds the SOP blueprint layer on top of VSS 3.1:

cd <bp-repo>
# Follow vss-sop-build SKILL.md Quick Start steps 1–10

Key stages:

1. Clone upstream VSS 3.1.0 repository.
2. Copy all reference files verbatim.
3. Copy and modify foundational services for SOP.
4. Copy and modify VIOS for SOP.
5. Copy agents from upstream and apply SOP profile.
6. Copy NIM from upstream with SOP conventions.
7. Build ds-sop:1.0.0 image from the generated ../ds_sop_microservice.
8. Verify the build with verify_build.sh.

See vss-sop-build SKILL.md for full details.

Phase 4 — Deploy VSS-SOP Blueprint (vss-sop-deploy)

Run the vss-sop-deploy skill from the blueprint repo root:

cd <bp-repo>
# Follow vss-sop-deploy SKILL.md Phases 0–5

Deployment phases:

• Phase 0: Prerequisites check and auto-install (preflight_check.sh --bp-repo . --fix) — already run in Phase Pre above; re-runs as an idempotent safety check.
• Phase 1: Download models & assets (already done in Phase 0 above — will skip if present).
• Phase 2: Start RTSP server simulation (start_rtsp_server.sh --bp-repo .).
• Phase 3: Build DS-SOP Docker image & verify components (already built in Phase 3 above — verify only).
• Phase 4: Configure & deploy the blueprint (configure_blueprint.sh, deploy.sh).
• Phase 5: Test with RTSP stream (test_rtsp.sh --bp-repo .).

See vss-sop-deploy SKILL.md for full details.

Phase 5 — Test VSS-SOP Blueprint (vss-sop-test)

Run the vss-sop-test skill from the blueprint repo root:

cd <bp-repo>
python agentic/vss-sop-skills/vss-sop-test/scripts/vss_sop_test.py \
  --bp-repo . \
  --env-file deployments/sop/.env

Test phases:

• Phase 1: Service health — all required containers running.
• Phase 2: ELK data pipeline — Elasticsearch, indices, VLM messages, Kibana fields.
• Phase 3: VIOS recording & livestream — sensors, streams, recording, livestream.
• Phase 4: VSS Agent end-to-end — MCP, LLM, VLM, snapshot, video, report generation.

See vss-sop-test SKILL.md for full details.

Error Handling

Pass Criteria and Verification

Ensure all unit tests, build checks, deploy checks, and test phases pass with zero failures.

• Do not classify any failure as environmental, intermittent, infrastructure, or acceptable. Treat every failure as a bug to be fixed.
• Do not proceed to the next phase if the current phase has any failing test. Fix all failures first.
• Do not declare completion if any test is failing. A result like "36/38 checks pass" is a failure, not a success.
• If a test times out, fix the timeout or the underlying latency rather than dismissing it.
• If stuck on the same failure after three fix attempts, stop and ask the user for help. Do not skip the failure.

Auto-Debug and Fix Loop

When any phase fails:

1. Read the failure output and auto-debug hints.
2. Collect container logs (docker logs <container> --tail 50).
3. Apply the fix (env, compose, patches, image rebuild, source regeneration).
4. Re-run the failing phase to confirm the fix.
5. Repeat until all checks in that phase pass.

If the root cause is in the generated DS-SOP source code, update ds-sop-skills references and regenerate. If the root cause is in the build/deploy config, update vss-sop-build references.

Examples

After all phases complete with 100% pass rate, present a summary table:

Include:

• Any vss-sop-build or ds-sop-skills files updated during debugging.
• Service endpoints: VSS-UI, Kibana, VIOS-UI, Grafana.

If any test is still failing, the report must clearly state "Pipeline Incomplete — X failures remain" and list them. Do not present a partial success as completion.

Service Endpoints (after successful deploy)

License

Use of this skill is governed by the Creative Commons Attribution 4.0 International License (CC BY 4.0) and the Apache License, Version 2.0.