Sop Data Augmentation

SOP Data Augmentation

Overview

The Data and QA Augmentation microservice transforms annotated video data into structured question-answer (QA) formats used to train a Vision-Language Model (VLM) for SOP monitoring. It takes an annotated dataset (video action chunks with timestamps) and generates up to 8 types of QA training data — from simple binary yes/no questions to complex hard-negative mining with shuffled video frames.

This skill covers how to configure, trigger, monitor, and troubleshoot the augmentation pipeline.

Bundled resources (paths relative to this skill's directory):

• scripts/launch_vllm.sh — Helper to start/stop a local vLLM server for GQA generation
• references/augmentation-types.md — Detailed config parameters and guidance for all 8 augmentation types
• references/config-template.md — Full annotated augment_config.yaml template
• references/launch-vllm-usage.md — vLLM script options and Docker details

Prerequisites

Before running augmentation, ensure:

1. Service is running: The sop-data-gen container must be up on port 5487

   # Verify service health
   curl http://<server_ip>:5487/health
   # Expected: {"message": "VLM Data Augmentation API is running", "status": "healthy"}
   

2. Annotated dataset exists: The dataset must be in assets/data/<dataset_id>/ with:

   • actions.json — action definitions (e.g., {"actions": ["(1) standing by...", "(2) installing..."]})
   • One or more video folders (e.g., video-1-pass/) each containing:
     • Action video chunks (mp4 files named <action_number>_<video_name>_<duplication>_<timeline>.mp4)
     • <video_name>_annotation.json — frame-by-frame annotations

3. Config file prepared: assets/config/augment_config.yaml configured with desired parameters

4. LLM access (for GQAs stage only): The GQAs stage requires an LLM. Two options:

   • NVIDIA NIM API: Set NGC_API_KEY in .env and configure llm_type: "nvidia" in the config
   • Local LLM (no API key needed): Use the helper script scripts/launch_vllm.sh to start a local vLLM server, then configure llm_type: "local" in the config. See the workflow below for details.

Workflow

If the user has an NVIDIA API key (NGC_API_KEY set in .env), use the NVIDIA NIM path. If not, use the local LLM path — start vLLM before augmentation and stop it after to free GPU memory. Note: The NGC_API_KEY might set to a placeholder, do check if the value is a valid NVIDIA API key before deciding to use NVIDIA NIM

Configuration policy (read before editing the config)

Treat the user's existing assets/config/augment_config.yaml as authoritative for which stages are enabled. Only modify the minimum settings required to satisfy the request.

                                     ┌─ Has NGC_API_KEY? ─┐
                                     │                     │
                                    yes                    no
                                     │                     │
                                     │         1a. Start local vLLM
                                     │             scripts/launch_vllm.sh
                                     │             (wait for ready)
                                     │                     │
                                     └──────────┬──────────┘
                                                │
                              1b. Edit augment_config.yaml
                                                │
                              2.  POST /api/v1/augment
                                                │
                              3.  GET /api/v1/augmentation_status
                                  (poll until 100%)
                                                │
                              4.  Verify output
                                                │
                                     ┌─ Used local LLM? ──┐
                                     │                     │
                                    no                    yes
                                     │                     │
                                     │         5. Stop vLLM
                                     │            scripts/launch_vllm.sh --stop
                                     │            (frees GPU for training)
                                     │                     │
                                     └──────────┬──────────┘
                                              Done

Step 1a: Start Local LLM (skip if using NVIDIA NIM API)

If the user does not have an NVIDIA API key, start a local vLLM server before editing the config:

# Default model on port 9000 (takes ~6 min to load)
scripts/launch_vllm.sh

# Or with custom model/port:
scripts/launch_vllm.sh --model Qwen/Qwen2.5-7B --port 8000 --tp 2

When ready, the script prints a ready-to-paste gqas: config snippet (with the actual served model id, machine IP, and port). Capture that snippet — Step 1b uses it verbatim.

For full script options, see references/launch-vllm-usage.md.

Step 1b: Configure

Edit assets/config/augment_config.yaml on the host machine. The file is mounted into the container — no rebuild needed.

Per the Configuration policy above: only change fields that are required for the request.

For local LLM, ensure the gqas section has:

gqas:
  llm_type: "local"
  local_llm_url: "http://<vllm_machine_ip>:<port>/v1"   # from launch_vllm.sh output
  llm: <served_model_id>                                # exact id from /v1/models — do not guess
  enable_thinking: "false"                              # required for thinking-mode models

Step 2: Trigger

curl -X POST \
  'http://<server_ip>:5487/api/v1/augment?label_data_id=<dataset_id>' \
  -H 'accept: application/json' \
  -d ''

The label_data_id is the dataset folder name under assets/data/. This is the original annotated dataset ID (not an augmented one).

Step 3: Poll Status

curl 'http://<server_ip>:5487/api/v1/augmentation_status/<augmented_dataset_id>'

The augmented_dataset_id is returned by the POST request (format: <label_data_id>_augmented_<N>).

Step 4: Verify Output

Check the output directory exists and contains expected subdirectories:

ls assets/data/<augmented_dataset_id>/
# Expected: bcq/ mcq/ golden_gqa/ gqas/ (and dmcq/ ds/ en/ wmcq/ if those stages were enabled)

Step 5: Stop Local LLM (skip if using NVIDIA NIM API)

After augmentation completes, stop the vLLM server to free GPU memory for downstream tasks (e.g., VLM fine-tuning, DDM training):

scripts/launch_vllm.sh --stop

API Reference

POST /api/v1/augment

Trigger a data augmentation job. Runs asynchronously — returns immediately.

Response (200):

{
  "dataset_id": "<label_data_id>_augmented_<N>",
  "message": "Augmentation actions submitted successfully"
}

Errors:

• 400 — Missing label_data_id, dataset path not found, actions.json missing, or no video folders
• 500 — Internal server error

GET /api/v1/augmentation_status/{dataset_id}

Poll augmentation progress.

Response (200):

{
  "dataset_id": "<augmented_dataset_id>",
  "status": "running",
  "progress": 50.0
}

Status values: pending | running | completed | failed

Progress: Float 0.0–100.0 (percentage of stages that finished or failed). With default 4 stages: 25% = 1 done (BCQ), 50% = 2 done (+MCQ), 75% = 3 done (+Golden GQA), 100% = all done (+GQAs).

GET /api/v1/augmented_datasets

List all completed augmented datasets with statistics.

Response (200):

{
  "<augmented_dataset_id>": {
    "status": "completed",
    "video_count": 3,
    "total_clips": 12
  }
}

GET /health

Health check.

Response (200):

{
  "message": "VLM Data Augmentation API is running",
  "status": "healthy"
}

Augmentation Types

8 types, executed sequentially. Each independently enabled/disabled in config. Default: first 4 enabled.

Stages 5-8 require non_sop_action (the "none of the above" action index from your actions.json).

Stages 1-7 consume the pre-cut action chunks from the video split. Stage 8 (WMCQ) does not — it reads the source videos and their annotations and cuts its own windows, so merge_small_chunks does not apply to it.

Reach for WMCQ when DDM temporal chunking performs badly and inference falls back to uniform chunking. The other stages train on exactly-trimmed action chunks, which match inference only when the chunk boundaries are accurate. When they are not, inference feeds the VLM fixed-length sliding windows — a key-step buried in surrounding footage — which the model has never been trained on. WMCQ cuts training clips with that same geometry. If DDM chunking is good, WMCQ adds redundant data; diagnose the chunking first.

WMCQ assumes long videos with sparse key-steps — brief procedure steps separated by long non-SOP stretches. It needs enough non-SOP footage to draw negatives from, and enough spacing that a window containing one key-step does not also contain the next. It is not a general action-recognition augmentation: for densely-packed actions the single-label window it emits is the wrong shape. See references/augmentation-types.md, "What WMCQ assumes about your data".

For detailed config parameters, examples, hard mode explanations, and guidance on when to enable each type, read references/augmentation-types.md.

Config File Reference

The config file is at assets/config/augment_config.yaml (mounted volume — edit on host, no rebuild needed).

Key concepts:

• non_sop_action: Action index for "none of the above." Required for dynamic_mcq, dynamic_shuffling, extra_negative, wmcq.
• window (wmcq): Must equal the evaluation sliding-window length. Nothing validates this and a mismatch fails silently — see references/augmentation-types.md.
• exclude_action: Underscore-separated indices to skip (e.g., "1_2" excludes actions 1 and 2).
• extra_negative_data_id: Must be a different annotated dataset ID already in assets/data/.

For the full annotated config template with all parameters, read references/config-template.md.

Reference Configurations

These are templates for first-time setup, not actions to take when a config already exists. If assets/config/augment_config.yaml is already populated, see the Configuration policy in the Workflow section.

Template: 4-stage starter (BCQ + sequential MCQ + golden GQA + GQAs)

Produces well-rounded training data without requiring non_sop_action setup. Use this only when initializing a fresh config.

bcq:
  enable: true
sequential_mcq:
  enable: true
golden_gqa:
  enable: true
gqas:
  enable: true

Template: All 8 stages

Enable all stages. Requires non_sop_action and (for extra_negative) a second annotated dataset.

Set non_sop_action in dynamic_mcq, dynamic_shuffling, extra_negative, and wmcq to your dataset's "none of the above" action index. Set extra_negative_data_id to a different dataset's ID.

Start conservative: num_pos: 1, num_neg: 2 for dynamic_mcq. num_runs: 1 for dynamic_shuffling and extra_negative.

Template: Poor DDM chunking / uniform chunking at inference

Add WMCQ on top of the 4-stage starter so the VLM sees clips shaped like the windows it will actually be given. Set window to the evaluation sliding-window length — a mismatch silently defeats the whole stage.

wmcq:
  enable: true
  non_sop_action: <your "none of the above" index>
  window: 3.0            # = evaluation sliding-window length

Everything else is already defaulted to the configuration our own runs converged on — tiling on, variants: 4, neg_ratio: 1.5. Turn tile_long off only if you specifically want the enlarge behaviour: without it, key-steps longer than window are stretched to fit, which makes clip duration correlate with the action — a shortcut that does not exist at inference. The stage warns when this happens.

Template: Local LLM for GQAs

To use a self-hosted vllm server instead of NVIDIA NIM. Replace <served_model_id> with the exact id reported by curl <local_llm_url>/models:

gqas:
  enable: true
  llm_type: "local"
  local_llm_url: "http://<llm_server_ip>:<port>/v1"    # Must include /v1
  llm: <served_model_id>                                # exact id from /v1/models — do not guess
  enable_thinking: "false"                              # Set for thinking-mode models

Output Structure

After successful augmentation, a new directory is created:

assets/data/<dataset_id>_augmented_<N>/
  bcq/                  # Binary choice QA
    videos/             # Copied video files
    bcq.json            # LLaVA-format annotations
  mcq/                  # Sequential multiple choice QA
    videos/             # Includes merged multi-action videos
    mcq.json
  golden_gqa/           # Golden grounded QA
    videos/
    golden_gqa.json
  gqas/                 # LLM-expanded QA
    videos/
    gqas.json
    GQA2GQAs/           # Raw LLM outputs per action per video
  dmcq/                 # Dynamic MCQ (if enabled)
    videos/
    dmcq.json
  ds/                   # Dynamic shuffling (if enabled)
    videos/             # Contains generated shuffled videos
    ds.json
  en/                   # Extra negative (if enabled)
    videos/
    en.json
  wmcq/                 # Window-matched MCQ (if enabled)
    videos/             # Fixed-length windows cut from the source videos
    wmcq.json

Each annotation JSON follows LLaVA format:

[
  {
    "id": 0,
    "conversations": [
      {"from": "human", "value": "<video>\n<question>"},
      {"from": "gpt", "value": "<answer>"}
    ],
    "video": "videos/<filename>.mp4"
  }
]

Troubleshooting

Check Docker logs:

docker compose logs sop-data-gen --tail 100 -f

Important Notes

• Augmentation is async. The POST API returns immediately with the augmented dataset ID. You must poll augmentation_status to track completion.
• Config is hot-reloadable. The config YAML is a mounted volume — edit on host, changes take effect on next augmentation run without rebuilding the container.
• One failure = all fail. If any enabled stage fails, all stages are marked as failed and the entire output directory is deleted. Check logs for the root cause.
• Output ID auto-increments. The augmented dataset ID is <label_data_id>_augmented_<N> where N starts at 0 and increments with each new augmentation of the same source dataset.
• Advanced stages need non_sop_action. Dynamic MCQ, Dynamic Shuffling, Extra Negative, and WMCQ all require the non_sop_action index to be correctly set to your dataset's "none of the above" action.
• WMCQ needs the source videos, not just the chunks. It reads <dataset>/<video_name>.mp4 alongside <dataset>/<video_name>/<video_name>_annotation.json. It is unaffected by merge_small_chunks.
• WMCQ's window is not validated. It must match the evaluation sliding-window length. If it does not, the stage still succeeds and still reports healthy sample counts while producing clips that no longer match eval geometry — the one thing it exists to do. Check the stage log: it prints the window it used and the clip-duration spread per action.
• Extra Negative needs a second dataset. The extra_negative_data_id must point to a different annotated dataset that is already in assets/data/.
• GQAs is the only LLM-dependent stage. All other stages are deterministic and run locally without external API calls.
• Adjust min_options/max_options to your action count. For a dataset with N actions, max_options should not exceed N.