# Vss Troubleshoot > Diagnose a running or failing video-search-and-summarization deployment. Probes Pipeline Manager health and feature/config endpoints to detect whether the backend is up and which mode is live, then runs structured cross-service triage grounded in setup.sh, Docker Compose files, health routes, and OVMS config. Use when users say "is vss up", "what mode is running", "check vss health", "debug vss", "VSS isn't working", "OVMS won't start", "no summary appears", "search returns nothing", containers are crash-looping, healthchecks fail, or ports are conflicting in the VSS sample app. - Skill: `open-edge-platform/vss-troubleshoot` (Agent Skill, multi-file: 15 files) - Install (CLI): `npx skillmds@latest add open-edge-platform/vss-troubleshoot` - Raw SKILL.md: https://api.skillmd.com/api/skills/open-edge-platform/vss-troubleshoot/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: open-edge-platform (https://skillmd.com/u/open-edge-platform) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/open-edge-platform/vss-troubleshoot --- # VSS Troubleshoot Use this skill to diagnose a broken Video Search & Summarization deployment without guessing. The app is started with `source setup.sh --summary`, `--search`, `--summary --search`/`--dual`, or `--summary-and-search`/`--unified`; stopped with `source setup.sh --down`; user data reset with `source setup.sh --clean-data`. ## Environment setup (run first) This skill drives the Video Search & Summarization app through its real source files, so the VSS application must be present and you must run commands from its app root. **Do this before anything else**, and it works whether or not the VSS source is already in your workspace. Run the bundled bootstrap. It first tries to find an existing VSS checkout - walking up from the current directory and inspecting the enclosing git repo - and reuses it **without ever re-cloning**. Only when no checkout is found does it do a shallow, single-branch, sparse checkout of just `sample-applications/video-search-and-summarization` from `main`. It prints the resolved app root on stdout: ```bash # SKILL_DIR is THIS skill's own directory (shown to you when the skill loads); # in-repo it is .github/skills/vss-troubleshoot. Works the same if the skill is installed standalone. SKILL_DIR=".github/skills/vss-troubleshoot" APP_ROOT="$(bash "$SKILL_DIR/scripts/vss-bootstrap.sh")" cd "$APP_ROOT" ``` Every command below assumes the working directory is this `APP_ROOT`. To pull from a fork/branch or reuse a specific checkout dir, override `VSS_REPO_URL`, `VSS_REPO_BRANCH`, or `VSS_CLONE_DIR` before running it. ## First collect status Run the read-only collector: ```bash "$SKILL_DIR/scripts/triage.sh" ``` It prints Docker Compose/container status, tails recent logs, curls key health endpoints, checks documented host ports, and reports GPU/NPU device visibility. This matters because most failures are dependency chains: `pipeline-manager` depends on storage/database/search/summary services, and UI symptoms often originate in OVMS, vLLM, EVAM, VDMS, MinIO, RabbitMQ, or Postgres. If Docker Compose cannot resolve services, run from the app root and compare with setup's own generated config: ```bash source setup.sh --summary config # or --search config / --summary-and-search config ``` ## Quick health & mode check Before diving into the decision tree, confirm whether the backend is even up and which mode is live. Set `HOST=http://${HOST_IP:-localhost}:${APP_HOST_PORT:-12345}` and **run each command yourself**, then relay the result. If nothing is deployed, hand off to the `vss-deploy` skill at `.github/skills/vss-deploy/SKILL.md`. ```bash # 1. Is the Pipeline Manager reachable? curl -sf --max-time 5 "$HOST/manager/health" && echo " ← Pipeline Manager healthy" \ || echo "UNREACHABLE - backend down or wrong HOST_IP/APP_HOST_PORT" # 2. Which capabilities/mode are live, and the resolved config curl -s "$HOST/manager/app/features" | jq . # search/summary flags curl -s "$HOST/manager/app/config" | jq . # resolved system config # 3. Subsystem probes curl -s "$HOST/metrics-manager/health" # optional live metrics service curl -s "$HOST/manager/audio/models" | jq . # whisper models (summary modes) curl -s "$HOST/manager/pipeline/evam" | jq . # EVAM pipeline status ``` `app/features` returns **string flags**, not booleans - `{"summary":"FEATURE_ON","search":"FEATURE_OFF"}` - so test against the string (e.g. `jq -e '.search=="FEATURE_ON"'`). Use it to decide which workflow applies: `vss-search-index` needs `search==FEATURE_ON`; `vss-summarize-video` needs `summary==FEATURE_ON`. A backend that 404s on `/manager/health` while the model servers (`ovms-service`, `vllm-cpu-service`, embedding server) are still loading is usually **starting**, not broken - wait and re-probe. ## Decision tree ### 1. Containers are missing, stopped, unhealthy, or crash-looping Check these exact services first: `nginx`, `pipeline-manager`, `postgres-service`, `minio-service`, `ovms-service`, `vllm-cpu-service`, `vllm-xpu-service`, `video-ingestion`, `audio-analyzer`, `rabbitmq-service`, `video-search`, `vdms-vector-db`, `multimodal-dataprep`, `vector-retriever`, `multimodal-embedding-serving`, and optional `metrics-manager`. Why: Compose `depends_on` gates many services on health. For example, summary mode needs `ovms-service` or `vllm-cpu-service`, `video-ingestion`, `rabbitmq-service`, and `audio-analyzer`; search mode needs `multimodal-dataprep`, `vector-retriever`, and `multimodal-embedding-serving` healthy. Actions: - Read the first failing dependency's logs from `triage.sh`; later services often fail only because they waited for it. - Verify required environment variables from `setup.sh`: MinIO, Postgres, RabbitMQ credentials; `VLM_MODEL_NAME`, `ENABLED_WHISPER_MODELS`, `OD_MODEL_NAME` for summary; `MULTIMODAL_EMBEDDING_MODEL` for search; `TEXT_EMBEDDING_MODEL` for unified mode. - If containers start but app state is corrupt, only then consider `source setup.sh --clean-data` (this deletes Docker volumes listed by setup, including MinIO/Postgres/VDMS/data-prep data). ### 2. Port conflict or UI unreachable Default host ports from `setup.sh`/Compose: - UI/nginx `12345`; pipeline-manager `3001`; search-ms `7890` - OVMS REST/gRPC `8300`/`9300`; vLLM `8200`; EVAM `8090`; audio `8999` - RabbitMQ AMQP/management/MQTT `5672`/`15672`/`1883` - MinIO API/console `4001`/`4002`; Postgres `5432`; VDMS `55555`; multimodal-dataprep `6016`; vector-retriever `6008`; embedding service `9777`; telemetry `9273` - Model-download REST `8640` is loopback-only and transient while `setup.sh` downloads missing summary-path models; it should not remain running afterward. Why: Compose publishes these host ports. If another process owns one, the container may fail to bind or the UI may talk to the wrong service. Actions: - Use the port section in `triage.sh` to identify listeners. - Stop the conflicting process or override the corresponding environment variable before rerunning `source setup.sh ...`. - Curl `http://localhost:3001/health` for `pipeline-manager` and `http://localhost:7890/health` for `video-search` when applicable. ### 3. OVMS will not start or final summary is stuck Inspect `ovms-service` logs and `ov_models/ovms/config.json`. Converted models live under `ov_models/ovms/openvino_models///`; setup registers storage-aware names such as `Qwen_Qwen2.5-VL-3B-Instruct_CPU_int8`. Why: `pipeline-manager` sends VLM/LLM requests to `http://ovms-service/v3` when `ENABLE_VLLM` is false. If OVMS is unhealthy, summary jobs can remain `Ready` or `In Progress`. Likely fixes: - Incomplete or incompatible host model cache: identify the affected model entry and directory first, stop VSS, then remove only that model directory and rerun setup so model-download recreates it. Do not delete all `ov_models/` content unless the user accepts re-downloading every model. - Token limit error like prompt tokens + max tokens exceed model length: lower `PM_SUMMARIZATION_MAX_COMPLETION_TOKENS` below the default `4000`, or use a model with a larger context window. - `CL_OUT_OF_RESOURCES` or cache at 100%: split VLM/LLM across CPU/GPU, use smaller/quantized models, or tune `OVMS_CACHE_SIZE_GB` cautiously. - NPU errors: verify the model supports NPU; otherwise set `VLM_TARGET_DEVICE=CPU` or another supported device. ### 4. Setup fails during model download The model-download container runs before Compose only when an OD artifact or an OVMS VLM/split LLM artifact is missing. Inspect the error's `ov_models/model-download-*.log`, or `docker logs vss-model-download` while the job is still running. Verify `MODEL_DOWNLOAD_IMAGE`, proxy settings, optional Hugging Face token, selected model/device/precision, free disk space, and that `MODEL_DOWNLOAD_HOST_PORT` (default `8640`) is available. Increase `MODEL_DOWNLOAD_JOB_TIMEOUT` from its `5400` second default only when a valid download/conversion legitimately needs longer. ### 5. vLLM backend fails When `ENABLE_VLLM=true`, setup adds `compose.vllm.yaml`, starts `vllm-cpu-service` on host port `8200`, and points VLM/LLM APIs to `http://vllm-cpu-service:8000/v1`. Experimental `ENABLE_VLLM_GPU=true` instead adds `compose.vllm.xpu.yaml`, starts `vllm-xpu-service` on the same default host port, and points both APIs to `http://vllm-xpu-service:8000/v1`. Why: In vLLM mode OVMS is not the active inference backend. Debugging OVMS logs will not explain vLLM request failures. Actions: check the active vLLM service's `/health` endpoint and logs for model download/context/cache problems. Verify `VLM_MODEL_NAME`, `HUGGINGFACE_TOKEN`, and `VLLM_MAX_MODEL_LEN`; for CPU also inspect `VLLM_CPU_KVCACHE_SPACE`, and for XPU inspect device visibility and `VLLM_GPU_MEM`. ### 6. DLStreamer/EVAM pipeline errors or ingestion stalls Check `video-ingestion` health (`http://localhost:8090/pipelines`) and logs, then `rabbitmq-service` and `minio-service` health/logs. Why: summary ingestion uses DLStreamer Pipeline Server/EVAM to process video, publishes over RabbitMQ MQTT port `1883`, and stores media through MinIO. If any of those fail, no chunks reach downstream summarization. Actions: - Confirm `OD_MODEL_NAME` is a generic YOLO id supported by the model-download Ultralytics plugin and that the IR exists under `ov_models/object-detection/ultralytics/public//FP32/`. - Check GPU/device visibility if `EVAM_DEVICE` or detection uses accelerators. - Verify RabbitMQ credentials match `RABBITMQ_USER`/`RABBITMQ_PASSWORD` and MinIO credentials match `MINIO_ROOT_USER`/`MINIO_ROOT_PASSWORD`. ### 7. No summary appears Follow this order: `pipeline-manager` health → Postgres health → MinIO health → `video-ingestion` → RabbitMQ → `audio-analyzer` → inference backend (`ovms-service` or `vllm-cpu-service`). Why: `pipeline-manager` persists job state in Postgres, uses MinIO for assets, EVAM/RabbitMQ for video events, audio-analyzer for transcripts, and OVMS/vLLM for captions/final summaries. Actions: - Look for `Ready`/`In Progress` stuck states and correlate with OVMS/vLLM logs. - For hallucinated or poor final summaries, try a larger `VLM_MODEL_NAME`; smaller models may have insufficient capacity. - On OpenCV/OpenGL/Mesa errors in summary/video processing, install `libgl1-mesa-dri libgl1-mesa-dev`, remove `ov_models/` if needed, redeploy, and retest. ### 8. Search returns nothing Check `video-search` (`http://localhost:7890/health`), `multimodal-dataprep` (`/v1/dataprep/health` on port `6016`), `vector-retriever` (`/ready` on port `6008`), `vdms-vector-db` (`55555`, or the Milvus stack when `VECTORDB_BACKEND=milvus`), `multimodal-embedding-serving` (`9777`), MinIO, and whether videos were actually ingested. Why: search requires embeddings generated by `multimodal-dataprep`, stored in the active vector database under `VS_INDEX_NAME` (`video_frame_embeddings` for search, `video_summary_embeddings` for unified). `video-search` does not query the vector DB itself — it delegates all similarity search to `vector-retriever`, which embeds the query via `multimodal-embedding-serving` and reads the vector DB, then `video-search` aggregates the returned frames into ranked videos. Actions: - If `MULTIMODAL_EMBEDDING_MODEL` or `TEXT_EMBEDDING_MODEL` changed, old vectors may have incompatible dimensions. Re-ingest, or reset data with `source setup.sh --clean-data` and rerun the correct setup mode. - Confirm the `vector-retriever` container is healthy and matches the backend: `vector-retriever-vdms` for VDMS, `vector-retriever-milvus` for Milvus (selected by `VECTORDB_BACKEND`). A mismatch, or a metric/index mismatch (`VDB_METRIC_TYPE`/`VDB_INDEX_TYPE`) between `multimodal-dataprep` and `vector-retriever`, yields empty results. - Check `vector-retriever` logs for the embedding call to `multimodal-embedding-serving` and the vector-DB read; check `video-search` logs for the delegation call to `http://vector-retriever:8000/query`. - Check accuracy settings: model dimensionality, `FRAME_INTERVAL`, `ENABLE_OBJECT_DETECTION`, and video diversity affect result quality. ## Log and data locations The Compose files do not define application log files; use Docker stdout/stderr via `docker logs` or `triage.sh`. Important persistent locations are Docker volumes `docker_minio_data`, `docker_pg_data`, `docker_vdms-db`, `docker_audio_analyzer_data`, `docker_data-prep`, `docker_collector_signals`; the host-backed OVMS repository at `ov_models/ovms/`; object detection models under `ov_models/object-detection/ultralytics/public/`; and failed setup logs at `ov_models/model-download-*.log`. See `references/common-failures.md` for a compact symptom/cause/fix table.