Nvidia RAG Blueprint

NVIDIA RAG Blueprint

Autonomy Principles

• Auto-detect everything: GPU, VRAM, drivers, Docker, CUDA, disk, OS, ports, existing services, NGC key, repo state.
• If it can be checked with a command, check it — don't ask the user.
• Ask only when user action is required: providing an API key, confirming data deletion, or choosing between equally valid options.
• Once analysis is done, route to the correct workflow and execute.

Intent Detection

Determine what the user wants and route immediately:

If the intent is ambiguous, infer from context (e.g., "RAG isn't working" → troubleshoot; "get RAG running" → deploy). Only ask if genuinely unclear.

Configure

Requires a running RAG deployment. If services are not running, deploy first via references/deploy.md.

Match the user's request to a reference file, then read and follow it:

Configure Flow

1. Match the user's request to a reference file from the table above.

2. Detect what's running:

   echo "=== NIM ===" && docker ps --format '{{.Names}}' 2>/dev/null | grep -iE '(nim-llm|nemoretriever-embedding|nemoretriever-ranking|nemo-vlm|nemotron-vlm)' || echo "NO_LOCAL_NIMS"; echo "=== RAG ===" && docker ps --format '{{.Names}}' 2>/dev/null | grep -iE '(rag-server|ingestor-server|milvus)' || echo "NO_DOCKER_RAG"; echo "=== K8S ===" && kubectl get pods -n rag 2>/dev/null | head -5 || echo "NO_K8S"; echo "=== LIBRARY ===" && ps aux 2>/dev/null | grep -E '(nvidia_rag|uvicorn.*rag)' | grep -v grep || echo "NO_LIBRARY"
   

3. Use this table to determine platform, deployment type, and where config lives:

   Local NIMs running?	RAG services running?	Deployment Type	Config Location
   Yes (Docker)	Any	Self-hosted	deploy/compose/.env
   No	Yes (Docker)	NVIDIA-hosted	deploy/compose/nvdev.env
   Yes (K8s pods)	Any	Self-hosted	values.yaml (NIM sections)
   No	Yes (K8s pods)	NVIDIA-hosted	values.yaml (envVars)
   —	Library processes	Library mode	notebooks/config.yaml
   No	No	Not running	Deploy first via references/deploy.md

   Tell the user what you detected and ask to confirm. Example: "I see local NIM containers running (nim-llm-ms, nemoretriever-embedding-ms) — this is a self-hosted deployment. Config file is deploy/compose/.env. Correct?"

4. Check current feature state before changing anything — read the config location from step 3, then cross-check the live service:

   • Docker: docker exec rag-server env 2>/dev/null | grep -E "<VAR_NAME>"
   • Helm: kubectl get pod -n rag -l app=rag-server -o jsonpath='{.items[0].spec.containers[0].env}' 2>/dev/null

   If the config file and live service disagree, tell the user the service has stale config and will need a restart.

5. If the feature needs extra GPUs, check availability against hardware restrictions (see below):

   nvidia-smi --query-gpu=index,name,memory.total,memory.used --format=csv,noheader 2>/dev/null || echo "NO_GPU"
   

6. Read the reference file and apply changes:

   • Docker: edit the env file (uncomment to enable, re-comment to disable — the env file is the source of truth). Then restart the affected service:

     source <env-file> && docker compose -f deploy/compose/<compose-file> up -d
     

     Service	Compose File
     rag-server	docker-compose-rag-server.yaml
     ingestor-server	docker-compose-ingestor-server.yaml
     milvus, etcd, minio	vectordb.yaml
     NIM containers (LLM, embedding, ranking, VLM, OCR)	nims.yaml
     guardrails	docker-compose-nemo-guardrails.yaml
     observability (Grafana, Prometheus, Zipkin)	observability.yaml

   • Helm: edit values.yaml, then upgrade: helm upgrade rag <chart> -n rag -f values.yaml
   • Library: edit notebooks/config.yaml, then restart the Python process

7. Verify:

   • Docker: docker ps --format "table {{.Names}}\t{{.Status}}" | head -20; curl -s http://localhost:8081/v1/health?check_dependencies=true 2>/dev/null | head -1
   • Helm: kubectl get pods -n rag; kubectl rollout status deployment/rag-server -n rag --timeout=120s
   • Library: curl -s http://localhost:8081/v1/health 2>/dev/null | head -1

8. If restart fails, read references/troubleshoot.md. If multiple features requested, repeat from step 1 for each.

When User Says "Configure" Without Specifics

Run steps 2–3 above, then read the identified config file to list what's currently enabled:

grep -E "^(export )?(ENABLE_|APP_)" <config-file> 2>/dev/null | sort

Summarize what's running and enabled, then ask which feature to change.

Hardware Restrictions

Read docs/support-matrix.md for current GPU requirements per deployment mode. Read docs/service-port-gpu-reference.md for port mappings and GPU assignments.