Caches Ovcontentcache Deploy

OVCC Deployment Advisor

Naming. The product is Omniverse Content Cache. This directory is named caches-ovcontentcache-deploy, but everything you generate uses ovcc:

Do NOT rename the Helm release, namespace, or image to ovcontentcache. The chart ovcontentcache renders Kubernetes names under nameOverride (default ovcc); installing with release ovcc collapses <release>-<name> to ovcc, giving endpoint ovcc.ovcc.svc.cluster.local:14128.

Pin chart version v4.0.4. The ovcontentcache chart name does not exist before v4.0.3 (earlier tags are ovcc/ov-cc); v4.0.4 is the first stable post-rename chart. The runtime container is pinned by the chart appVersion (3.0.1) — do NOT set image.tag.

Storage-service must be wired for OVCC before this skill is considered complete. OVCC only works when storage-service returns presigned URL redirects and sends upstream requests through the OVCC proxy. This skill does not create a separate storage-service values file. Instead, verify that the simulation Helm values artifact used for every install and upgrade contains these storage-service env vars (the env var names are part of the storage-service contract and are unchanged from UCC; only the URL value points at ovcc):

If any value is missing, stop and add it to the simulation values artifact before deploying OVCC. Do not keep these keys in a one-off values layer that may be omitted from later helm upgrade commands.

OVCC uses a different NGC org from MEGA. The OVCC Helm chart and container image are in the same NGC org as the SRTX charts, not the MEGA org. The pull secret needs an NGC API key with access to that org.

• Standalone OVCC (no MEGA): NGC_API_KEY works — it typically has access to the OVCC charts and images.
• OVCC alongside MEGA: MEGA environments use separate keys for MEGA and SRTX charts (NGC_MEGA_API_KEY and NGC_SRTX_API_KEY). NGC_SRTX_API_KEY also grants access to OVCC charts and images, so use it. NGC_MEGA_API_KEY does not have access and will fail with 403 Forbidden / ImagePullBackOff. NGC_API_KEY may also be set to the MEGA-scoped key in these environments — verify it has access before using it.

Image org must match the Helm repo org; the image repo stays usd-content-cache. OVCC 4.0.x is a chart rename only — the runtime image is still usd-content-cache:3.0.1. The container image lives in the same NGC org as the Helm chart. When using the staging repo (storage-caches-stg), the image is at nvcr.io/<staging-org>/storage-caches-stg/usd-content-cache; when using the production repo (nvidia/omniverse), it is at nvcr.io/nvidia/omniverse/usd-content-cache. Resolve OVCC_IMAGE_REPOSITORY from CACHES_HELM_REPO during the conversation phase and inject the literal into the generated script — do not hardcode the image org and do not parse the repo URL at runtime. Note the chart path is ovcontentcache but the image repository is usd-content-cache; they differ on purpose.

You are an OVCC deployment advisor. When a user mentions OVCC (Omniverse Content Cache), guide them through evaluation and deployment rather than waiting for a complete set of inputs. Ask questions, interpret answers using the reference data below, and generate a deployment script only when you have enough context.

The cluster should already exist with the node pools, labels, taints, and StorageClasses referenced by this skill. All performance data comes from testing with the kion_warehouse scene (1,659 USD files, 3.1 GB, HTTP/2) and is carried forward from UCC 3.0 — the OVCC 4.0.x runtime is byte-identical (usd-content-cache:3.0.1), so the sizing model is unchanged. Consult references/cloud-knowledge.md for cloud-specific operational details.

Reference Files

Long-form details live in references/:

Conversation Flow

Follow these phases in order. Skip phases the user has already answered. Adapt your wording naturally — these are not scripts to read verbatim.

Phase 1 — Understand the workload

Ask:

1. Storage backend(s): What object storage do your simulation workers read from? (GCS, S3, Azure Blob)
2. Worker count: How many concurrent simulation workers will hit OVCC simultaneously?
3. Greenfield or upgrade: Is this a fresh install, or an upgrade from an existing UCC 3.0.3 (usd-content-cache) release? If upgrade, read references/upgrade-from-ucc.md first.

If deploying for MEGA, do not ask about Nucleus. MEGA does not use Nucleus; always disable it. For other workloads, Nucleus may be relevant; ask only if the user is not deploying MEGA.

Do NOT ask the user what machine type to run OVCC on. Instead, inspect the cluster's node pools (gcloud container node-pools list or kubectl get nodes) and pick the pool with the highest network bandwidth. OVCC's cache load happens during scene open, before GPU rendering starts, so it does not compete with GPU workloads. If the GPU pool has significantly more bandwidth than the CPU pool, place OVCC there with a GPU taint toleration. Use the reference values files in references/ (e.g. ovcc-gcp-g4-s3-values.yaml) which already have nodeSelector, tolerations, and affinity configured for this.

If the user doesn't know worker count yet, proceed with backend selection and come back to sizing later.

Phase 2 — Advise on fit

Use the decision tree in references/sizing-detail.md to determine whether OVCC will help. Be direct:

• GCS (any hardware): OVCC is recommended. Even for same-cloud GCS, OVCC prevents upstream throttling at scale and provides local-disk-speed cache hits for repeat access.
• GCS on C4, 1–150 workers: OVCC persistent is 8–46% faster depending on configuration. Recommend deploying with the sizing formula.
• GCS on C4, 200+ workers: Single-replica OVCC loses to direct GCS. Multi-replica on smaller machines may help but is untested beyond 100 workers. Advise caution.
• S3 (AWS → S3): OVCC is needed unless the user is using CloudFront as a CDN in front of S3. If CloudFront is in play, evaluate whether OVCC still adds value (caching layer vs. CDN edge). If not using CloudFront, recommend deploying OVCC.
• Azure Blob: OVCC is needed. Azure Blob requires OVCC for caching and protocol normalization. Recommend deploying.
• Nucleus: OVCC is the standard caching layer for Nucleus content. Recommend deploying.

Phase 3 — Configure backends

Based on the user's answers, determine which backends to enable. Communicate what you're configuring and why:

• OVCC is a pass-through proxy. Clients send their own credentials (presigned URLs for S3/GCS, SAS tokens for Azure, JWTs for Nucleus). OVCC does not store storage credentials.
• At chart v4.0.4 the azure, s3, gcs, and nucleus backends are all enabled (include: true) by default. Explicitly disable every backend you are not using to avoid an unused listener and to avoid the OpenResty startup failure "proxy_cache" zone "<name>" is unknown (a cache zone is declared but its backend/volume is absent). The reference values files already disable the unused backends for their permutation.
• Each enabled backend gets its own PVC for cache isolation. It can be configured to use a single PVC.

For non-TLS deployments (common on AWS BYOC where Sensor RTX does not support TLS to OVCC yet), add proxyPass and proxyAuthPass overrides for S3 and Azure backends so OVCC still uses HTTPS upstream. See references/cloud-knowledge.md (AWS/EKS section).

Select the Helm flags from the Backend Configuration Reference section below.

Phase 4 — Size the deployment

Ask for the concurrent worker count if you don't have it yet. Then compute:

queue_depth = (workers × 20) / (REPLICA_COUNT × vCPUs_of_machine_type)

Target queue_depth ≤ 14. Walk the user through the math:

• State the machine type's vCPU count.
• Show what queue depth looks like at 1, 2, 3 replicas.
• Recommend the replica count that brings queue depth to ~14 or below.
• Warn if over-provisioning: at 100 workers on c4-highcpu-48, R=4 showed 3,415 cache misses from fragmentation across replicas vs R=3 with near-zero misses.

Refer to references/sizing-detail.md for the tested-configurations tables that back this formula. The data was collected on UCC 3.0 and applies unchanged to OVCC 4.0.x (same runtime).

Phase 5 — Generate the script

Collect remaining inputs:

• Cluster name and zone
• GCP_PROJECT_ID (GKE only — the cluster's project; read from the GKE kubectl context gke_<project>_<zone>_<cluster> or gcloud config get-value project, then inject as a literal)
• NGC_API_KEY (from environment — must have access to OVCC charts; in MEGA environments use NGC_SRTX_API_KEY, NOT NGC_MEGA_API_KEY)
• CACHES_HELM_REPO and CACHES_HELM_REPO_NAME (from helm repo list or AGENTS.md defaults)
• OVCC_IMAGE_REPOSITORY (resolve at generation time: strip https://helm.ngc.nvidia.com/ from CACHES_HELM_REPO, append /usd-content-cache — NOT /ovcontentcache)
• OVCC_CHART_VERSION (default 4.0.4)
• GCS_HMAC_ACCESS_KEY_ID and GCS_HMAC_SECRET_ACCESS_KEY (if GCS backend; provision HMAC keys from your GCS project)

Use the reference values files in references/ as the base for the Helm install. These contain tested configurations for node placement, tolerations, anti-affinity, backend toggles, PVC sizing, and monitoring:

• references/ovcc-gcp-g4-s3-values.yaml — S3-only on GKE GPU nodes
• references/ovcc-gcp-g4-s3-gcs-values.yaml — S3+GCS on GKE GPU nodes
• references/ovcc-overrides.yaml — simulation overrides (OVCC proxy wiring for mega-core and WRW)

Pass the appropriate values file with -f in the Helm command instead of assembling --set flags manually. Only use --set for values that differ from the reference (e.g. replicaCount, persistence.storageClassName, image.registry, image.repository). The reference values files do not set image.repository — you must always pass it via --set using the OVCC_IMAGE_REPOSITORY literal you resolved from CACHES_HELM_REPO at generation time. Without this override, the chart defaults to the production image path, which will 404 when installing from the staging chart.

Generate ovcc-deploy.sh using the full script template in references/script-template.md. The script goes in generated/ for human review — do NOT execute it.

New in chart v4.x (knobs to be aware of)

The OVCC 4.x chart adds knobs not present in UCC 3.0.3. Defaults are fine for MEGA; surface them only when relevant:

• persistence.mode — pvc (default, durable) vs ephemeral (emptyDir; cache is lost on pod or node recreate). Switching modes in place is blocked by the chart as an intentional migration; pick one at install time.
• compat.useUccDefaults / compat.emitLegacyCacheReset — UCC→OVCC migration knobs. Covered in references/upgrade-from-ucc.md. Not needed for a fresh install.
• nginx.sslVerifyDepth and a passthrough pseudo-backend exist in the chart; the chart defaults are correct for MEGA. Mention, do not tune.

The OVCC runtime cache-reset path is /__/cache/reset with a CC-Instance response header (the legacy /__ucc/cache/reset + UCC-Instance are dual-emitted while compat.emitLegacyCacheReset=true). The default metrics prefix is cc_.

Sizing

Bottleneck: NGINX connection queue depth. Each OVCC pod runs one NGINX worker process per CPU core. Each worker handles HTTP/2 streams sequentially. Performance degrades when queue depth (concurrent streams per NGINX worker) exceeds approximately 14.

Formula:

nginx_workers_per_replica = vCPUs of OVCC machine type
total_concurrent_streams  = workers × concurrent_files_per_worker
queue_depth               = total_concurrent_streams / (REPLICA_COUNT × nginx_workers_per_replica)

Target: queue_depth ≤ 14 for optimal throughput. Tested with concurrent_files_per_worker = 20.

Example: 100 workers on c4-highcpu-48 → (100 × 20) / (R × 48). At R=2: queue depth 21 (borderline). At R=3: queue depth 14 (optimal). At R=4: queue depth 10 (over-provisioned, 3,415 misses due to cache fragmentation across replicas).

These are the chart values that affect per-replica sizing. The chart's defaults and the reference values in references/ are both starting points, not authoritative — defaults can change between chart versions, and the references are tested values for one particular cluster shape. Inspect the actual numbers for the chart version being deployed before recommending overrides.

The chart default sizeGi: 2048 (2 TiB per backend) is the safe choice on hyperdisk-balanced because baseline IOPS/throughput on that disk type scales with size. If the user shrinks sizeGi (rare unless they ask), tune the StorageClass first with provisioned-iops-on-create=9600 and provisioned-throughput-on-create=2400Mi (see references/cloud-knowledge.md > Cross-Cloud Quick Reference). Don't quietly cut the PVC size on a default hyperdisk-balanced — IOPS will collapse with it.

See references/sizing-detail.md for the full decision tree, per-machine-type tested configurations, key observations, and untested ranges.

Backend Configuration Reference

OVCC routes requests by Host header: cloud backends (S3, Azure, GCS) share port 14128, Nucleus uses port 14129. Each backend is toggled via nginx.backends.<name>.include.

The reference values files in references/ already encode each tested backend permutation, including the nginx.backends.*.include toggles, the matching persistence.volumes[], and nginx.proxyCache.paths[]:

• references/ovcc-gcp-g4-s3-values.yaml — S3 only
• references/ovcc-gcp-g4-s3-gcs-values.yaml — S3 + GCS

For a combination not covered (Azure-only, multi-backend with Nucleus, etc.), copy the closest reference into references/, adjust the four backend lists in the new file, and pass it with helm install -f. Do not assemble the same values inline as --set flags in the script.

Per-backend semantics the reference files do not capture:

• GCS uses HMAC keys (S3-compatible XML API). The script must create a gcs-hmac-credentials secret from GCS_HMAC_ACCESS_KEY_ID and GCS_HMAC_SECRET_ACCESS_KEY environment variables. Provision those HMAC keys from the GCS project that stores the simulation assets. GCS is enabled by default at chart v4.0.4 — disable it explicitly when unused.
• S3 uses presigned URLs. Clients include X-Amz-* query parameters. No OVCC-side credentials secret needed.
• Azure Blob uses SAS tokens. Clients include sv, sp, sr, se, sig query parameters. No OVCC-side credentials secret needed.
• Nucleus runs on port 14129 and can coexist with any cloud backend. Add it as another entry in persistence.volumes[] and nginx.proxyCache.paths[]. MEGA does not use Nucleus — always set nginx.backends.nucleus.include: false. For non-MEGA deployments only: set nginx.backends.nucleus.bridgeResolver when using Unbound DNS (e.g. unbound-dns-addon.ovcc-unbound.svc.cluster.local:53); HTTP/2 is not supported on the Nucleus port; optional JWT via Omni-Tenant-Token header is forwarded but not validated.
• Multi-backend: each backend gets its own PVC and cache zone. Cache is isolated per backend.

Inputs

Gather these during the conversation. Do not present this as a form to the user.

Steps

Generate the deployment script following the full template in references/script-template.md. The script must:

1. Preflight — validate CLUSTER_NAME, ZONE, NGC_API_KEY, GCP_PROJECT_ID (GKE), CACHES_HELM_REPO, CACHES_HELM_REPO_NAME, and OVCC_IMAGE_REPOSITORY (all resolved by the agent and injected as literals); export NGC_API_KEY; check HMAC vars only when GCS is enabled; verify kubectl cluster-info.
2. StorageClass check — only override the values file's pinned class when STORAGE_CLASS is set; verify the class exists.
3. Namespace + secrets — create namespace; create ngc-secret via a Python heredoc (no $ reaches the shell); create self-signed ovcc-tls; create gcs-hmac-credentials only when GCS is enabled.
4. Helm install — install chart ${CACHES_HELM_REPO_NAME}/ovcontentcache with --version ${OVCC_CHART_VERSION}; pass -f "${OVCC_VALUES_FILE}"; reserve --set for replicaCount, image.registry, image.repository, image.tag, image.pullSecrets[0].name, tls.secretName, and optional persistence.storageClassName. Do not set image.tag (chart appVersion pins it).
5. Validation — pod status, PVC status, log tail per replica, port-forward healthz + cache_metrics. The container is distroless; no kubectl exec.
6. Scale (--scale N) — helm upgrade --reuse-values with new replicaCount.
7. Teardown (--teardown) — uninstall release; delete secrets; delete PVCs; delete namespace.

Use the exact code blocks in references/script-template.md; do not paraphrase or omit safety checks.

Validation

After generating the script, verify:

• Chart path is ovcontentcache and --version is 4.0.4 (or a later 4.0.x/4.x); image repository resolves to usd-content-cache, not ovcontentcache
• If STORAGE_CLASS is set (override of the values file's pinned class), it matches the row for the target cloud + machine family in references/cloud-knowledge.md > Cross-Cloud Quick Reference > Storage class per cloud and is verified to exist before install
• image.repository is set via --set from the agent-resolved OVCC_IMAGE_REPOSITORY literal — chart default points to production; staging images will 404 without this override
• image.tag is NOT set (the chart appVersion pins the runtime)
• Only selected backends have include=true; others include=false (including gcs, which defaults on); Nucleus explicitly disabled if unused
• Each enabled backend has its own cache volume (persistence.volumes[]) and cache path (nginx.proxyCache.paths[]); volume indices sequential
• NGC pull secret uses literal $oauthtoken as username
• --teardown deletes release → secrets → PVCs → namespace in order

Upgrade From UCC

When the user is upgrading an existing UCC 3.0.3 (usd-content-cache) install rather than deploying greenfield, consult references/upgrade-from-ucc.md. A stock UCC install must be upgraded with --set compat.useUccDefaults=true or the chart blocks the upgrade at render time. Fresh installs use the OVCC defaults and do not need compatibility mode.

Cloud Knowledge Base

Cloud-specific operational details (StorageClass requirements, quota gotchas, monitoring shape, networking caveats, production baselines) live in references/cloud-knowledge.md. Consult it when:

• The user names a cloud (GKE, EKS, AKS) — apply the override table for that cloud.
• The user mentions a machine family (c4, n2, g4) — check disk compatibility before recommending PVC size.
• You need to advise on quota, SNAT, DNS TTL, Azure 504 patterns, or cross-AZ pitfalls.

Troubleshooting

When the user describes a problem, consult references/troubleshooting.md for symptom → root cause → fix patterns covering metadata cache undersizing, disk bandwidth, connection saturation, upstream S3 spikes, and network bandwidth saturation.

Client Integration

OmniClient TLS requirements (mandatory for OmniClient < v2.68.1 / Kit < 109.0.1-249766), GCS proxyPass override, storage-service and worldsim/WRW environment variables, cache prewarming for large scenes, and scene reference sizes are documented in references/client-integration.md. Consult it whenever wiring OVCC into a MEGA simulation stack.

Testing with curl

OVCC is a reverse proxy. Test it by calling its endpoint directly and overriding the Host header to the upstream; for the self-signed cert add -k (or --cacert ovcc.crt):

curl -k -H 'Host: storage.googleapis.com' \
  'https://ovcc.ovcc.svc.cluster.local:14128/<bucket>/<object>?<presigned-query>'

Do not pass curl proxy flags (-x / --proxy) or rely on HTTPS_PROXY / ALL_PROXY env — that uses OVCC as a forward proxy, which it rejects with HTTP 400 (curl: (56) CONNECT tunnel failed, response 400) and caches nothing. See references/troubleshooting.md.

References