Nvidia Training Eval

NVIDIA-Olympus Training Skill

End-to-end experiment workflow for NVIDIA data integration into the Olympus v1.4 training pipeline. Covers the full lifecycle from checkpoint lookup through results publishing.

Reference Documents:

• Nvidia Data Specification -- sensor layout, data format, label schema
• Nvidia Data Consumption Strategy -- ingestion plan, pipeline strategy
• NVIDIA Data Training / Eval Plan -- phases, hyperparameters, comparison matrix, metrics reference
• Platform Comparison and Integration Report -- camera mapping, feature/config comparison, calibration bugs, MSL label gaps
• Local plan: ~/.cursor/plans/nvidia_olympus_v1.4_integration_8f9779e9.plan.md

Platform Context

NVIDIA data is remapped into the A110 namespace at Stage 1 featurization. Key facts:

• 7 NVIDIA cameras mapped to A110 LDM names (camera_front_medium, camera_front_right_40, camera_front_left_80, camera_front_right_80, camera_rear_medium, camera_back_left, camera_back_right). camera_front_wide_120fov is duplicated into left_40 + right_40; the 7-camera config drops left_40.
• Feature format: NVIDIA matches A110_MSL on 8/11 dimensions (prefix, extrinsics, longpos, intent, metadata, MSL labels). Map format differs: NVIDIA uses map_v2_* (NGM) vs A110's map_semantic_* (L4). Image resolution matches A110 (940x1824).
• Data source type: A110_MSL with cameras_override=get_nvidia_7_cameras()
• Architecture: FasterViT (vision) -> DimAdjust -> Accumulator -> FusionEncoderV2 (LaRa) -> GLOBAL_EMBEDDING. Map head consumes GLOBAL_EMBEDDING. No separate map encoder.
• Temporal sequence: 1.6s window, 16 ticks at 10Hz. NVIDIA uses 4-tick (BATCH_SHIFTS_PRETRAIN = [6, 4, 2, 0]), matching A110 pre-training. SC3 fine-tuning uses 8-tick (BATCH_SHIFTS_FINETUNE = [14, 12, 10, 8, 6, 4, 2, 0]). Feature Accumulator (window size=2) gives the FusionEncoder current + previous frame at each step. 4 forward passes per NVIDIA sample (vs 8 for SC3), each outputting 400 detection slots.
• Calibration fixes: Two bugs fixed on branch (transform direction inversion + optical_joint duplication). After fixes, NVIDIA flows through identical code paths as SC3/A110.
• MSL label gaps: total_num_lidar_points=0 (mitigated by min_valid_lidar_point_number=None), object_bearing_angle=None (no-op), classification fall-through (extend map in Phase 2).

Pipeline Context

The NVIDIA training step sits between A110 pre-training and SC3 fine-tuning:

Fusion Encoder Pre-training -> A110 Pre-training -> NVIDIA Fine-tune -> SC3 Fine-tune -> SC3 Eval

Five branches (4 variants + 1 baseline) run in parallel:

All 5 branches feed into SC3 fine-tune + eval on the same SC3 test set.

Key files (relative to repo root):

0. Pre-flight: Authentication

Before ANY bazel run, bq query, or gcloud command, verify auth:

gcloud auth application-default print-access-token 2>&1 | head -1

If output is a token starting with ya29.: Auth is valid, proceed.

If output contains "Reauthentication", "refresh", or "error": Use two-tier refresh:

1. Tier 1 (instant, no MFA): authcli refresh — silently refreshes all tokens using the existing Okta session. Works ~95% of the time, completes in <20 seconds.
2. Tier 2 (full SSO): If authcli status still shows INVALID after Tier 1, escalate to the cruise-auth-refresh skill (~/.cursor/skills/cruise-auth-refresh/SKILL.md).

Re-check auth whenever a command fails with: Reauthentication, 401, credentials, Permission denied, access token, RefreshError.

Persistent fix: Set up a cron job to auto-refresh tokens every 30 minutes:

(echo "*/30 * * * * $(which authcli) refresh >/dev/null 2>&1") | crontab -

Ladybug MCP auth: The Ladybug MCP uses a ladybug-mcp-dev app token fetched via authcli app get ladybug-mcp-dev -ttl 120h. The remote server has a proxy that auto-refreshes, but the initial token fetch requires a valid Okta session. If tools show as unavailable, the cron job above prevents the issue. If already broken: run authcli refresh, then reload the Cursor window (the MCP restarts and fetches a fresh token on startup).

1. Checkpoint Lookup

1a. V1.4 Deployed Checkpoints

The deployed V1.4 pipeline has 3 stages. Each stage's output checkpoint is the next stage's input.

The A110 checkpoint is the starting point for branches 0-2. The FusionEncoder PT checkpoint is the starting point for branches 3-4. The SC3 checkpoint is the starting point for Phase 0 (zero-shot eval).

1b. Find A110 Pre-training Checkpoint (for NVIDIA fine-tune)

The A110 pre-training checkpoint is the starting point for Phase 1 (NVIDIA fine-tune). Find a completed olympus_v1_4_a110 run:

Option A: BQ query (preferred)

bq query --use_legacy_sql=false --format=csv --max_rows=10 "
SELECT id, root_run_id, name, future_state, created_at
FROM \`cruise-mlp-prod-13d0.mli_analytics.roboflow_export_rooted\`
WHERE name LIKE '%olympus_v1_4_a110%'
AND future_state = 'RESOLVED'
ORDER BY created_at DESC LIMIT 10"

Option B: Glean search

Search for "Olympus Hybrid Release Flow" or the V1.4 release doc to find the specific A110 run ID used in the deployed release.

Option C: lumen_agent MCP

CallMcpTool:
  server: project-0-cruise-ladybug-agents
  toolName: lumen_agent
  arguments:
    query: "Find resolved Roboflow runs with name containing 'olympus_v1_4_a110'.
            Return the run ID, name, and checkpoint GCS path from the output
            artifact. Show the 5 most recent."
    start_fresh_session: true

1c. Extract Checkpoint GCS Path from a Run

CRITICAL: For train-eval-scene-enc pipelines, checkpoints are stored under the train_pytorch_lightning subtask's run ID, NOT the orchestrator ID or the pytorch_trainer_substrate ID.

Step 1: Find the train_pytorch_lightning run ID from the orchestrator DAG

RF_TOKEN=$(authcli app get roboflow -out stdout)
curl -s -H "Authorization: Bearer $RF_TOKEN" \
  "https://roboflow.robot.car/api/v1/runs/<ORCHESTRATOR_ID>/dag" | \
  python3 -c "
import sys,json
def find(node, name):
    if node.get('transformerName') == name: return node
    for c in node.get('children',[]):
        r = find(c, name)
        if r: return r
d=json.load(sys.stdin)
n = find(d, 'train_pytorch_lightning')
if n: print(n.get('id'))
"

Step 2: List checkpoints in GCS using that ID

GCP_TOKEN=$(gcloud auth application-default print-access-token)
curl -s -H "Authorization: Bearer $GCP_TOKEN" \
  "https://storage.googleapis.com/storage/v1/b/robotorch2-prod/o?prefix=scene_encoder/<TRAIN_PYTORCH_LIGHTNING_ID>/checkpoints/&delimiter=/" | \
  python3 -c "import sys,json; [print(i['name']) for i in json.load(sys.stdin).get('items',[])]"

The final checkpoint follows this pattern: gs://robotorch2-prod/scene_encoder/<TRAIN_PYTORCH_LIGHTNING_ID>/checkpoints/final-checkpoint-epoch=<N>-step=<M>.ckpt

Common mistake: Using the orchestrator ID or pytorch_trainer_substrate ID for GCS paths returns empty results. Only the train_pytorch_lightning wrapper ID maps to the GCS directory.

1d. Checkpoint Chaining

Each stage loads the previous stage's checkpoint:

--ckpt_path=<prev_stage_checkpoint> --only_load_ckpt_weights

--only_load_ckpt_weights uses load_state_dict_strict_overwrite_to_false = True, allowing key mismatches (different heads, different camera counts).

1e. Hyperparameter Defaults

Mirror SC3 fine-tuning. The NVIDIA config must call modify_config_for_training_optimization (currently missing -- see plan Section 3 "Hyperparameters" for details).

Data splits: All three needed -- train (training), val (loss monitoring), test (eval job for Phase 0/1 metrics). Final decision metric is SC3 eval (Phase 2), but NVIDIA eval gives intermediate signal.

Smoke Test (10K samples via CLI limit) -- COMPLETE

Validated via run f387ad21. Loss converged 8.53 -> 3.07 (-64%). See runbook for full results.

Full Run (382K training samples)

LR scheduler (OneCycleLR, pct_start=0.1, div_factor=25) and optimizer (Adam, lr=1e-4) are inherited from the base config for both settings.

Knobs to explore (priority order): backbone frozen/unfrozen > epochs > learning rate > num GPUs. Keep everything else fixed at SC3 values for clean comparisons.

2. Launch Jobs

All commands use the ml_ranker_pipeline entry point:

//cruise/mlp/robotorch/project/trajectory_ranking:ml_ranker_pipeline

Pre-flight: GPU Quota Selection (MANDATORY)

Before every bazel run that submits a training or eval job, you MUST run the select-gpu-quota skill (~/.cursor/skills/select-gpu-quota/SKILL.md):

1. Query real-time GPU quota via Grafana Prometheus (capacity, in-use, queued)
2. Compute availability for each quota path
3. Present the quota table to the user
4. Get explicit user approval for the --business_attribution value
5. Only then proceed with the bazel run command

Do NOT default to structured_prediction without checking. It is often fully subscribed. Submitting to a full quota means hours in queue when alternative paths have immediate capacity.

This applies to: train-scene-enc, train-eval-scene-enc, eval-scene-enc, and any ml_ranker_pipeline submission that requests GPU resources.

Pre-flight: Eval GPU Type Selection (MANDATORY)

T4 GPUs are deprecated for eval jobs. Per the AV Infra migration plan (EoQ2 2026 deadline), all eval workloads must migrate to L4. See the T4 deprecation plan.

How to select L4 for eval:

Config defaults by version:

For olympus_v1_4_sc3_road eval submissions, always add --apply_macros eval_with_l4_gpu:

bazel run --config=no-tty \
  //cruise/mlp/robotorch/project/trajectory_ranking:ml_ranker_pipeline -- \
  eval-scene-enc \
  --config-choice=olympus_v1_4_sc3_road \
  --apply_macros eval_with_l4_gpu \
  ...

Macro specs (from cluster_specs.py):

Consistency rule: Within a single experiment comparison (e.g., all 7 NVIDIA branches), all OL evals must use the same GPU type. If some branches already completed on T4, either rerun them all on L4 or keep all on T4 for that experiment. Do not mix GPU types within a comparison set.

2a. Train + Eval (remote, Roboflow)

bazel run --config=no-tty \
  //cruise/mlp/robotorch/project/trajectory_ranking:ml_ranker_pipeline -- \
  train-eval-scene-enc \
  --config-choice=<config_name> \
  --exp_name=<experiment_name> -v <version> \
  --ckpt_path=<gs://checkpoint_path> --only_load_ckpt_weights \
  --business_attribution=<USER_APPROVED_QUOTA> \
  --skip-prevalidate --ignore_time_limit --ignore_cost_warning \
  --model_selection final

2b. Eval Only (remote, Roboflow)

bazel run --config=no-tty \
  //cruise/mlp/robotorch/project/trajectory_ranking:ml_ranker_pipeline -- \
  eval-scene-enc \
  --config-choice=<config_name> \
  --apply_macros eval_with_l4_gpu \
  --exp_name=<experiment_name> \
  --ckpt_path=<gs://checkpoint_path> --only_load_ckpt_weights \
  --business_attribution=<USER_APPROVED_QUOTA> \
  --skip-prevalidate --ignore_time_limit

Note: --apply_macros eval_with_l4_gpu ensures L4 GPUs. Omit only if you need T4 for backward-compatibility with an existing experiment set.

<USER_APPROVED_QUOTA>: Run the select-gpu-quota skill first to determine the best quota path, then get explicit user approval before substituting the value. Default is structured_prediction but may change based on real-time availability.

For eval-only on a previously trained run's checkpoint, use --train_job_id instead of --ckpt_path:

  --train_job_id=<roboflow_train_job_id>

2b-1. Handling Stale Branch Errors

If bazel run fails with a stale-branch error, there are two layers:

Full bypass command (both layers):

CRUISE_BYPASS_BAD_COMMITS=true \
STALE_CODE_REASON="Experiment branch must match prior runs for fair comparison" \
bazel run --config=no-tty \
  //cruise/mlp/robotorch/project/trajectory_ranking:ml_ranker_pipeline -- \
  train-eval-scene-enc \
  --config-choice=<config_name> ...

Decision: bypass vs rebase -- ASK THE USER. Present both options:

• Bypass: Use when experiments must run on identical code as prior runs for fair comparison, or when develop has unrelated breaking changes. Provide a meaningful STALE_CODE_REASON.
• Rebase: Use when the user wants the latest code, or when CI/PR submission requires it. See rebase-to-develop skill.

Never assume rebase is required. A rebase introduces confounding variables into experiment comparisons.

Source: cruise/mlp/roboflow2/server/stale_branch_blocking_utils.py, cruise/mlp/roboflow2/api_client_internal.py.

2c. Local Debug (quick validation)

bazel run --config=no-tty \
  //cruise/mlp/robotorch/project/trajectory_ranking/scene_encoder:local_debug -- \
  --config-choice=<config_name>

Add --shrink_trainer_config for minimal data. This does NOT submit to Roboflow; it runs locally on the current machine.

2d. Phase-Specific Commands

Phase 0: Zero-Shot Eval (V1.4 SC3 model on NVIDIA data)

bazel run --config=no-tty \
  //cruise/mlp/robotorch/project/trajectory_ranking:ml_ranker_pipeline -- \
  eval-scene-enc \
  --config-choice=olympus_v1_4_nvidia \
  --exp_name=nvidia_zero_shot_v1_4_baseline \
  --ckpt_path=gs://robotorch2-prod/scene_encoder/bc25b4cab83c4a4c969001fe6d9629d7/checkpoints/final-checkpoint-epoch=26-step=70808.ckpt \
  --only_load_ckpt_weights \
  --business_attribution=structured_prediction \
  --skip-prevalidate --ignore_time_limit

Phase 1: Backbone-frozen (NVIDIA fine-tune)

bazel run --config=no-tty \
  //cruise/mlp/robotorch/project/trajectory_ranking:ml_ranker_pipeline -- \
  train-eval-scene-enc \
  --config-choice=olympus_v1_4_nvidia_backbone_frozen \
  --exp_name=nvidia_backbone_frozen -v smoke_1 \
  --ckpt_path=<a110_checkpoint> --only_load_ckpt_weights \
  --business_attribution=structured_prediction \
  --skip-prevalidate --ignore_time_limit --ignore_cost_warning \
  --model_selection final

Phase 1: Backbone-unfrozen (NVIDIA fine-tune)

bazel run --config=no-tty \
  //cruise/mlp/robotorch/project/trajectory_ranking:ml_ranker_pipeline -- \
  train-eval-scene-enc \
  --config-choice=olympus_v1_4_nvidia \
  --exp_name=nvidia_backbone_unfrozen -v smoke_1 \
  --ckpt_path=<a110_checkpoint> --only_load_ckpt_weights \
  --business_attribution=structured_prediction \
  --skip-prevalidate --ignore_time_limit --ignore_cost_warning \
  --model_selection final

2d-1. Phase 2: SC3 Fine-tune (from any Phase 1 checkpoint)

CRITICAL: Phase 2 SC3 fine-tuning MUST use olympus_v1_4_sc3_road (30 epochs), regardless of which Phase 1 config was used. Do NOT re-use the Phase 1 config for Phase 2 -- Phase 1 configs have different epoch counts (12-23) and train on different datasets (NVIDIA or SC3 autolabel). Phase 2 always fine-tunes on SC3 road data.

To find the Phase 1 checkpoint, follow Section 1c:

1. Get the Phase 1 Roboflow orchestrator ID (from Centra URL)
2. Query the DAG to find the train_pytorch_lightning subtask ID
3. List GCS checkpoints: gs://robotorch2-prod/scene_encoder/<train_pytorch_lightning_id>/checkpoints/
4. Use the final-checkpoint-epoch=*-step=*.ckpt file

Phase 2 command template:

CRUISE_BYPASS_BAD_COMMITS=true \
STALE_CODE_REASON="Experiment branch must match prior runs for fair comparison" \
bazel run --config=no-tty \
  //cruise/mlp/robotorch/project/trajectory_ranking:ml_ranker_pipeline -- \
  train-eval-scene-enc \
  --config-choice=olympus_v1_4_sc3_road \
  --exp_name=<branch_name>_then_sc3 \
  --ckpt_path=gs://robotorch2-prod/scene_encoder/<parent_run_id>/checkpoints/final-checkpoint-epoch=<N>-step=<M>.ckpt \
  --only_load_ckpt_weights \
  --business_attribution=<USER_APPROVED_QUOTA> \
  --skip-prevalidate --ignore_time_limit --ignore_cost_warning \
  --model_selection final

Naming convention: <branch_descriptor>_then_sc3 (e.g., nvidia_frozen_then_sc3, nvidia_autolabel_frozen_then_sc3).

2e. Common Knobs

Defaults mirror SC3 (see Section 1e). Override via CLI flags:

IMPORTANT: Epoch and GPU settings should be baked into the Python config (_make_olympus_v1_4_nvidia_base_config in olympus_v1_4.py). CLI overrides via roboparser are unreliable for nested config fields. In particular, --config.max_train_epochs N does NOT work -- use --step_limit=N instead.

3. Monitor Jobs

After bazel run prints a Centra URL like https://centra.robot.car/roboflow/runs/<run_id>:

3a. Poll for Completion

bazel run //cruise/e2e_gym:roboflow_job_status -- <run_id> --poll-for-completed \
  --timeout-sec 14400 --poll-interval-sec 60

Run in background (block_until_ms: 0), check periodically.

3b. Quick State Check

bazel run //cruise/e2e_gym:roboflow_job_status -- <run_id>

States: RESOLVED (done), RAN (Dataflow finished, may still be processing), NESTED_FAILED, FAILED.

3c. Via lumen_agent

CallMcpTool:
  server: project-0-cruise-ladybug-agents
  toolName: lumen_agent
  arguments:
    query: "Check the status of Roboflow run <run_id>. Is it completed?
            What is the future_state? If resolved, get the checkpoint path."
    start_fresh_session: true

3d. Early Health Check (IMPORTANT)

After submitting, wait 5 minutes and run the debug tool to catch startup errors fast:

bazel run --config=no-tty \
  //cruise/mlp/robotorch/project/trajectory_ranking/dcv_1/notebooks:debug_roboflow_errors -- <run_id>

If the run has already failed (NESTED_FAILED), fix immediately instead of polling for hours. Common startup failures resolve in <5 minutes.

3e. What to Watch For

3f. Dataset Config Checklist (Pre-launch)

Before launching eval or train-eval, verify the dataset config:

1. Split weights: Weights(train=X, val=Y, test=Z) -- all splits you need must have weight > 0. For eval-only (eval-scene-enc): test > 0 required. For train-eval: all three > 0 recommended.
2. val_split_available: Must be True in the training config if val weight > 0.
3. job_id: Must point to a Stage 2 (map+intent) job, not Stage 1.
4. BQ tables exist: Verify with lumen_agent or BQ query before launching.

4. W&B Analysis

4a. Find W&B Link from a Centra Run

Note: The BQ analytics table roboflow_export_rooted has a replication lag of 30-60 minutes after RESOLVED. If the W&B query returns empty, wait and retry, or check the Centra URL directly (the W&B link appears in the run's artifacts tab).

Primary method: lumen_agent (requires ladybug-mcp-dev token):

CallMcpTool:
  server: project-0-cruise-ladybug-agents
  toolName: lumen_agent
  arguments:
    query: "Get the W&B URL and eval metrics for resolved Roboflow run <run_id>"
    start_fresh_session: true

Fallback: BQ query (may lag behind):

Use the analyze-olympus-runs skill pattern:

SELECT
  e.producer_run_id,
  JSON_VALUE(a.json_representation, '$.wandb_url.value_serialization') as wandb_url
FROM `ca-silver-prd-e2h4.ml_pipelines_mart.run_edges_vw` e
JOIN `ca-silver-prd-e2h4.ml_pipelines_mart.fact_run_edge_artifacts` a
  ON e.artifact_id = a.artifact_id
WHERE e.producer_run_id IN (
  SELECT id FROM `cruise-mlp-prod-13d0.mli_analytics.roboflow_export_rooted`
  WHERE root_run_id = '<root_run_id>'
  AND transformer_path LIKE '%pytorch_trainer_substrate%'
)
AND e.name = 'streaming_metrics'

4b. Compare Loss Curves

Use the analyze-wandb skill for structured run comparison. Provide both run IDs and ask for convergence analysis:

Key metrics to compare between backbone-frozen and backbone-unfrozen:

• train/total_loss convergence rate
• train/tracker_loss and train/trajectory_loss separately
• Learning rate schedule alignment
• Gradient norm stability

4c. Key Metrics for NVIDIA Experiments

Phase 0 vs. Phase 1 (open-loop, NVIDIA data):

• Detection 3D AP by range [0-25m], [25-50m], [50-100m]
• Tracking lateral/longitudinal error by distance bin

Phase 2 (open-loop + closed-loop, SC3 data):

• All Phase 0/1 SP metrics on SC3 data
• Map head: Lane F1, chamfer distance (gating -- must not regress)
• SC3 Protocol: Lane Keep, ORA Actions pass rates (from decorated_test_requests); TTC, collision, hard brake mi/event, swerve mi/event, controllability, sim safety proxy (from decorated_test_executions.execution.scores -- ~300 per-execution metrics). See sc3-protocol skill for extraction queries.

4d. Automated BQ Metrics Extraction (Primary Method)

The eval pipeline writes per-sample 3D detection metrics to BQ. Use lumen_agent to discover the tables, then query directly.

Step 1: Find BQ tables via lumen_agent

CallMcpTool:
  server: project-0-cruise-ladybug-agents
  toolName: lumen_agent
  arguments:
    query: "For Roboflow run <run_id>, find the pytorch_evaluator_substrate
            nested transformer. Get its streaming_metrics side artifact (W&B URL)
            and ModelEvaluationResults output (BQ table refs in local_measurements).
            Return ALL BQ table references found."
    start_fresh_session: true

Key artifacts from ModelEvaluationResults.local_measurements:

• tracking_3d → cruise-mlp-prod-13d0.visual_detection_metrics.detection_3d_<eval_id>
• inference_metrics → cruise-mlp-prod-13d0.inferences_dev.scene_encoder_inference_metrics_<parent_id>

Step 2: Run aggregate detection metrics queries

-- Precision by range bin (primary detection metric)
SELECT
  CASE WHEN detection_distance_from_av BETWEEN 0 AND 25 THEN '0-25m'
       WHEN detection_distance_from_av BETWEEN 25 AND 50 THEN '25-50m'
       WHEN detection_distance_from_av BETWEEN 50 AND 100 THEN '50-100m'
       ELSE '100m+' END as range_bin,
  COUNTIF(match_outcome_class LIKE 'MATCH%') as tp,
  COUNTIF(match_outcome_class = 'NO_LABEL') as fp,
  ROUND(COUNTIF(match_outcome_class LIKE 'MATCH%')
    / NULLIF(COUNTIF(dt_valid), 0), 4) as precision,
  ROUND(AVG(CASE WHEN match_outcome_class LIKE 'MATCH%'
    THEN centroid_error_3d END), 3) as centroid_err,
  ROUND(AVG(CASE WHEN match_outcome_class LIKE 'MATCH%'
    THEN heading_error_3d END), 3) as heading_err
FROM `cruise-mlp-prod-13d0.visual_detection_metrics.detection_3d_<eval_id>`
WHERE dt_valid GROUP BY range_bin ORDER BY range_bin

Step 3: Run trajectory generation metrics queries

The inference_metrics table contains per-sample traj generation metrics. Key queries:

-- ADE/FDE at key time horizons
SELECT COUNT(*) as n,
  ROUND(AVG(top_mode_ade_lat_0_5s), 4) as ade_lat_0_5s,
  ROUND(AVG(top_mode_ade_lat_2s), 4) as ade_lat_2s,
  ROUND(AVG(top_mode_ade_lat_3s), 4) as ade_lat_3s,
  ROUND(AVG(top_mode_ade_long_1s), 4) as ade_long_1s,
  ROUND(AVG(top_mode_ade_long_2s), 4) as ade_long_2s,
  ROUND(AVG(top_mode_ade_long_9s), 4) as ade_long_9s,
  ROUND(AVG(min_l2_fde_1s), 4) as min_fde_1s
FROM `cruise-mlp-prod-13d0.inferences_dev.scene_encoder_inference_metrics_<parent_id>`

-- KCM, hard brake, NMC violations, route compliance
SELECT
  ROUND(AVG(av_pred_min_lateral_kcm), 4) as min_lat_kcm,
  ROUND(AVG(av_pred_min_longitudinal_kcm), 4) as min_long_kcm,
  ROUND(AVG(av_pred_hard_brake_rate_1s), 4) as hard_brake_1s,
  ROUND(AVG(top_1_av_pred_hard_brake_rate_with_gt_masking_1s), 4) as hard_brake_gt_1s,
  ROUND(AVG(top_1_av_pred_max_lat_acc_violation), 4) as lat_acc_viol,
  ROUND(AVG(fraction_trajectories_diverged_from_route), 4) as route_diverge,
  ROUND(AVG(av_too_slow_rate_6s), 4) as too_slow_6s,
  ROUND(AVG(top_mode_avg_vel_long_err_3s), 4) as vel_long_err_3s,
  ROUND(AVG(union_area_coverage_2d_radial_3s), 4) as coverage_3s,
  ROUND(AVG(total_loss), 4) as total_loss
FROM `cruise-mlp-prod-13d0.inferences_dev.scene_encoder_inference_metrics_<parent_id>`

4e. W&B Metric Download (for Loss Curves)

bazel run --config=no-tty \
  //cruise/mlp/robotorch/project/trajectory_ranking/scene_encoder/notebooks/tools:download_wandb_metric_csv -- \
  --wandb_url <wandb_url> \
  --metric_pattern "train/total_loss" --metric_pattern "valid/total_loss" \
  --output loss_curves.csv

4f. Existing Tools Reference

4g. Baseline Metrics (Phase 0 Zero-Shot)

Detection (from detection_3d table):

• Recall: 1.07% (1,285 / 120,423) -- misleading, see note below
• Only VEHICLE class matched (1,284/1,285 TPs); GT has 1,483 PED + 119 BIKE
• 429 scenes, avg GT distance 66.81m (48K GT at 50-100m, 23K at 100m+)

Detection thresholds: Two-stage filtering applies: (1) existence score >= 0.3 determines if a detection is valid (binary "is this a detection?" gate), then (2) BEV IoU >= 0.45 determines if a valid detection matches a GT label.

Important: Low recall does NOT mean the model can't detect NVIDIA objects. At 0-25m, the model produces 8,366 detections for 15,975 GT (52% DT/GT ratio), but only 675 meet BEV IoU >= 0.45 (4.2% recall). The bottleneck is box alignment (size/orientation/position), not detection ability. Additionally, 28% of matched objects have 180-degree heading flip, suggesting a heading convention mismatch in the featurization pipeline.

Trajectory Generation (from inference_metrics table, 3,432 samples):

5. Experiment Tracking

First experiment completed: Phase 0 zero-shot eval (dc99ccafec8244bc925f7b072d0d0186) resolved with 0/14 failures on 2026-03-26.

5a. Registry Format

Maintain a local YAML file at ~/.cursor/nvidia_training_experiments.yaml:

experiments:
  - name: "nvidia_zero_shot_v1_4_baseline"
    phase: 0
    variant: "zero-shot"
    config: "olympus_v1_4_nvidia"
    ckpt_source: "V1.4 SC3 deployed"
    ckpt_path: "gs://robotorch2-prod/scene_encoder/bc25b4cab83c4a4c969001fe6d9629d7/checkpoints/final-checkpoint-epoch=26-step=70808.ckpt"
    centra_url: "https://centra.robot.car/roboflow/runs/dc99ccafec8244bc925f7b072d0d0186"
    wandb_url: null  # Check Centra artifacts tab
    status: "completed"
    notes: "RESOLVED 0/14 failures. First attempt (e44c66c3) failed with InvalidConfigError due to test weight=0; fixed Weights to (1.0,1.0,1.0)."
    launched_at: "2026-03-26T01:00:08"
    completed_at: "2026-03-26T01:32:16"

  - name: "nvidia_backbone_frozen_smoke_1"
    phase: 1
    variant: "backbone-frozen"
    config: "olympus_v1_4_nvidia_backbone_frozen"
    ckpt_source: "A110 pre-training"
    ckpt_path: "<a110_checkpoint>"
    centra_url: null
    wandb_url: null
    status: "pending"
    notes: ""
    launched_at: null
    completed_at: null

5b. Workflow

1. Before launching, add an entry with status: pending
2. After launch, update with centra_url and status: running
3. After completion, update with wandb_url, status: completed, metrics summary
4. On failure, update with status: failed, error notes

TODO: Automate registry updates in the launch/monitor workflow.

6. Results Publishing

Choose the publishing target based on image density:

6a. Markdown-First Workflow (all reports)

All reports start as markdown, co-located with their images:

1. Extract images from executed notebooks into the notebook's output/ directory (e.g., cruise/.../notebooks/output/report_*.png). Use descriptive filenames.
2. Draft markdown in the same output/ directory so image references use bare filenames (![alt](filename.png)).
3. Add narrative around every figure: Explain what the plot shows, what indices mean (e.g., "Sample 0 = one driving sequence"), what to look for, and why the result looks the way it does.
4. Co-edit the markdown file with the user until content is finalized.
5. Publish to the appropriate target (see 6b/6c below).

6b. Google Doc (image-heavy reports)

• Target doc: Structured Perception - Nvidia Training & Inference
• Google Docs accepts markdown paste directly
• User manages images manually (screenshots from W&B, notebooks, extracted PNGs from output/ directory)
• Include Centra run URLs, W&B links, and metrics tables in the markdown
• For images the agent can't upload, use > **TODO:** ... placeholders with exact source locations

6c. Confluence (image-light reports)

Use the confluence-publish skill (~/.cursor/skills/confluence-publish/SKILL.md):

• Plan page: NVIDIA Data Training / Eval Plan for Olympus v1.4
• Results index: Structured Perception -- NVIDIA Zero-Shot Inference Results
• Size limit: The Atlassian MCP has a ~46KB payload limit. For larger reports, use the child page pattern:
  • Parent page = lightweight index (summary table, links, no inline images)
  • Child pages = one per section (e.g., "Detection Results", "Trajectory Results")
  • Always use contentFormat: markdown -- it's ~10x smaller than ADF
• Never overwrite a page with manually-uploaded images -- the ADF for media nodes is huge and fragile. Add new content as child pages instead.

6d. Slack

Use the slack-thread-reply skill (~/.cursor/skills/slack-thread-reply/SKILL.md) to share results in relevant channels. Include:

• One-line summary (which variant won, key metric deltas)
• Links to Centra, W&B, Google Doc or Confluence as appropriate
• Next steps

6e. Report Template

When publishing results, follow the write-technical-report skill (~/.cursor/skills/write-technical-report/SKILL.md) structure:

## NVIDIA Training Results: <Phase> - <Variant>

**Run**: [Centra](<url>) | [W&B](<url>)
**Config**: `<config_name>` | **Checkpoint source**: <source>
**Dataset**: NVIDIA MAP_AND_INTENT (<N> samples)

### Detection by Range
| Metric | Zero-shot | Backbone-frozen | Backbone-unfrozen |
|--------|-----------|-----------------|-------------------|
| Precision [0-25m] | 7.95% | ... | ... |
| Precision [25-50m] | 7.04% | ... | ... |
| Precision [50-100m] | 9.36% | ... | ... |
| Centroid Err [0-25m] | 0.865m | ... | ... |
| Centroid Err [25-50m] | 0.827m | ... | ... |
| Heading Err [0-25m] | 0.779 rad | ... | ... |
| Overall Recall | 1.07% | ... | ... |
| Overall Precision | 7.61% | ... | ... |

### Trajectory Generation
| Metric | Zero-shot | Backbone-frozen | Backbone-unfrozen |
|--------|-----------|-----------------|------------

…(truncated)