Dataviewer

Dataviewer Skill

Launch and interact with the Dataset Analysis Tool — a full-stack application for analyzing and annotating robotic training data from episode-based datasets.

Prerequisites

The backend virtual environment and frontend node_modules are auto-created on first launch by start.sh.

Launch and Connect Workflow

Follow these steps in order every time the dataviewer is started.

Step 1 — Start the app

Launch start.sh as a background terminal process. The script prints [OK] Both services are running and the URLs when both services are healthy.

cd data-management/viewer && ./start.sh

With a custom dataset path:

cd data-management/viewer && DATA_DIR=/path/to/datasets ./start.sh

Step 2 — Open SimpleBrowser

After confirming both services are running (look for [OK] Backend is healthy in terminal output), open the frontend in VS Code's SimpleBrowser using the open_browser_page tool:

open_browser_page("http://localhost:5173")

SimpleBrowser is the primary visual interface for the user. All Playwright automation operates headlessly in the background — the user sees results in SimpleBrowser.

If a non-default FRONTEND_PORT was set, substitute that port instead of 5173.

Step 3 — Load the Playwright MCP tools

Playwright runs in headless mode so it does not open a separate browser window. All visual feedback goes through SimpleBrowser (Step 2). The Playwright MCP server must be declared in .vscode/mcp.json with the --headless flag:

// .vscode/mcp.json
{
  "servers": {
    "playwright": {
      "command": "npx",
      "args": ["@playwright/mcp@latest", "--headless"]
    }
  }
}

[!IMPORTANT] The --headless flag is required. Without it, Playwright opens a separate Chromium window instead of working invisibly behind SimpleBrowser.

Before issuing any browser actions, always load the Playwright tools with:

tool_search_tool_regex("playwright|browser_snapshot|browser_navigate|browser_click|browser_type")

If the search returns no results the MCP server has not started. Ask the user to open the VS Code Command Palette and run MCP: Start Server → playwright, then retry the search.

Step 4 — Interact via Playwright MCP

Playwright operates headlessly on the same URL as SimpleBrowser. Both see the same backend state, so API-driven changes (labels, annotations) appear in both.

Once the tools are available, use the following patterns for all UI interaction:

Always call browser_snapshot first to inspect the current DOM before issuing click or type actions. Reference the selector patterns in the Frontend UI Structure section below.

Quick Start

Start the dataviewer with the default dataset path:

cd data-management/viewer && ./start.sh

Start with a custom dataset path:

cd data-management/viewer && DATA_DIR=/path/to/datasets ./start.sh

Parameters Reference

Dataset Path Configuration

The DATA_DIR environment variable controls which datasets are visible in the app. Each subdirectory under this path is treated as a separate dataset_id.

Methods to set DATA_DIR:

1. Environment variable override (recommended for ad-hoc use):

   DATA_DIR=/path/to/datasets ./start.sh
   

2. Edit backend/.env (persists across restarts):

   DATA_DIR=/path/to/datasets
   

3. Export before launch (session-scoped):

   export DATA_DIR=/path/to/datasets
   cd data-management/viewer && ./start.sh
   

When a dataset path is provided, update backend/.env so the value persists:

1. Read the current backend/.env file.
2. Replace the DATA_DIR= line with the new absolute path.
3. Start the app with ./start.sh.

Architecture

data-management/viewer/
├── start.sh              # Orchestrator: launches backend + frontend
├── backend/
│   ├── .env              # DATA_DIR and test config
│   ├── pyproject.toml    # Python dependencies (uv)
│   └── src/api/
│       ├── main.py       # FastAPI app, CORS, router registration
│       ├── routers/      # REST endpoints: datasets, annotations, labels, export, detection, analysis
│       ├── routes/       # AI analysis routes
│       ├── services/     # Business logic and dataset service
│       ├── models/       # Pydantic models
│       └── storage/      # Persistence layer
├── frontend/
│   ├── vite.config.ts    # Dev server + API proxy to :8000
│   └── src/
│       ├── App.tsx       # Root: dataset selector, episode list, annotation workspace
│       ├── api/          # HTTP client and typed API functions
│       ├── components/   # UI components (annotation, dashboard, episode viewer, export)
│       ├── hooks/        # React Query hooks for datasets, episodes, annotations
│       ├── stores/       # Zustand stores for episode and dataset state
│       └── types/        # TypeScript type definitions

API Reference

Core Endpoints

Label Endpoints

VLM-as-Judge Endpoints

[!NOTE] Mounted only when VLM_JUDGE_ENABLED=true in backend/.env. The frontend's JudgePanel auto-hides when the backend reports enabled: false.

POST response (snake_case on the wire, camelCased by the frontend client) is the composite JudgeResult:

{
  "episode_id": "leisaac-pick-orange/episode_000007",
  "instruction": "Grab orange and place into plate",
  "judge_model": "Qwen/Qwen3-VL-4B-Instruct",
  "prompt_version": "outcome-mcq-v1+gvl-process-v1+milestones-v1+failuremode-v1",
  "n_frames": 12,
  "outcome_success": true,
  "outcome_confidence": 0.83,
  "outcome_n_valid_votes": 3,
  "progress_per_frame": [0, 14, 28, 42, 57, 71, 85, 100],
  "voc": 0.92,
  "milestones": [
    {"name": "approach_object", "completed": true, "frame_range": "0-3", "evidence": "..."}
  ],
  "failure_mode": null,
  "cached": false
}

Key knobs (backend/.env):

Annotation Endpoints

Export and Analysis Endpoints

Annotation Workflow

Annotation combines API calls for efficiency with Playwright UI interaction for verification. Use the API for bulk operations and the UI for visual review and spot-checking.

Annotation surfaces

The annotation panel exposes three structured surfaces in addition to free-form labels:

Multi-camera selection

Datasets that record multiple camera streams expose a camera selector in the annotation workspace header. Default selection is episode.cameras[0] (or the first key of videoUrls when cameras is empty). User selections persist for the current episode; switching to an episode that no longer contains the selected camera resets selection back to cameras[0]. Both video playback and /frames/{idx} thumbnail extraction follow the active camera.

Language instruction (VLA annotation)

The LanguageInstructionWidget writes a structured payload through PUT /api/datasets/{id}/episodes/{idx}/annotations:

When a dataset task description is available the widget seeds the instruction with source = template; otherwise it creates a blank instruction with source = human. The source dropdown allows changing the value at any time.

Step 1 — Analyze trajectory data

Fetch episode trajectory data from the API to determine labels programmatically:

curl -s "http://localhost:8000/api/datasets/{dataset_id}/episodes/{idx}" | python3 -c "
import sys, json
d = json.load(sys.stdin)
traj = d['trajectory_data']  # List of frames with joint_positions and timestamps
print(f'Frames: {len(traj)}')
print(f'First joint positions: {traj[0]["joint_positions"][:8]}')
print(f'Last joint positions: {traj[-1]["joint_positions"][:8]}')
"

Episode trajectory data is a list of frame dictionaries, each containing:

The meta field of the episode response contains index, length, task_index, and has_annotations.

Step 2 — Determine labels from trajectory

Analyze gripper and joint data at multiple time points to classify episodes. Check the midpoint first, then 25% and 75% for episodes where grasp actions happen earlier or later:

# Example: check grip values at multiple points for robust classification
for pct in [25, 50, 75]:
    idx = int(len(traj) * pct / 100)
    jp = traj[idx]['joint_positions']
    right_grip = jp[7]   # Right arm gripper index
    left_grip = jp[15]   # Left arm gripper index

[!IMPORTANT] Some episodes have late or early grasp actions, so checking only the midpoint may yield UNKNOWN results. Always check multiple time points (25%, 50%, 75%) and the minimum grip value across the full trajectory for robust classification.

Step 3 — Apply labels via API

Use the PUT endpoint for each episode:

curl -s -X PUT "http://localhost:8000/api/datasets/{dataset_id}/episodes/{idx}/labels" \
  -H "Content-Type: application/json" \
  -d '{"labels": ["RIGHT", "SUCCESS"]}'

For bulk annotation, loop over episodes in a script:

import json, urllib.request

def annotate(dataset_id, episode_idx, labels):
    data = json.dumps({"labels": labels}).encode()
    req = urllib.request.Request(
        f"http://localhost:8000/api/datasets/{dataset_id}/episodes/{episode_idx}/labels",
        data=data, method="PUT",
        headers={"Content-Type": "application/json"})
    return json.loads(urllib.request.urlopen(req).read())

Step 4 — Persist labels

After applying labels via the API, persist them to disk:

curl -s -X POST "http://localhost:8000/api/datasets/{dataset_id}/labels/save"

[!WARNING] Labels applied via PUT are held in memory until saved. Always call the save endpoint after bulk annotation to avoid data loss.

Label storage on disk

The save endpoint writes labels to a JSON file inside the dataset's meta/ directory:

{DATA_DIR}/{dataset_id}/meta/episode_labels.json

For example, with the default dataset path:

datasets/ur10e_episodes/meta/episode_labels.json

File structure:

{
  "dataset_id": "ur10e_episodes",
  "available_labels": ["SUCCESS", "FAILURE", "PARTIAL", "LEFT", "RIGHT"],
  "episodes": {
    "0": ["LEFT", "SUCCESS"],
    "1": ["RIGHT", "SUCCESS"]
  }
}

To clear all labels for a fresh start, overwrite the file with an empty episodes object:

{
  "dataset_id": "{dataset_id}",
  "available_labels": ["SUCCESS", "FAILURE", "PARTIAL", "LEFT", "RIGHT"],
  "episodes": {}
}

After editing the file on disk, restart the backend or reload the page for changes to take effect.

Step 5 — Verify in UI with Playwright

After applying labels via API, refresh the browser and verify using Playwright:

1. Navigate to the app: browser_navigate to http://localhost:5173.
2. Wait for episode list to load: browser_wait_for with text like "64 Episodes".
3. Take a screenshot to confirm labels appear in the sidebar.
4. Use label filter buttons in the sidebar to verify counts match expectations.
5. Click individual episodes and scroll to the "Episode Labels" section to verify correct labels are applied.

Step 6 — Interactive annotation via UI

For individual episode review or correction:

1. Click an episode in the sidebar (aside li button elements).
2. Scroll to the "Edit Tools" / "Episode Labels" section using browser_evaluate with scrollIntoView.
3. Toggle label buttons (SUCCESS, FAILURE, PARTIAL, or custom labels) — clicking a selected label removes it.
4. Click "Save All" to persist.

Frontend UI Structure

The React app has these key areas for Playwright interaction:

Troubleshooting

VLM-as-Judge Workflow

The VLM judge scores each episode with an outcome MCQ (success/fail with N-sample self-consistency), a GVL process reward (per-frame 0-100 progress + Spearman VOC), milestone decomposition, and failure-mode attribution. Both the dataviewer UI and the CLI under evaluation/vlm_judge/ consume the same JudgeService, so cache hits and prompt versions stay aligned across surfaces.

Enable the judge

Edit data-management/viewer/backend/.env before launch:

DATA_DIR=/abs/path/to/datasets
DATAVIEWER_AUTH_DISABLED=true

VLM_JUDGE_ENABLED=true
VLM_JUDGE_BACKEND=echo                        # echo | qwen3-vl | openai-compat
VLM_JUDGE_MODEL_ID=Qwen/Qwen3-VL-4B-Instruct
VLM_JUDGE_N_FRAMES=12
VLM_JUDGE_CACHE_DIR=outputs/vlm-judge/cache

[!IMPORTANT] Restart the backend after editing .env. Uvicorn --reload re-reads code, not env vars. The frontend auto-detects backend state via GET /api/datasets/{id}/episodes/{idx}/judge.

Backends at a glance

UI workflow (Trajectory tab)

1. Open the dataviewer (open_browser_page("http://localhost:5173")).
2. Pick a dataset, select an episode, switch to the Trajectory tab.
3. The VLM Judge panel sits between Episode Labels and Language Instructions.
4. Click Run judge → outcome badge, progress sparkline, VOC, optional milestones + failure mode appear. The result also lands on disk under VLM_JUDGE_CACHE_DIR.
5. Re-visiting the same episode shows a cached badge. Click Force fresh to bypass the cache and re-run.

Playwright UI verification

Use the same MCP tooling as the rest of the skill, but route through the new panel selectors. After a browser_snapshot, click using element refs from the snapshot. As a JS-fallback when the snapshot lacks button refs:

browser_evaluate: () => {
  const headers = Array.from(document.querySelectorAll('h3'))
  const judge = headers.find((el) => el.textContent?.includes('VLM Judge'))
  judge?.scrollIntoView({ behavior: 'smooth', block: 'center' })
  const run = Array.from(judge?.parentElement?.querySelectorAll('button') ?? [])
    .find((b) => /run judge|re-evaluate/i.test(b.textContent ?? ''))
  run?.click()
  return run ? 'clicked' : 'panel not visible'
}

To assert the result rendered, wait for the outcome badge text:

browser_wait_for(text="SUCCESS")  # or "FAILURE", "Inconclusive"

Direct API calls (curl / Python)

CSRF must be honored on POST. The frontend hook does this transparently; for ad-hoc shell:

CSRF=$(curl -s http://localhost:8000/api/csrf-token -c /tmp/dv-cookie | jq -r .csrf_token)

# Cache lookup (no inference)
curl -s http://localhost:8000/api/datasets/leisaac-pick-orange/episodes/0/judge | jq

# Run the judge (cache-first)
curl -s -X POST http://localhost:8000/api/datasets/leisaac-pick-orange/episodes/0/judge \
  -H "Content-Type: application/json" -H "X-CSRF-Token: $CSRF" -b /tmp/dv-cookie \
  -d '{"force": false}' | jq

Bulk evaluation via the CLI (no dataviewer needed)

# Single dataset
evaluation/vlm_judge/scripts/evaluate-leisaac-pick-orange.sh --limit 5

# Generic
python -m evaluation.vlm_judge.run \
  --dataset datasets/cnc_lerobot \
  --views observation.images.color \
  --backend qwen3-vl \
  --model-id Qwen/Qwen3-VL-4B-Instruct \
  --output outputs/vlm-judge/cnc_lerobot.jsonl \
  --limit 5

# Policy-rollout MP4s (e.g. leisaac-tests/pickup-orange/)
evaluation/vlm_judge/scripts/evaluate-policy-rollouts.sh

The CLI writes a JSONL with the same composite schema as the API, keyed on the same SHA256 cache, so a CLI run primes the dataviewer's cache (and vice versa).

Brought to you by physical-ai-toolchain