Research Pipeline

Full Research Pipeline: Idea → Real Experiments → Submission

End-to-end autonomous research workflow for: $ARGUMENTS

Constants

• AUTO_PROCEED = true — When true, Gate 1 auto-selects the top-ranked idea. When false, waits for explicit user confirmation at every gate.
• ARXIV_DOWNLOAD = false — When true, /research-lit downloads arXiv PDFs.
• HUMAN_CHECKPOINT = false — When true, pause after each review round.

Override: /research-pipeline "topic" — AUTO_PROCEED: false, human checkpoint: true

Overview

/idea-discovery → implement → TRAIN → VALIDATE → /run-experiment → VALIDATE → /auto-review-loop → VISUALIZE → /paper-writing
├── Workflow 1 ──┤            ├─ Gate 2 ─┤          ├── Gate 3 ──┤            ├── Workflow 2 ──┤          ├── Gate 4 ──┤         ├── WF3 ──┤

Four validation gates prevent fake research from reaching the paper:

Pipeline

Stage 1: Idea Discovery (Workflow 1)

/idea-discovery "$ARGUMENTS"

Internally: /research-lit → /idea-creator → /novelty-check → /research-review

Output: IDEA_REPORT.md

🚦 Gate 1 — Idea Checkpoint: Present ranked ideas. Wait for user confirmation (or auto-select if AUTO_PROCEED=true).

🚦 Gate 1b: Idea Validation (MANDATORY — added after GraphVLA failure)

BEFORE any implementation, validate the idea's core assumptions.

/idea-validation "[selected idea description]"

This gate:

1. Extracts core assumptions — what MUST be true for the idea to work?
2. Tests the most fragile assumption with a 1-hour pilot experiment
3. Cross-validates with literature — has this been tried before?
4. Defines kill criteria — what result would prove the idea wrong?
5. Runs minimum viable experiment — 5-10 episodes, throwaway code

Kill criteria from GraphVLA failure:

• If the core mechanism HURTS the baseline → idea is fundamentally broken
• If the baseline already achieves near-perfect performance → no room for improvement
• If the approach requires changing input distribution of a frozen model → high risk

If assumption fails → ABANDON or PIVOT. Do NOT proceed to implementation. Do NOT rationalize "maybe it'll work with more engineering."

Why this gate exists: GraphVLA spent 15K lines of code and dozens of GPU-hours building a system whose core assumption (instruction swapping doesn't break VLAs) was falsified by a 10-minute pilot. This gate ensures that never happens again.

Stage 2: Implementation

Once an idea is confirmed AND validated (Gate 1b passed):

1. Implement the method — model architecture, training loop, evaluation scripts
2. Implement baselines — matching compute and data
3. Code review — reproducibility checks (seeds, configs, logging)

CRITICAL: Implementation MUST include:

• A training script that produces a model checkpoint
• An evaluation script that loads the checkpoint and runs actual inference
• A visualization script that captures simulation renders during evaluation

If the method builds on an existing model (e.g., adding a scheduler on top of a pretrained VLA), the implementation must include:

• Integration code that actually calls the pretrained model's inference
• NOT an oracle/mock that simulates what the model would do

🚦 Gate 2: Training Validation (NEW)

Before deploying experiments, verify a trained model exists.

/validate-experiment [project-dir]

Check 1: Does a model checkpoint exist (trained or pretrained for the target task)? Check 2: Can the model run inference on the target environment?

If FAIL:

• If training is needed: estimate GPU hours, write SLURM training script, deploy training job, wait for completion.
• If using pretrained model: verify the checkpoint loads and produces reasonable outputs on one example.
• DO NOT skip to evaluation with oracle mode. Oracle results are supplementary upper bounds, not primary evidence.

If PASS: Proceed to Gate 2b.

🚦 Gate 2b: Pilot Experiment (NEW)

Before deploying full-scale experiments, run ONE pilot episode:

1. Load the actual trained/pretrained model
2. Run one episode on the actual environment with rendering enabled
3. Verify:
   • Model produces non-trivial actions (not zeros, not random)
   • Environment renders actual RGB frames (saved to disk)
   • At least partial task completion occurs
   • Results are saved in the expected JSON format
4. Save the pilot frame as proof: figures/viz/pilot_frame.png

If the pilot fails: Debug the integration. Common issues:

• Model expects different observation format than env provides
• Action space mismatch (dimensions, normalization)
• Rendering not configured (MUJOCO_GL, EGL setup)
• HuggingFace offline mode not set on compute nodes

If the pilot succeeds: Proceed to Stage 3 with confidence that the full evaluation will produce real results.

This gate prevents wasting hours of GPU time on broken integrations.

Stage 3: Deploy Real Experiments

/run-experiment [experiment command]

MANDATORY requirements for experiments:

1. Model inference must use the actual trained/pretrained model, not an oracle
2. Environment must use actual physics simulation with rendering, not mock mode
3. Evaluation must capture RGB frames from the simulation at key timesteps
4. All results must include per-episode data with seeds for reproducibility

What to deploy:

• Main evaluation: model on all tasks, multiple seeds
• Baseline evaluation: same tasks, same seeds, without the proposed method
• Ablation evaluation: remove one component at a time
• Frame capture: save simulation renders at critical moments (success, failure, recovery)

Monitor and collect results:

/monitor-experiment [server]

🚦 Gate 3: Experiment Reality Check (NEW)

Before entering the review loop, verify experiments are real.

/validate-experiment [project-dir]

Run ALL five checks from the validate-experiment skill:

1. Trained model exists → PASS/FAIL
2. No oracle/mock in evaluation → PASS/FAIL
3. Simulation renders actual observations → PASS/FAIL
4. Visualization shows real system output → PASS/FAIL
5. Results are reproducible → PASS/FAIL

If ANY check fails: BLOCK the pipeline. Fix the issue. Re-run experiments.

Oracle results policy:

• Oracle experiments MAY be run as supplementary analysis (e.g., "upper bound if policy were perfect")
• Oracle results MUST be clearly labeled as such in the paper
• Oracle results MUST NOT be the primary evidence for any claim
• The paper MUST include results from actual model execution as the primary evaluation

If PASS: Proceed to Stage 4.

Stage 4: Auto Review Loop (Workflow 2)

/auto-review-loop "$ARGUMENTS — [chosen idea title]"

Up to 4 rounds of: GPT-5.4 reviews → Claude implements fixes → re-evaluate → re-review.

IMPORTANT: The review prompt MUST include:

• Whether results are from real model execution or oracle
• Whether simulation frames exist
• The exact model used (name, checkpoint, training details)

The reviewer should be asked to verify that the evidence is real, not just internally consistent.

Output: AUTO_REVIEW.md

🚦 Gate 4: Visualization Validation (NEW)

Before writing the paper, verify that real visualizations exist.

Required visualizations:

1. Simulation frames at key timesteps showing the actual robot/agent executing the task
2. Success case: frames showing the proposed method succeeding
3. Failure case: frames showing a baseline or ablation failing on the same task
4. Comparison: side-by-side showing the method's advantage visually
5. Quantitative figures: charts generated from real (not oracle) data

If visualization is missing: Capture frames by running the system with rendering enabled. Do NOT proceed to paper writing without real visual evidence.

What counts as real visualization:

• RGB frames from MuJoCo/Isaac/real robot showing objects, robot arm, workspace
• Resolution ≥ 256×256
• Shows actual task execution (objects being manipulated, not just static scenes)

What does NOT count:

• PIL-generated colored rectangles with text labels
• Abstract node diagrams without simulation context
• Charts from oracle/mock data

Stage 5: Paper Writing (Workflow 3)

/paper-writing "NARRATIVE_REPORT.md — venue: [target]"

Internally: /paper-plan → /paper-figure → /paper-write → /paper-compile → /auto-paper-improvement-loop

MANDATORY: The paper-figure stage MUST include:

• Actual simulation frames (from Gate 4 validation)
• Charts from real experiment data (not oracle)
• Architecture diagrams created programmatically (TikZ/code) or with high-quality tools

Oracle results in the paper:

• MAY appear in an ablation/analysis section as "oracle upper bound"
• MUST NOT appear in the main results tables without clear labeling
• MUST NOT be the only evidence for any claim

Stage 6: Final Summary

# Research Pipeline Report

**Direction**: $ARGUMENTS
**Chosen Idea**: [title]
**Date**: [start] → [end]

## Validation Status
| Gate | Status | Details |
|------|--------|---------|
| Gate 1: Idea | PASS | [idea title] selected |
| Gate 2: Training | PASS | [checkpoint path], [training hours] |
| Gate 3: Real Experiments | PASS | [model name], [no oracle in primary results] |
| Gate 4: Visualization | PASS | [N] simulation frames captured |

## Results Summary
- Primary results: [from real model execution]
- Oracle upper bound: [if applicable, clearly labeled]
- Baselines: [list]
- Ablations: [list]

## Deliverables
- paper/main.pdf — [pages] pages, [venue]
- figures/viz/ — [N] simulation frames
- results/ — [N] episodes with per-seed data

Key Rules

• ALL FOUR GATES ARE MANDATORY AND BLOCKING. Skipping a gate produces fake research.
• NO SHORTCUTS. If a gate check fails, you MUST fix it before proceeding — not suggest "we can skip it" or "we can add it later." There is no "later" — if it's not done now, it won't be done.
• NO PARTIAL COMPLETION. If the experiment plan calls for N baselines, run ALL N baselines. Do not run 4 of 6 and say "the remaining two won't change the story." You don't know that until you have the data.
• NO PREMATURE STAGE TRANSITIONS. Do not suggest moving to the next stage while the current stage has incomplete work. Complete ALL experiments, ALL frame captures, ALL downloads, ALL validations before proceeding.
• ENFORCEMENT — STAGE TRANSITION CHECKLIST: Before ANY stage transition, you MUST explicitly output this checklist and verify every item. If ANY item is incomplete, the transition is BLOCKED:

  STAGE TRANSITION CHECKLIST:
  [ ] All planned experiments completed (list each)
  [ ] All results downloaded locally
  [ ] All simulation frames downloaded
  [ ] /validate-experiment passed (no oracle contamination, real frames exist)
  [ ] Failed experiments fixed and re-run (not skipped)
  [ ] Multi-model comparison complete (if applicable)
  

• Oracle mode is supplementary, not primary. Never build a paper on oracle results alone.
• Real simulation frames are required. A robotics paper without robot images is not publishable.
• The reviewer (GPT-5.4) must be told whether results are real or oracle. Hiding this from the reviewer produces inflated scores.
• If training takes too long, use a pretrained model and fine-tune, or use a simpler task. Do not substitute oracle mode for real execution.
• Fail gracefully: If Gate 2 or 3 fails, report what's missing and estimate the cost to fix it. Let the user decide — but DO NOT decide for them by silently skipping.

Early Problem Detection and Rollback

Problems must be identified at the EARLIEST possible stage, not the latest.

Rollback Protocol: When a problem is detected, the pipeline does NOT continue forward. Instead:

1. STOP all running experiments
2. DIAGNOSE the root cause (not the symptom)
3. DECIDE: fix, pivot, or abandon
   • Fix: if the problem is implementation (e.g., black images)
   • Pivot: if the problem is the approach but the direction is viable
   • Abandon: if the core assumption is falsified
4. DOCUMENT what went wrong and why in FAILURE_LOG.md
5. RESUME only after the fix is validated with a pilot

Time limits for each stage:

• Idea validation (Gate 1b): 1-4 hours max
• Implementation: 1-2 days max before first real result
• First real experiment: must produce result within 4 hours of submission
• If any stage exceeds 2x its expected duration → stop and diagnose

The cost of detecting a problem:

• At Gate 1b: 1 hour wasted
• At Gate 2: 1 day wasted
• At Gate 3: 1 week wasted
• At paper writing: 1 month wasted
• After submission: career damage

ALWAYS prefer early detection over continued hope.