adu2021 Component ID
Two-Stage Language-Action Alignment Framework for VLAs
Motivation
Multimodal large language models (MLLMs) excel at embodied reasoning tasks, but these improvements don't consistently translate to better robot control. The gap exists because "modality gap between language and low-level actions"—language understanding and motor control are optimized separately. Reasoning improvements don't guarantee action accuracy.
The Modification
RoboAlign introduces two-stage training that directly aligns MLLM capabilities with robot action generation:
Stage 1: Supervised Fine-Tuning (SFT) - Train the model to generate low-level action tokens through reasoning. The model learns to emit action tokens as direct outcomes of spatial and temporal reasoning over visual inputs.
Stage 2: Reinforcement Learning (RL) - Apply RL with action-accuracy rewards to refine reasoning toward precise action execution. Instead of optimizing reasoning through language alone, sample action tokens as direct outcomes and evaluate reasoning quality through action accuracy feedback.
This grounds linguistic understanding in concrete motor control, bridging the modality gap.
How It Works
The mechanism operates by:
- Sampling action tokens as concrete realizations of the model's reasoning (not as a separate decoding step)
- Evaluating reasoning quality through action accuracy metrics (robot task success, trajectory closeness, etc.)
- Backpropagating rewards to improve both the reasoning process and the action emission mechanism
- Using minimal additional data for RL training (less than 1% of task trajectories)
The key insight is coupling language reasoning directly to motor control outcomes rather than optimizing them independently.
Ablation Results
The paper demonstrates substantial gains:
- LIBERO benchmark: 17.5% improvement over SFT baseline
- CALVIN benchmark: 18.9% improvement over SFT baseline
- Real-world environments: 106.6% improvement over SFT baseline
- Achieves these gains using less than 1% additional data for RL training
- Improvements hold across diverse robot embodiments and task families
Conditions
- Requires action-level ground truth or task success signals for RL reward computation
- Works best when task accuracy is measurable (robot manipulation, navigation)
- Assumes sufficient coverage of diverse reasoning scenarios in initial SFT data
- RL stage requires relatively small amounts of additional interaction data (< 1%)
Drop-In Checklist
- Implement SFT training on (image, reasoning, action token) triplets
- Design action-accuracy reward function appropriate for target tasks
- Integrate RL training loop with action token sampling
- Implement backpropagation of rewards to both reasoning and action generation
- Collect/prepare minimal RL training data (< 1% of total trajectories)
- Benchmark on LIBERO, CALVIN, and real-world robot tasks
- Validate that language reasoning improvements transfer to action accuracy