Larybench Eval

larybench-eval

LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignment — Nie et al. (2026) (arXiv:2604.11689, 2026)

What this evaluates

Evaluates how well vision models and latent action representations capture semantic action categories and map visual features to low-level robotic control trajectories. It probes both high-level action understanding and physical grounding for generalizable vision-to-action alignment across diverse robotic and human motion datasets.

Datasets

• VLABench — total ?; splits: train (-1), val (-1)
• CALVIN — total ?; splits: train (-1), val (-1)
• RoboCOIN — total ?; splits: train (-1), val (-1)
• AgiBotWorld-Beta — total ?; splits: train (-1), val (-1)

Metrics

• Top-1 Accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted action categories averaged across all atomic and composite classes. Computed as the number of correct predictions divided by the total number of samples.
• Mean Squared Error (MSE) — range: [0, inf)
  • Average squared difference between predicted and ground-truth end-effector trajectory values. Computed as the mean of (predicted - true)^2 across all dimensions and time steps.

Input / output format

Input: Classification: 9 uniformly sampled frames per video clip, resized to 224×224. Regression: Pairs of images separated by a 5-frame interval. Both tasks use continuous latent embeddings extracted by the vision encoder or latent action model.

Output: Classification: Discrete action category label (atomic or composite). Regression: Continuous end-effector trajectory chunk (7/12/16-DoF per chunk).

Scoring recipe

# Classification
preds = probe(latent_features)
acc = (preds == gold_labels).float().mean()

# Regression
traj_pred = mlp(latent_features)
mse = ((traj_pred - gold_traj) ** 2).mean()

Common pitfalls

• Using discretized codebook indices instead of continuous latent embeddings causes significant information loss for regression tasks.
• Uniform frame sampling fails on composite tasks with varying FPS and motion speeds; requires motion-guided sampling to capture temporal dynamics.
• Latent dimensions vary across encoders; a projection layer is required to align dimensions before classification for fair comparison.

Evidence (verbatim from paper)

For semantic action classification, we report the Top-1 Accuracy averaged across all action categories. For low-level control regression, we utilize the Mean Squared Error (MSE) to measure physical fidelity.

Citation

@misc{nie2026lary,
  title={LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignment},
  author={Nie et al. (2026)},
  year={2026},
  note={arXiv:2604.11689}
}

• arXiv: 2604.11689