Flame Robotic Manipulation Eval

flame-robotic-manipulation-eval

FLAME: A Federated Learning Benchmark for Robotic Manipulation — Bou Betran et al. (2025) (arXiv:2503.01729, 2025)

What this evaluates

Evaluates federated learning algorithms for decentralized robotic manipulation across heterogeneous environments. It probes a model's ability to generalize from distributed, non-IID demonstrations under visual and physical perturbations, measuring both action prediction fidelity and task completion success.

Datasets

• FLAME — total 42000; splits: train (400), val (10), test (10)

Metrics

• RMSE (primary) — range: other
  • Root Mean Squared Error between the model's predicted continuous actions and the expert demonstration actions. Lower values indicate better action prediction.
• Normalized Success Rate — range: [0, 1]
  • The fraction of episodes where the robot successfully completes the manipulation task, averaged over 50 episodes per test environment.

Input / output format

Input: RGB image (64x64x3) concatenated with low-dimensional robot joint positions.

Output: Continuous action vector (joint velocities) and a binary gripper command, with velocities constrained to [-1, 1].

Scoring recipe

def compute_metrics(predictions, gold, envs, episodes_per_env=50):
    # Offline: RMSE on action predictions
    rmse = sqrt(mean((predictions - gold) ** 2))
    
    # Online: Normalized Success Rate
    total_successes = 0
    total_episodes = len(envs) * episodes_per_env
    for env in envs:
        for ep in range(episodes_per_env):
            if check_task_completion(env, model):
                total_successes += 1
    success_rate = total_successes / total_episodes
    return rmse, success_rate

Common pitfalls

• The split sizes (400/10/10) refer to environments, not individual demonstrations; each environment contains 100 demonstrations.
• RMSE measures offline action prediction error, while Normalized Success Rate measures online task completion; they often diverge and should not be conflated.
• Evaluation is averaged across 10 test environments and 50 episodes per task; reporting per-environment or per-episode results without averaging violates the protocol.

Evidence (verbatim from paper)

We show the RMSE and the normalized success rate of the different methods across the four manipulation tasks. The values are averaged across 10 test environments and 50 episodes for each task.

Citation

@misc{boubetran2025flame,
  title={FLAME: A Federated Learning Benchmark for Robotic Manipulation},
  author={Bou Betran et al. (2025)},
  year={2025},
  note={arXiv:2503.01729}
}

• arXiv: 2503.01729