Metaworld Eval

metaworld-eval

Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning — Yu et al. (2019) (arXiv:1910.10897, 2019)

What this evaluates

Evaluates reinforcement learning agents on their ability to learn and generalize across multiple robotic manipulation tasks. It tests multi-task RL by measuring performance on a shared set of training tasks, and meta-RL by measuring rapid adaptation to completely unseen tasks from the same distribution.

Datasets

• Meta-World — total 50; splits: meta-train (-1), meta-test (-1); repo https://github.com/rlworkgroup/metaworld

Metrics

• average expected return (primary) — range: other
  • The sum of discounted rewards over the episode horizon: E_pi[sum_{t=0}^{T} gamma^t R_t(s_t, a_t)]. Evaluated as the average across tasks for multi-task RL, or on held-out tasks for meta-RL.

Input / output format

Input: State vector s (fixed dimensionality across tasks), optional task encoding z (for multi-task RL), and environment dynamics.

Output: Action a sampled from policy pi(a|s) or pi(a|s,z).

Scoring recipe

def compute_return(policy, env, horizon, gamma):
    total_reward = 0
    s = env.reset()
    for t in range(horizon):
        a = policy(s)
        s, r, done, _ = env.step(a)
        total_reward += (gamma ** t) * r
        if done: break
    return total_reward

# Multi-task eval
avg_return = mean([compute_return(policy, env_t, H, gamma) for env_t in train_envs])

# Meta-RL eval
adapted_policy = adapt(policy, new_env, few_shot_data)
meta_test_return = compute_return(adapted_policy, new_env, H, gamma)

Common pitfalls

• Multi-task RL is evaluated solely on training tasks with no held-out test set, which can overestimate generalization.
• State dimensionality is fixed across all tasks even when some tasks use fewer objects, requiring policies to handle unused state coordinates.
• Meta-RL evaluation requires adaptation with a very small number of samples; using too many adaptation steps violates the rapid-adaptation premise.

Evidence (verbatim from paper)

The goal of multi-task RL is to learn a single, task-conditioned policy π(a|s,z), where z indicates an encoding of the task ID. This policy should maximize the average expected return across all tasks from the task distribution p(T), given by E_{Tp(T)}[E_π[∑_{t=0}^{T}γ^t R_t(s_t,a_t)]]. ... At meta-test time, a new task T_jp(T) is sampled that was not seen during meta-training, and the meta-trained policy must quickly adapt to this task to achieve the highest return with a small number of samples.

Citation

@misc{yu2019metaworld,
  title={Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning},
  author={Yu et al. (2019)},
  year={2019},
  note={arXiv:1910.10897}
}

• arXiv: 1910.10897