Dmcontrol Metaworld Eval

dmcontrol-metaworld-eval

Temporal Difference Learning for Model Predictive Control — Hansen et al. (2022) (arXiv:2203.04955, 2022)

What this evaluates

Evaluates sample efficiency, asymptotic performance, and generalization of reinforcement learning agents on high-dimensional continuous control tasks with varying observation modalities (state, pixels, multi-modal) and reward structures (dense, sparse, goal-conditioned).

Datasets

• DeepMind Control Suite (DMControl) — total ?; splits: eval (-1)
• Meta-World v2 — total ?; splits: eval (-1)

Metrics

• Cumulative Episode Return (primary) — range: other (task-dependent)
  • Sum of discounted rewards collected over a single episode until termination or a fixed step limit. Reported as mean ± standard deviation across multiple independent random seeds.

Input / output format

Input: Continuous state vectors, or 3 stacked 84x84 RGB frames (with ±4 pixel shift augmentation) for image-based tasks, or multi-modal inputs (proprioception + egocentric camera). Goal-conditioned tasks include target goal states.

Output: Continuous action vectors (dimensionality varies by task, e.g., R^6 to R^38).

Scoring recipe

def evaluate(policy, envs, num_runs=10, steps=100000):
    returns = []
    for _ in range(num_runs):
        ep_returns = []
        for env in envs:
            obs = env.reset()
            ret = 0.0
            for _ in range(steps):
                action = policy(obs)
                obs, reward, done, _ = env.step(action)
                ret += reward
                if done: break
            ep_returns.append(ret)
        returns.append(ep_returns)
    return np.mean(returns, axis=0), np.std(returns, axis=0)

Common pitfalls

• Action repeat of 2 is applied to image-based tasks, meaning environment steps are counted in multiples of 2.
• Planning hyperparameters (horizon and iterations) are task-specific (e.g., 8 iterations for Dog, 12 for Humanoid) rather than fixed globally.
• Some baselines (MuZero, EfficientZero) discretize action spaces, making them infeasible for high-dimensional continuous tasks like Dog, while TD-MPC handles continuous actions natively.

Evidence (verbatim from paper)

We evaluate TD-MPC with a TOLD model on a total of 92 diverse and challenging continuous control tasks from DeepMind Control Suite (DMControl; Tassa et al. (2018)) and Meta-World v2 (Yu et al., 2019), including tasks with sparse rewards, high-dimensional state and action spaces, image observations, multi-modal inputs, goal-conditioning, and multi-task learning settings; see Appendix L for task visualizations. We choose these two benchmarks for their great task diversity and availability of baseline implementations and results. Table 1: Learning from pixels. Return of our method (TD-MPC) and state-of-the-art algorithms on the image-based DMControl 100k benchmark used in Srinivas et al. (2020); Kostrikov et al. (2020); Ye et al. (2021). Baselines are tuned specifically for image-based RL, whereas our method is not. Results for SAC, CURL, DrQ, and PlaNet are partially obtained from Srinivas et al. (2020); Kostrikov et al. (2020), and results for Dreamer, MuZero, and EfficientZero are obtained from Hafner et al. (2020b); Ye et al. (2021). Mean and std. deviation over 10 runs.

Citation

@misc{hansen2022tdmpc,
  title={Temporal Difference Learning for Model Predictive Control},
  author={Hansen et al. (2022)},
  year={2022},
  note={arXiv:2203.04955}
}

• arXiv: 2203.04955