Ragen Agent Eval

ragen-agent-eval

RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning — Wang et al. (2025) (arXiv:2504.20073, 2025)

What this evaluates

Evaluates LLM agents' multi-turn decision-making and reasoning capabilities across symbolic planning, risk-sensitive reasoning, and realistic web interaction environments. It probes the agent's ability to complete interactive tasks under noisy or probabilistic feedback while maintaining exploration and training stability.

Datasets

• Bandit — total 256; splits: val (256)
• Sokoban — total 256; splits: val (256)
• Frozen Lake — total 256; splits: val (256)
• WebShop — total 256; splits: val (256)

Metrics

• success rate (primary) — range: [0, 1]
  • Fraction of the 256 fixed prompts where the agent successfully completes the task within the turn limit.
• rollout entropy — range: other
  • Entropy of the action distribution across N=16 rollouts per prompt, measuring exploration diversity.
• in-group reward variability — range: other
  • Variance or standard deviation of rewards within a batch of rollouts, indicating behavioral diversity.
• response length — range: other
  • Average number of actions or tokens generated per episode, capped at 5 turns and 10 actions per turn.
• gradient norm — range: other
  • L2 norm of the policy gradients computed during updates, used to monitor training stability.

Input / output format

Input: 256 fixed prompts per environment, processed by the LLM agent in an interactive environment with a maximum of 5 turns and 10 actions per turn.

Output: Agent's action sequence (up to 10 actions per turn, max 5 turns) and final task completion status.

Scoring recipe

def compute_metrics(prompts, agent_actions, rewards, gradients):
    success_rate = sum(1 for r in rewards if r == 1) / len(rewards)
    rollout_entropy = compute_entropy(agent_actions)
    reward_var = np.var(rewards)
    response_len = np.mean([len(actions) for actions in agent_actions])
    grad_norm = np.linalg.norm(gradients)
    return success_rate, rollout_entropy, reward_var, response_len, grad_norm

Common pitfalls

• Confusing training diagnostics (gradient norm, in-group reward variability) with final evaluation metrics; these are computed over validation instances but primarily track training dynamics.
• Ignoring the hard truncation limit of 5 turns and 10 actions per turn, which artificially caps response length and distorts entropy/variability calculations if not enforced.
• Using evaluation temperature T=0.5 inconsistently; the paper explicitly fixes T=0.5 during evaluation, which significantly impacts exploration and success rates compared to greedy decoding.

Evidence (verbatim from paper)

We evaluate on 256 fixed prompts per environment with temperature T=0.5, truncating episodes after 5 turns. Metrics include: (i) success rate (task completion), (ii) rollout entropy (exploration), (iii) in-group reward variability (behavioral diversity), (iv) response length (reasoning verbosity), and (v) gradient norm (training stability). All are computed over validation instances.

Citation

@misc{wang2025ragen,
  title={RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2504.20073}
}

• arXiv: 2504.20073