C3 Bench Eval

c3-bench-eval

$C^3$-Bench: The Things Real Disturbing LLM based Agent in Multi-Tasking — Yu et al. (2025) (arXiv:2505.18746, 2025)

What this evaluates

Evaluates the robustness and multi-tasking capabilities of LLM-based agents by probing their ability to handle complex tool dependencies, propagate hidden information across tasks, and maintain stable decision policies under dynamic, multi-round interactions.

Datasets

• C^3-Bench — total 1024; splits: test (1024); repo https://github.com/Tencent-Hunyuan/C3-Benchmark

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly completed multi-task instances out of the total test set, calculated as (correct predictions / total instances) * 100.
• accomplish_progress (AP) — range: percent
  • A soft metric measuring the degree of task completion progress, providing finer-grained assessment than binary accuracy.
• optimal_path_rate (OP Rate) — range: percent
  • Rate at which the model's policy follows the optimal execution path for tool invocations.

Input / output format

Input: Multi-task prompts containing tool definitions, hidden/critical information, and dynamic decision histories requiring sequential or parallel tool calls across 1-4 tasks.

Output: Sequential or parallel tool invocation sequences followed by final task answers.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_inst in zip(predictions, gold):
        if pred.matches_exact(gold_inst):
            correct += 1
    return (correct / len(gold)) * 100

Common pitfalls

• Confusing single-task (256) and multi-task (768) subsets when reporting overall performance or comparing model rankings.
• Overlooking the distinction between parallel ($P_{multi}^P$) and sequential ($P_{multi}^S$) tool call dependencies, which cause different accuracy drops and require separate evaluation.
• Ignoring policy transition frequency (PTF) and volatility (VF) when diagnosing model instability, as high PTF sensitivity directly correlates with accuracy degradation in multi-task settings.

Evidence (verbatim from paper)

The $C^{3}$-Bench has 256 single-task and 768 multi-task testdata. Challenge 1 contains 256 entries with 1-4 tasks, while Challenges 2 and 3 contain 768 multi-task data with 2-4 tasks. This section reports the accuracy of 19 representative models. For each model series, we show only the best result. Testing follows Challenge 3’s method.

Citation

@misc{yu2025c3bench,
  title={$C^3$-Bench: The Things Real Disturbing LLM based Agent in Multi-Tasking},
  author={Yu et al. (2025)},
  year={2025},
  note={arXiv:2505.18746}
}

• arXiv: 2505.18746