Tweac Agent Selection Eval

tweac-agent-selection-eval

TWEAC: Transformer with Extendable QA Agent Classifiers — Geigle et al. (2021) (arXiv:2104.07081, 2021)

What this evaluates

This evaluation probes a model's ability to correctly route natural language questions to the most appropriate domain-specific QA agent from a large, heterogeneous pool. It measures both sample efficiency (performance with few training examples per agent) and scalability (maintaining accuracy as the number of candidate agents grows to hundreds).

Datasets

• QA-Tasks — total ?; splits: train (-1), test (-1)
• Many-Agents — total ?; splits: train (-1), test (-1)

Metrics

• Accuracy@1 (primary) — range: [0, 1]
  • The fraction of test questions where the correct agent is ranked at position 1 in the model's output list.
• mean reciprocal rank (MRR) — range: [0, 1]
  • The average of 1/rank for the correct agent across all test questions, where rank is the position of the correct agent in the predicted list.

Input / output format

Input: A natural language question (query) to be classified and routed to a candidate QA agent.

Output: A ranked list of candidate QA agents, or the top-1 predicted agent ID.

Scoring recipe

def compute_metrics(predictions, golds):
    # predictions: list of top-1 agent IDs or ranked lists
    # golds: list of correct agent IDs
    acc1 = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    mrr = sum(1.0 / (rank + 1) for p_list, g in zip(predictions, golds) if g in p_list) / len(golds)
    return {'Accuracy@1': acc1, 'MRR': mrr}

Common pitfalls

• The evaluation assumes only the dataset-specific agent is relevant; all other agents are treated as irrelevant distractors, which simplifies the ranking task compared to open-world routing.
• Accuracy drops by 20-30 points when scaling from 10 to 200 agents due to increased task difficulty and topic overlap between agents, not necessarily model failure.
• Sample efficiency varies significantly by agent specialization; broad-topic agents require substantially more training data than highly specialized ones like Weather Report.

Evidence (verbatim from paper)

We report Accuracy@1 and mean reciprocal rank (MRR) as performance scores. We make the assumption that only the agent from the respective dataset from which we draw the test question is relevant, all other agents are irrelevant. This assumption is a result of our dataset construction.

Citation

@misc{geigle2021tweac,
  title={TWEAC: Transformer with Extendable QA Agent Classifiers},
  author={Geigle et al. (2021)},
  year={2021},
  note={arXiv:2104.07081}
}

• arXiv: 2104.07081