Noisytoolbench Eval

noisytoolbench-eval

Learning to Ask: When LLM Agents Meet Unclear Instruction — Wang et al. (2024) (arXiv:2409.00557, 2024)

What this evaluates

This benchmark evaluates how well LLM agents handle ambiguous or unclear user instructions by measuring their ability to ask clarifying questions, execute correct tool calls, and generate accurate final answers. It also assesses interaction efficiency by tracking redundant questions and total action steps.

Datasets

• NoisyToolBench — total ?; splits: test (-1)

Metrics

• A1 (primary) — range: [0, 1]
  • Binary metric measuring whether the LLM asks the anticipated clarifying question to resolve instruction ambiguity. Success if the correct question is asked at any point during execution.
• A2 — range: [0, 1]
  • Binary metric measuring whether the LLM invokes all anticipated APIs with correct arguments using available information.
• A3 — range: [0, 1]
  • Binary metric measuring whether the LLM successfully extracts information from API responses to fulfill the user instruction.
• Re — range: other
  • Average count of irrelevant or redundant questions asked during the interaction. Lower values indicate better precision.
• Steps — range: other
  • Average total number of actions (inference, questioning, API calls) required to complete the instruction. Lower values indicate higher efficiency.

Input / output format

Input: Ambiguous or unclear natural language user instructions, optionally accompanied by tool/API documentation context.

Output: A sequence of actions including API calls with arguments, clarifying questions for the user, and a final answer.

Scoring recipe

def score(predictions, gold):
    # A1: Semantic similarity check for questions
    a1 = 1 if semantic_similarity(predictions['asked_question'], gold['expected_question']) > 0.8 else 0
    # A2: Exact match for API calls
    a2 = 1 if set(predictions['api_calls']) == set(gold['api_calls']) else 0
    # A3: LLM-as-a-judge for final answer alignment
    a3 = 1 if judge_llm(predictions['final_answer'], gold['expected_intent']).is_aligned else 0
    # Re & Steps: Direct counts from interaction log
    re = count(predictions['irrelevant_questions'])
    steps = count(predictions['total_actions'])
    return {'A1': a1, 'A2': a2, 'A3': a3, 'Re': re, 'Steps': steps}

Common pitfalls

• ToolEvaluator uses a sentence-transformer for A1, which may misclassify questions that are semantically similar but functionally distinct.
• A2 and A3 rely on GPT-4o as a judge, introducing potential variability compared to deterministic exact-match evaluation.
• The 'Steps' metric counts inference generation as an action, making it incomparable to benchmarks that only count tool-use steps.
• IBTC (Instructions Beyond Tool Capabilities) cases are explicitly excluded from A2/A3 scoring, which can skew aggregate accuracy if not reported separately.

Evidence (verbatim from paper)

Specifically, we design the following five metrics: Accuracy 1 (A1). A1 evaluates the capability of LLMs to ask the anticipated questions that pinpoint the ambiguous elements in user instructions. A1 is considered a success if the LLMs manage to ask the correct questions at any point. Conversely, it is deemed a failure if they do not.

Citation

@misc{wang2024learningtoask,
  title={Learning to Ask: When LLM Agents Meet Unclear Instruction},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2409.00557}
}

• arXiv: 2409.00557