Gca Tool Use Eval

gca-tool-use-eval

GCA Framework: A Gulf-Grounded Dataset and Agentic Pipeline for Climate Decision Support — Muhammad Umer Sheikh et al. (2026) (arXiv:2604.12306, 2026)

What this evaluates

Evaluates an LLM's ability to use region-specific climate tools in a multi-step agentic pipeline. It probes structured tool invocation, argument schema adherence, step-wise reasoning, and end-to-end answer accuracy on Gulf-focused climate queries.

Datasets

• GCA-DS — total 200000; splits: unspecified (-1)

Metrics

• ToolAcc — range: percent
  • Percentage of steps where the model selects the correct tool from the available tool signatures.
• ArgAcc — range: percent
  • Percentage of steps where the model provides correct argument names and adheres to the required schema fields.
• InstAcc — range: percent
  • Percentage of steps where the model correctly follows the instruction and formatting requirements for that step.
• SummAcc — range: percent
  • Percentage of steps where the model's step-conditioned summary is correct given the actual tool outputs.
• AnsAcc (primary) — range: percent
  • Percentage of queries where the final answer is accurate after executing the model's predicted tool trace.
• AnsAcc+I — range: percent
  • AnsAcc variant that enables image generation during response composition to support visual explanations of temporal trends.

Input / output format

Input: A user query paired with available tool signatures. For evaluation, instances also include a gold tool-usage trace for reference during step-by-step scoring.

Output: Structured tool calls (following a specific format with tool name and schema-compliant arguments) and a final synthesized answer. In step-by-step mode, outputs are evaluated per step against gold traces.

Scoring recipe

def evaluate(predictions, golds):
    step_scores = {'InstAcc': 0, 'ToolAcc': 0, 'ArgAcc': 0, 'SummAcc': 0}
    for p_step, g_step in zip(predictions.steps, golds.steps):
        step_scores['InstAcc'] += int(p_step.format_valid)
        step_scores['ToolAcc'] += int(p_step.tool_name == g_step.tool_name)
        step_scores['ArgAcc'] += int(p_step.args == g_step.args)
        step_scores['SummAcc'] += int(p_step.summary_correct)
    ans_acc = int(predictions.final_answer == golds.final_answer)
    return {k: v/len(predictions)*100 for k,v in step_scores.items()}, ans_acc*100

Common pitfalls

• Format errors in tool-call structure prevent execution entirely, artificially deflating end-to-end accuracy.
• Argument errors (missing or incorrect schema fields) yield invalid tool runs that propagate downstream.
• Models may prematurely answer directly (N/A error) instead of invoking required tools, bypassing necessary grounding.

Evidence (verbatim from paper)

We report results in two complementary evaluation modes (Table 3). Step-by-step mode evaluates each step against gold traces: InstAcc (instruction-following for the step), ToolAcc (correct tool choice), ArgAcc (correct argument names/schema fields), and SummAcc (step-conditioned summary correctness given tool outputs). End-to-end mode evaluates the full execution outcome: AnsAcc measures final answer accuracy after executing the predicted tool trace, and AnsAcc+I enables image generation during response composition (useful when answers require visual explanation of temporal trends).

Citation

@misc{sheikh2026gcaframework,
  title={GCA Framework: A Gulf-Grounded Dataset and Agentic Pipeline for Climate Decision Support},
  author={Muhammad Umer Sheikh et al. (2026)},
  year={2026},
  note={arXiv:2604.12306}
}

• arXiv: 2604.12306