Siwarex Eval

siwarex-eval

A System and Benchmark for LLM-based Q&A on Heterogeneous Data — Fokoue et al. (2024) (arXiv:2409.05735, 2024)

What this evaluates

Evaluates LLM-based natural language question answering over heterogeneous data sources by testing the model's ability to generate SQL queries that correctly invoke both database tables and external APIs. It probes complex query planning, API sequencing, and routing across mixed data modalities.

Datasets

• Spider (modified with API-replaced tables) — total ?; splits: test (-1)

Metrics

• execution accuracy (primary) — range: [0, 1]
  • Execution accuracy measures the fraction of questions where the model's generated query produces results identical to the gold standard SQL query's results on the database. Uses the sophisticated matching approach from Zhong et al. (2020) to handle result set equivalence.

Input / output format

Input: Natural language question

Output: SQL query (potentially containing User-Defined Function calls for API invocations)

Scoring recipe

def compute_execution_accuracy(predictions, gold_queries, db):
    correct = 0
    for pred_sql, gold_sql in zip(predictions, gold_queries):
        pred_res = execute_sql(pred_sql, db)
        gold_res = execute_sql(gold_sql, db)
        if sophisticated_match(pred_res, gold_res): # Zhong et al. 2020
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Models often fail at sequencing multiple API calls or merging/aggregating their results.
• Routing errors occur when the LLM decomposes a question correctly but sends it to the wrong tool (DB vs API).
• API argument hallucination leads to incorrect function invocations even when the right API is selected.

Evidence (verbatim from paper)

The evaluation metric is the execution accuracy measured by comparing the results produced by our system (or the baseline system) against those produced by the evaluation of the gold standard spider sql query on the original spider db. We use the sophisticated comparison approach that was introduced by (Zhong et al., 2020).

Citation

@misc{fokoue2024siwarex,
  title={A System and Benchmark for LLM-based Q&A on Heterogeneous Data},
  author={Fokoue et al. (2024)},
  year={2024},
  note={arXiv:2409.05735}
}

• arXiv: 2409.05735