Bull Eval

bull-eval

FinSQL: Model-Agnostic LLMs-based Text-to-SQL Framework for Financial Analysis — Zhang et al. (2024) (arXiv:2401.10506, 2024)

What this evaluates

Evaluates the ability of LLMs to generate correct SQL queries from natural language questions in financial domains. It tests schema linking, cross-database transfer, and output calibration capabilities specific to fund, stock, and macroeconomic data.

Datasets

• BULL — total 4966; splits: train (3966), dev (1000)

Metrics

• execution accuracy (EX) (primary) — range: percent
  • Executes both the predicted and ground-truth SQL queries on the target database and returns 1 if the result sets are identical, 0 otherwise. Averaged over all instances.

Input / output format

Input: Natural language question and the relevant database schema (tables and columns).

Output: A syntactically valid SQL query string.

Scoring recipe

def calc_ex(predictions, golds, databases):
    correct = 0
    for pred, gold, db in zip(predictions, golds, databases):
        try:
            pred_res = set(db.execute(pred))
            gold_res = set(db.execute(gold))
            if pred_res == gold_res:
                correct += 1
        except Exception:
            pass
    return correct / len(predictions) * 100

Common pitfalls

• Execution accuracy requires running queries against a live database, not just parsing or string matching.
• Result sets must be compared as unordered collections (e.g., using sets) to account for SQL engines returning rows in arbitrary order.
• GPT-based baselines often exceed context windows, forcing truncation or expensive 32k models, which skews cost and accuracy comparisons.

Evidence (verbatim from paper)

We choose execution accuracy (EX) as our evaluation metric, as implemented by Test Suite Accuracy*(Zhong et al., 2020). This metric is also the official evaluation metric used by the popular Text-to-SQL leaderboard, Spider(Yu et al., 2018b)*. EX executes the predicted SQL query and golden SQL query in the database and judges whether the two have the same execution results.

Citation

@misc{zhang2024finsql,
  title={FinSQL: Model-Agnostic LLMs-based Text-to-SQL Framework for Financial Analysis},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2401.10506}
}

• arXiv: 2401.10506