Indicdb Eval

indicdb-eval

IndicDB -- Benchmarking Multilingual Text-to-SQL Capabilities in Indian Languages — Dawar et al. (2026) (arXiv:2604.13686, 2026)

What this evaluates

Evaluates cross-lingual semantic parsing and Text-to-SQL capabilities across Indian languages, probing a model's ability to link natural language queries to complex, high-join-depth relational schemas and generate syntactically and semantically correct SQL.

Datasets

• IndicDB — total 15617; splits: test (15617)

Metrics

• Execution Accuracy (EX) (primary) — range: [0, 1]
  • Standard Text-to-SQL metric: the predicted SQL is executed on the target database, and its result set is compared against the result set of the gold SQL. A match yields a score of 1, otherwise 0. Averaged over all instances.

Input / output format

Input: Natural language question (in English or one of six Indic languages) and the target database schema (table names, column names, foreign key relationships, and sample values).

Output: A single SQL query string.

Scoring recipe

def evaluate(predictions, golds, databases):
    correct = 0
    for pred_sql, gold_sql, db in zip(predictions, golds, databases):
        pred_res = execute_sql(db, pred_sql)
        gold_res = execute_sql(db, gold_sql)
        if sets_equal(pred_res, gold_res):
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Models often hallucinate filter values not present in the database; the benchmark mitigates this by grounding SQL generation in actual database values.
• Translation errors in Indic languages can invert sorting directions or swap entities, leading to logically valid but semantically wrong SQL; rigorous HITL verification is required to isolate model reasoning gaps from translation artifacts.
• Schema-linking requires navigating deep foreign-key chains; models may fail to identify the minimal relevant sub-schema or misinterpret join paths.

Evidence (verbatim from paper)

The English dataset contains 3,684 validated natural language query and SQL pairs with a calibrated difficulty distribution as shown in Table 1. Logical and syntactic integrity is maintained through a two-tier validation protocol involving PostgreSQL execution and a manual audit by three database experts. The semantic alignment between Indic queries and SQL logic was confirmed using the Fleiss’ Kappa (κ) statistic, which yielded a coefficient of 0.84.

Citation

@misc{dawar2026indicdb,
  title={IndicDB -- Benchmarking Multilingual Text-to-SQL Capabilities in Indian Languages},
  author={Dawar et al. (2026)},
  year={2026},
  note={arXiv:2604.13686}
}

• arXiv: 2604.13686