Sm3 Text To Query Eval

sm3-text-to-query-eval

SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmark — Sivasubramaniam et al. (2024) (arXiv:2411.05521, 2024)

What this evaluates

Evaluates text-to-query systems across relational, document, and graph database models using four query languages (SQL, MQL, Cypher, SPARQL). It probes the ability of models to translate natural language medical questions into correct, executable database queries using standardized SNOMED-CT aligned synthetic patient data.

Datasets

• SM3-Text-to-Query — total 10000; splits: test (10000); repo https://github.com/jf87/SM3-Text-to-Query

Metrics

• correctness (primary) — range: [0, 1]
  • Percentage of generated queries that exactly match the expert-verified gold query. Execution-based equivalence may also be used per standard text-to-query protocols.

Input / output format

Input: Natural language question augmented with parameterized values (e.g., patient IDs, disease descriptions, names) drawn from the synthetic database.

Output: A single database query string in one of four target languages: SQL, MQL, Cypher, or SPARQL.

Scoring recipe

def compute_correctness(predictions, gold_queries):
    correct = 0
    for pred, gold in zip(predictions, gold_queries):
        if normalize_string(pred) == normalize_string(gold):
            correct += 1
    return correct / len(gold_queries)

Common pitfalls

• Models may overfit to template structures or specific parameterized values rather than learning general text-to-query translation.
• Non-answerable questions (15 total) require models to recognize missing information rather than generate a query, which can skew accuracy if not handled separately.
• Cross-model schema differences (relational vs. document vs. graph) mean a single prompt template cannot be used across all four database backends without adaptation.

Evidence (verbatim from paper)

Overall, we create 408 template questions... For each question template, we manually develop the corresponding query in SQL, SPARQL, Cypher and MQL. The queries are then verified by a second expert for correctness. For scaling the template questions, we augment them by automatically inserting values such as IDs, descriptions of diseases, and patient names queried from the database. This data augmentation step is fully configurable and can be used to generate enriched and linguistically diverse text/query pairs for arbitrary Synthea databases...

Citation

@misc{sivasubramaniam2024sm3texttoquery,
  title={SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmark},
  author={Sivasubramaniam et al. (2024)},
  year={2024},
  note={arXiv:2411.05521}
}

• arXiv: 2411.05521