Salesforce Bi Eval

salesforce_bi_eval

Business Logic-Driven Text-to-SQL Data Synthesis for Business Intelligence — Liu et al. (2026) (arXiv:2601.14518, 2026)

What this evaluates

Evaluates the quality of synthetically generated Text-to-SQL data by measuring question-SQL alignment and business realism in a sales analytics domain.

Datasets

• Salesforce Sales Analytics Database — total 240; splits: test (240)

Metrics

• Question-SQL Alignment (%) (primary) — range: percent
  • Percentage of generated question-SQL pairs where the SQL query correctly answers the natural language question, validated against the target schema.
• Question Realism (%) — range: percent
  • Percentage of generated questions rated as realistic and aligned with actual business intelligence workflows and personas.

Input / output format

Input: Business logic instances (persona, work scenario, workflow) and target database schema (SOQL).

Output: Natural language questions paired with corresponding SOQL queries.

Scoring recipe

def calculate_metrics(pairs):
    aligned = 0
    realistic = 0
    for q, sql in pairs:
        if verify_alignment(q, sql, schema): aligned += 1
        if verify_realism(q, domain_experts): realistic += 1
    return aligned / len(pairs), realistic / len(pairs)

Common pitfalls

• The database uses SOQL, not standard SQL, which affects query execution and compatibility with standard Text-to-SQL benchmarks.
• Data quality metrics rely on expert validation or proprietary scoring rubrics not fully detailed in the text.
• The dataset is private and production-scale, limiting direct reproducibility without access to the Salesforce schema.

Evidence (verbatim from paper)

In total, we generate 240 question–SQL pairs, spanning diverse personas and business reasoning complexity levels. Table 2: Data quality comparison with existing Text-to-SQL data generation methods. MethodEvaluation LLM Question-SQLAlignment(%) QuestionRealism(%) OmniSQLLi et al. ([2025a]) Gemini3 Pro95.47 $(\downarrow 3.12)$78.91 $(\downarrow 19.53)$

Citation

@misc{liu2026businesslogic,
  title={Business Logic-Driven Text-to-SQL Data Synthesis for Business Intelligence},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2601.14518}
}

• arXiv: 2601.14518