Dysql Bench Eval

dysql-bench-eval

Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration — Sun et al. (2025) (arXiv:2510.26495, 2025)

What this evaluates

Evaluates a model's ability to perform dynamic, multi-turn Text-to-SQL interactions that support full CRUD operations. It probes contextual reasoning, adaptability to evolving user requests, and error recovery within stateful database dialogues.

Datasets

• DySQL-Bench — total 1072; splits: test (1072); repo https://github.com/Aurora-slz/Real-World-SQL-Bench

Metrics

• state-equivalence accuracy (primary) — range: [0, 1]
  • A task is considered solved if the hash of the database state produced by the model's action sequence matches the hash of the golden reference state. Timestamp columns are excluded during serialization to avoid execution-time volatility.

Input / output format

Input: Initial database state S^1 and a user instruction I specifying the user's information and interaction style.

Output: A sequence of SQL operations A = (a_1, a_2, ..., a_n) generated by the model to manipulate the database.

Scoring recipe

def evaluate(model_actions, golden_actions, initial_db):
    model_state = execute(model_actions, initial_db)
    golden_state = execute(golden_actions, initial_db)
    model_hash = compute_hash(serialize(model_state, exclude_timestamps=True))
    golden_hash = compute_hash(serialize(golden_state, exclude_timestamps=True))
    return int(model_hash == golden_hash)

Common pitfalls

• Timestamp columns (e.g., updated_at, created_at) must be explicitly excluded during state serialization to prevent false negatives due to execution time differences.
• Tables must be serialized in a strictly predefined order to ensure deterministic hashing.
• The evaluation requires executing both the model's and golden action sequences on identical copies of the initial database state.

Evidence (verbatim from paper)

To objectively determine whether the model-generated action sequence A leads to the same final database state as Ag, we compute a hash value for the database states S2 and S3. Specifically, all tables in the database are stored in a predefined order, and the data from each table are retrieved sequentially and serialized into a unified structure. Columns related to update or creation timestamps (e.g., updated_at, created_at, timestamp) are excluded to eliminate volatility caused by execution time differences. The serialized data are then converted into a hash value using a consistent hashing function. After executing both the model-generated action sequence A and the golden sequence Ag, on identical initial database copies, we compare their resulting hash values. If the two hash values are identical, we consider the task successfully solved.

Citation

@misc{sun2025rethinking,
  title={Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration},
  author={Sun et al. (2025)},
  year={2025},
  note={arXiv:2510.26495}
}

• arXiv: 2510.26495