Dw Bench Eval

dw-bench-eval

DW-Bench: Benchmarking LLMs on Data Warehouse Graph Topology Reasoning — Ahmed et al. (2026) (arXiv:2604.18964, 2026)

What this evaluates

Evaluates LLMs' ability to reason about data warehouse graph topologies, specifically focusing on foreign key path enumeration, data lineage impact analysis, and multi-hop graph traversal. It probes whether models can perform structural graph reasoning versus relying on lexical cues, using heterogeneous schema graphs with foreign key and lineage edges.

Datasets

• DW-Bench — total 1046; splits: test (1046); repo https://github.com/AJamal27891/dw-bench

Metrics

• Micro-EM (primary) — range: percent
  • Overall exact match accuracy across all questions. For list-typed tasks, predictions and gold are normalized by removing the queried table before comparison to avoid penalizing harmless self-inclusion.
• Macro-EM — range: percent
  • Mean of per-subtype exact match scores, weighting all 13 subtypes equally to prevent subtype distribution bias from inflating the average.

Input / output format

Input: Schema graph topology (tables, foreign keys, lineage edges) presented as flat text or accessed via tool/code execution, paired with deterministic template questions.

Output: Predicted answers to topology reasoning questions (e.g., paths, sets of tables, counts), evaluated via exact match.

Scoring recipe

def compute_em(predictions, golds, queried_tables):
    em_scores = []
    for pred, gold, q_table in zip(predictions, golds, queried_tables):
        norm_pred = set(pred) - {q_table}
        norm_gold = set(gold) - {q_table}
        em_scores.append(1.0 if norm_pred == norm_gold else 0.0)
    return sum(em_scores) / len(em_scores)

Common pitfalls

• Models often rely on lexical cues (table names) rather than actual graph traversal, as obfuscation experiments show large performance drops for static baselines.
• Easy question scores are heavily inflated by the join_path subtype (33% of questions), masking severe failures on multi-hop compositional tasks.
• List-typed tasks require target-node normalization; failing to strip the queried table from both prediction and gold unfairly penalizes models for harmless self-inclusion.

Evidence (verbatim from paper)

We report pooled Micro-EM (overall exact match) and Macro-EM (mean of per-subtype EM, weighting all 13 subtypes equally). For list-typed topology tasks (e.g., connected component membership), we apply target-node normalization (stripping the queried table from both prediction and gold prior to scoring) ensuring models are not penalized for harmless self-inclusion.

Citation

@misc{ahmed2026dwbenc,
  title={DW-Bench: Benchmarking LLMs on Data Warehouse Graph Topology Reasoning},
  author={Ahmed et al. (2026)},
  year={2026},
  note={arXiv:2604.18964}
}

• arXiv: 2604.18964