Grasp Sparql Eval

grasp-sparql-eval

GRASP: Generic Reasoning And SPARQL Generation across Knowledge Graphs — Sebastian Walter et al. (2025) (arXiv:2507.08107, 2025)

What this evaluates

This evaluation probes an LLM's ability to generate correct SPARQL queries from natural language questions across diverse knowledge graphs. It measures how well the model can navigate graph structures, handle complex queries, and produce executable results that match ground-truth answers.

Datasets

• WebQuestionsSP (WQSP) — total ?; splits: test (200)
• ComplexWebQuestions (CWQ) — total ?; splits: test (200)
• QALD-7 — total 200; splits: test (200)
• QALD-10 — total ?; splits: test (200)
• SPINACH — total 200; splits: test (200)
• WikiWebQuestions (WWQ) — total ?; splits: test (200)

Metrics

• F1-score (primary) — range: percent
  • Averaged F1-score between predicted and ground-truth query results. Allows extra columns in predictions without penalty. Switches to exact F1 for results >1024 rows. ASK/SELECT mismatches score 1 if results are semantically equivalent. Empty GT samples excluded.

Input / output format

Input: Natural language question

Output: SPARQL query

Scoring recipe

def score(pred_sparkl, gt_sparkl, kg):
    pred_res = execute(pred_sparkl, kg)
    gt_res = execute(gt_sparkl, kg)
    if not gt_res: return None  # Skip empty GT
    if (is_ask(pred) and is_select(gt)) or (is_select(pred) and is_ask(gt)):
        return 1.0 if semantic_equiv(pred_res, gt_res) else 0.0
    if len(pred_res) > 1024:
        return exact_f1(pred_res, gt_res)
    return modified_f1(pred_res, gt_res, allow_extra_cols=True)
# Final metric = mean([score(p, g, kg) for p, g in dataset if score(p, g, kg) is not None])

Common pitfalls

• Ambiguous questions or differing LIMIT clauses cause valid predictions to receive unfairly low F1-scores.
• Samples with empty ground-truth results are excluded from the average, which can skew reported performance if not accounted for.
• The metric switches to exact F1 for large result sets (>1024 rows), removing the column-tolerance benefit and potentially penalizing otherwise correct queries.

Evidence (verbatim from paper)

All models are evaluated using the F1-score averaged across samples. We adopt SPINACH’s modification, which permits additional columns (e.g., labels) in the predicted results without penalty.121212For results exceeding 1,024 rows, we revert to the standard exact F1-score due to the computational cost of row-wise assignment. Samples with empty groundtruth are excluded. If the predicted query is an ASK query and the groundtruth is a SELECT query (or vice versa), we assign a score of 1 if their results are semantically equivalent.

Citation

@misc{walter2025grasp,
  title={GRASP: Generic Reasoning And SPARQL Generation across Knowledge Graphs},
  author={Sebastian Walter et al. (2025)},
  year={2025},
  note={arXiv:2507.08107}
}

• arXiv: 2507.08107