Scholarly Qald Eval

scholarly-qald-eval

NFDI4DS Shared Tasks for Scholarly Document Processing — Abu Ahmad et al. (2025) (arXiv:2509.22141, 2025)

What this evaluates

Tests natural language interfaces for querying scholarly knowledge graphs (DBLP, ORKG) and hybrid multi-source QA, evaluating question-to-SPARQL translation and answer generation accuracy.

Datasets

• Scholarly QALD — total 10000; splits: train (-1), test (-1); repo https://kgqa.github.io/scholarly-QALD-challenge/2024/

Metrics

• Exact Match (primary) — range: [0, 1]
  • Binary score: 1 if predicted answer exactly matches the gold answer, 0 otherwise.

Input / output format

Input: Natural language question over a scholarly knowledge graph or hybrid textual/KG resources.

Output: SPARQL query (for KGQA) or natural language answer (for Hybrid-QA).

Scoring recipe

pred_answer = model.generate_answer(question)
gold_answer = gold_label
exact_match = 1 if pred_answer == gold_answer else 0
f_score = f1_score(gold_answer, pred_answer)
meteor_score = meteor(gold_answer, pred_answer)

Common pitfalls

• The 2023 edition focused on KGQA (F1 score), while 2024 expanded to Hybrid-QA with Exact Match, F-Score, and METEOR, requiring careful version tracking.
• SPARQL generation requires exact syntactic matching for Exact Match, which is stricter than semantic equivalence and often penalizes valid but differently formatted queries.

Evidence (verbatim from paper)

The task attracted seven teams and was evaluated using the F1 score. In 2024, a new Hybrid Question Answering (Hybrid-QA) task was introduced, requiring participants to fetch answers by querying multiple scholarly KGs and textual resources, reflecting more realistic, multi-source user needs in the scholarly domain. Three teams submitted results, and the evaluation was expanded to a broader set of metrics, including Exact Match, F-Score, and METEOR.

Citation

@misc{abuhmad2025nfdi4ds,
  title={NFDI4DS Shared Tasks for Scholarly Document Processing},
  author={Abu Ahmad et al. (2025)},
  year={2025},
  note={arXiv:2509.22141}
}

• arXiv: 2509.22141