Glow Bench Eval

glow-bench-eval

Leveraging LLM-GNN Integration for Open-World Question Answering over Knowledge Graphs — Abdallah et al. (2026) (arXiv:2604.13979, 2026)

What this evaluates

Evaluates open-world knowledge graph question answering by testing a model's ability to answer single-hop and multi-hop questions over incomplete graphs. It probes the integration of structural graph signals with textual semantics to handle missing answer paths and domain-specific reasoning without relying on fine-tuning or retrieval-only pipelines.

Datasets

• GLOW-Bench — total 1000; splits: test (-1); repo https://github.com/CoDS-GCS/GLOW
• Arxiv2023 — total ?; splits: test (-1)
• ogbn-arxiv — total ?; splits: test (-1)
• ogbn-products — total ?; splits: test (-1)

Metrics

• Exact Match Accuracy (primary) — range: percent
  • Calculates the percentage of questions where the model's predicted answer string exactly matches the ground truth answer string. It penalizes paraphrases or semantically related but non-identical outputs.
• Hierarchical Match Accuracy — range: percent
  • Uses an LLM-as-a-judge (GPT-4o-mini) to determine if the prediction is semantically equivalent to or a valid superclass/synonym of the gold answer. It is calculated as the percentage of predictions accepted by the judge.

Input / output format

Input: A natural language question over a knowledge graph. Depending on the method, the input may also include retrieved KG facts, GNN-predicted top-k candidate answers, and neighborhood text.

Output: A natural language answer string (e.g., entity name or concept).

Scoring recipe

def compute_exact_match(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
    return correct / len(golds)

def compute_hierarchical_match(predictions, golds, llm_judge):
    correct = 0
    for p, g in zip(predictions, golds):
        prompt = f"Is '{p}' semantically equivalent to or a valid superclass of '{g}'? Answer yes/no."
        if llm_judge(prompt).strip().lower() == "yes":
            correct += 1
    return correct / len(golds)

Common pitfalls

• Exact Match penalizes semantically correct but non-identical outputs (e.g., 'Singer' vs 'Artist'), requiring Hierarchical Match or human evaluation for fair assessment.
• Closed-world baselines like GCR fail in open-world settings because they assume complete answer paths exist, leading to hallucinated paths when edges are missing.
• GNN performance does not linearly correlate with QA accuracy; weak GNNs can still yield strong QA results when combined with textual retrieval, whereas methods like AskGNN degrade significantly.

Evidence (verbatim from paper)

Exact vs. Hierarchical Match Accuracy: LLMs paraphrase answers or return semantically related concepts rather than producing exact matches. For instance, the occupation "Singer" may be returned in place of "Artist", its superclass—potentially acceptable in some contexts. Figure 2 analyzes this phenomenon by comparing Hierarchical-Match and Exact-Match accuracies, where GPT-4o-mini is used as a judge for the Hierarchical-Match.

Citation

@misc{abdallah2026glow,
  title={Leveraging LLM-GNN Integration for Open-World Question Answering over Knowledge Graphs},
  author={Abdallah et al. (2026)},
  year={2026},
  note={arXiv:2604.13979}
}

• arXiv: 2604.13979