Kbqa Hit1 Eval

kbqa-hit1-eval

Topology-Aware Reasoning over Incomplete Knowledge Graph with Graph-Based Soft Prompting — Shuai Wang et al. (2026) (arXiv:2604.12503, 2026)

What this evaluates

Evaluates a model's ability to answer complex questions over knowledge graphs by retrieving relevant subgraphs and generating correct answer entities. It probes multi-hop reasoning capabilities and robustness to missing edges in incomplete knowledge bases.

Datasets

• ComplexWebQuestions — total ?; splits: (unstated)
• WebQuestionsSP — total ?; splits: (unstated)
• WebQuestions — total ?; splits: (unstated)
• GrailQA — total ?; splits: (unstated)

Metrics

• Hit@1 (primary) — range: percent
  • Hit@1 measures whether the ground-truth answer entity appears in the model's top-1 prediction. It is calculated as the percentage of test instances where the predicted answer exactly matches the gold answer.

Input / output format

Input: Natural language question paired with a retrieved subgraph (typically 2-hop neighbors around a topic entity) from the Freebase knowledge graph.

Output: A single predicted answer entity.

Scoring recipe

def compute_hit_at_1(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
    return (correct / len(golds)) * 100

Common pitfalls

• KG incompleteness is simulated by randomly removing 5–25% of edges connected to topic entities, which differs from standard IID evaluation settings.
• Some baselines (e.g., KG-Agent) pre-extract entities from both training and test sets, artificially reducing the search space and inflating generalization scores on GrailQA.
• Hit@1 only checks exact top-1 matches, ignoring cases where the model retrieves correct entities but ranks them second or lower.

Evidence (verbatim from paper)

Following prior work..., we used Hit@1 score as the evaluation metrics.

Citation

@misc{wang2026grasp,
  title={Topology-Aware Reasoning over Incomplete Knowledge Graph with Graph-Based Soft Prompting},
  author={Shuai Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.12503}
}

• arXiv: 2604.12503