Bihopr Eval

bihopr-eval

BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain — Kim et al. (2025) (arXiv:2505.22240, 2025)

What this evaluates

This benchmark evaluates large language models on multi-hop, multi-answer reasoning tasks within the biomedical domain. It probes the model's ability to perform step-by-step inference over biomedical knowledge graphs and generate multiple valid answers for one-to-many-to-many relationships.

Datasets

• BioHopR — total ?; splits: test (-1)

Metrics

• Embedding-Based Precision (primary) — range: percent
  • Precision is computed using BioLORD-2023-C embeddings. For a prediction embedding p and ground truth answer embeddings {a_1,...,a_n}, compute cosine similarity cos(p, a_i). If max_i cos(p, a_i) > τ (where τ=0.9), the prediction is a true positive. Precision = |True Positives| / |Predicted Responses|. Reported separately for 1-hop (Prec_HOP1) and 2-hop (Prec_HOP2) tasks.

Input / output format

Input: A natural language question requiring 1-hop or 2-hop reasoning over biomedical knowledge graphs, expecting multiple valid answers.

Output: A single text string containing the predicted answer(s).

Scoring recipe

embeddings = BioLORD_2023_C()
pred_emb = embeddings.encode(prediction)
gt_embs = [embeddings.encode(a) for a in ground_truth_answers]
sims = [cosine_similarity(pred_emb, g) for g in gt_embs]
max_sim = max(sims)
if max_sim > 0.9:
    is_correct = True
else:
    is_correct = False
precision = sum(is_correct for all predictions) / len(predictions)

Common pitfalls

• The high similarity threshold (τ=0.9) strictly prioritizes precision, meaning semantically close but not highly similar answers are penalized.
• Using the maximum cosine similarity across multiple ground truth answers can mask partial correctness if one answer is very close but others are not.
• Evaluation is strictly zero-shot with temperature=0, which may disadvantage models that benefit from sampling or few-shot prompting.

Evidence (verbatim from paper)

The precision (Prec) is computed using the cosine similarity between the predicted response and the ground truth answer list, leveraging BioLORD-2023-C embeddings. Let p denote the embedding of the predicted response and {a_1,a_2,...,a_n} denote the embeddings of the ground truth answers. ... If the maximum cosine similarity across all ground truth answers satisfies: max_i cos(p,a_i) > τ, then the prediction is considered a true positive. The precision (Prec) is then calculated as: Prec = |True Positives| / |Predicted Responses|. We use τ=0.9 for BioLORD-2023-C embeddings after a grid search of threshold values from 0.5 to 0.9, which led an optimal setting with 0.9.

Citation

@misc{kim2025bihopr,
  title={BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain},
  author={Kim et al. (2025)},
  year={2025},
  note={arXiv:2505.22240}
}

• arXiv: 2505.22240