Ragppi Eval

ragppi-eval

RAGPPI: RAG Benchmark for Protein-Protein Interactions in Drug Discovery — Youngseung Jeon et al. (2025) (arXiv:2505.23823, 2025)

What this evaluates

Evaluates the factual accuracy and atomic fact alignment of LLMs and RAG systems when answering questions about protein-protein interactions (PPIs) in drug discovery. It probes whether models can correctly identify biological, functional, or physical effects between proteins without hallucinating domain-specific details.

Datasets

• RAGPPI — total 372; splits: test (372); repo https://github.com/youngseungjeon/RAGPPI

Metrics

• F1 (Cosine similarity of atomic facts) (primary) — range: [0, 1]
  • Mean cosine similarity between embeddings of atomic facts extracted from the model's answer and the ground-truth reference answer.
• F2 (Low-similarity fact counts) — range: count
  • Count of atomic facts in the model's answer whose cosine similarity to any reference fact falls below a predefined threshold.
• Accuracy (Ensemble) — range: percent
  • Binary correctness score from an auto-evaluation LLM, aggregated via majority voting across three distinct evaluation prompts (M1, M2, M3).

Input / output format

Input: Question about a specific protein-protein interaction ($Q_{ppi}$). For RAG systems, retrieved context/documents are appended. Prompt includes instructions and 5 expert-validated few-shot QA examples.

Output: Natural language answer ($A_{sys}$) describing the biological, functional, or physical effect of the PPI.

Scoring recipe

1. Generate answers for each of the 372 PPIs using the target system.
2. Extract atomic facts from the system answer and the ground-truth reference.
3. Compute pairwise cosine similarities between system and reference fact embeddings.
4. Calculate F1 as the mean of these similarities.
5. Calculate F2 as the count of similarities below a low-similarity threshold.
6. Pass each (Q, A_sys, A_ref) triple to an auto-eval LLM to obtain M1, M2, M3 scores.
7. Compute Accuracy_Ensemble as the majority vote of M1, M2, M3.

Common pitfalls

• Semantic similarity metrics (e.g., BERTScore) do not reliably indicate factual appropriateness in this domain.
• RAG systems may underperform base LLMs due to suboptimal retrievers introducing irrelevant or noisy context.
• Individual auto-evaluation prompts (M1-M3) show high variance; relying on a single evaluator yields unstable accuracy scores.

Evidence (verbatim from paper)

We sample 372 PPIs from our database, preserving the distribution of frequency levels and PPI types, and generate biological, functional, or physical effects from each PPI. For each system, a QA pair $\mathcal{P}(Q_{ppi},A_{sys})$ would be generated. Once the results are generated, the question-answer pairs, $\mathcal{P}(Q_{ppi},A_{sys1},A_{sys2},\dots,A_{sys6})$ , are passed to our auto-eval LLM. Table 4 shows performance of the LLM- and RAG-based models on the cosine similarity of the atomic facts with the reference $(F_1)$, low-similarity fact counts $(F_2)$, and the accuracy.

Citation

@misc{jeon2025ragppi,
  title={RAGPPI: RAG Benchmark for Protein-Protein Interactions in Drug Discovery},
  author={Youngseung Jeon et al. (2025)},
  year={2025},
  note={arXiv:2505.23823}
}

• arXiv: 2505.23823