Prism Eval

prism-eval

PRISM: Fine-Grained Paper-to-Paper Retrieval with Multi-Aspect-Aware Query Optimization — Park et al. (2025) (arXiv:2507.10057, 2025)

What this evaluates

Evaluates fine-grained, multi-aspect-aware paper-to-paper retrieval by decomposing long-form query papers into aspect-specific views and segmenting candidate papers into section-level representations for targeted retrieval.

Datasets

• SciFullBench — total ?; splits: ICLR-References (-1), ICLR-Citations (-1), NeurIPS-Citations (-1), ACL-Citations (-1); repo https://github.com/psw0021/PRISM
• PatentFullBench — total ?; splits: References (-1), Citations (-1); repo https://github.com/psw0021/PRISM

Metrics

• Recall@K (primary) — range: [0, 1]
  • The proportion of ground-truth relevant documents that appear in the top-K retrieved results. Calculated as |Top-K ∩ Gold| / |Gold|.
• CHV Ratio — range: other
  • Convex Hull Volume Ratio measuring semantic coverage diversity in the latent embedding space, computed as the ratio of coverage between two experimental configurations over the same test set.

Input / output format

Input: Query paper (abstract or full text/chunks) and a corpus of candidate papers (abstracts or full text/chunks).

Output: Ranked list of candidate papers.

Scoring recipe

def recall_at_k(predictions, gold, k):
    top_k = predictions[:k]
    relevant = sum(1 for p in top_k if p in gold)
    return relevant / len(gold)

Common pitfalls

• Full papers are truncated to the maximum context length of the embedding model.
• Iterative retrieval explicitly skips already-selected candidates to prevent redundancy.
• Aspect-aware cache is required to avoid redundant exploration within each aspect branch.

Evidence (verbatim from paper)

Table 1 presents the main results, where CoR outperforms all baselines across various settings, validating the effectiveness of our proposed framework for full paper-to-paper retrieval. Notably, when using the same domain-agnostic retriever, CoR surpasses abstract-to-abstract (A2A) baselines by an average of 5.16% in Recall, demonstrating that simply relying on abstracts is suboptimal compared to our aspect-driven approach.

Citation

@misc{park2025prism,
  title={PRISM: Fine-Grained Paper-to-Paper Retrieval with Multi-Aspect-Aware Query Optimization},
  author={Park et al. (2025)},
  year={2025},
  note={arXiv:2507.10057}
}

• arXiv: 2507.10057