Fanns Benchmark Eval

fanns-benchmark-eval

Benchmarking Filtered Approximate Nearest Neighbor Search Algorithms on Transformer-based Embedding Vectors — Iff et al. (2025) (arXiv:2507.21989, 2025)

What this evaluates

Evaluates the accuracy and efficiency of filtered approximate nearest neighbor search (FANNS) algorithms on high-dimensional transformer-based embeddings. It measures how well different indexing methods maintain recall under various real-world attribute filtering constraints while scaling to millions of vectors.

Datasets

• arxiv-for-fanns-medium — total 100000; splits: test (100000); repo https://github.com/spcl/fanns-benchmark
• arxiv-for-fanns-large — total 2700000; splits: test (2700000); repo https://github.com/spcl/fanns-benchmark

Metrics

• recall@10 (primary) — range: [0, 1]
  • Intersection over k of predicted and ground truth nearest neighbor sets: |knn_alg ∩ knn_gt| / k.
• QPS — range: other
  • Queries processed per second during single-threaded query execution.
• index construction time — range: other
  • Wall-clock time to build the index using all available hardware threads.
• peak memory usage — range: other
  • Maximum RAM consumed during index construction or query execution.
• index size — range: other
  • Disk/memory footprint of the built index.

Input / output format

Input: 4096-dimensional normalized transformer embedding vectors paired with categorical/numerical attribute filters (equality, range, or equality-with-missing-value-is-satisfied). Queries specify a target vector and a filter condition.

Output: A ranked list of k=10 candidate item identifiers (indices) returned by the FANNS algorithm.

Scoring recipe

def compute_recall_at_10(pred_ids, gt_ids, k=10):
    pred_set = set(pred_ids[:k])
    gt_set = set(gt_ids[:k])
    return len(pred_set & gt_set) / k

Common pitfalls

• Parameter tuning is highly sensitive to dataset scale and filter type; greedy search may converge to local optima, especially for methods with long construction times.
• ACORN's baseline implementation checks filters on all visited vertices rather than just traversal nodes, inflating overhead; the paper reports both upper and lower bound curves to address this inconsistency.
• Transformer embeddings are 4096-dimensional, causing int32 overflows in memory allocation for several methods when scaling to 2.7M vectors, requiring code modifications.

Evidence (verbatim from paper)

We focus on the well-established recall vs. queries per second (QPS) plots, which illustrate the trade-off between accuracy and query throughput achieved by each method. We define recall@k as: $$ \operatorname {r e c a l l} @ k = \frac {\left| \mathrm {k n n} _ {\mathrm {a l g}} \cap \mathrm {k n n} _ {\mathrm {g t}} \right|}{k} $$ where $\mathrm{knn}{\mathrm{alg}}$ denotes the set of $k$ nearest neighbors returned by the algorithm, and $\mathrm{knn}{\mathrm{gt}}$ is the ground truth.

Citation

@misc{iff2025benchmarking,
  title={Benchmarking Filtered Approximate Nearest Neighbor Search Algorithms on Transformer-based Embedding Vectors},
  author={Iff et al. (2025)},
  year={2025},
  note={arXiv:2507.21989}
}

• arXiv: 2507.21989