Lemur Retrieval Eval

lemur-retrieval-eval

LEMUR: A Corpus for Robust Fine-Tuning of Multilingual Law Embedding Models for Retrieval — Baba Ahmadi et al. (2026) (arXiv:2602.09570, 2026)

What this evaluates

Evaluates the ability of multilingual embedding models to retrieve relevant legislative documents given structured metadata queries. It probes cross-lingual semantic alignment and domain-adaptive retrieval performance across varying language resource levels.

Datasets

• LEMUR — total 24953; splits: train (-1), val (-1), test (-1); repo https://github.com/nargesbh/eur_lex

Metrics

• Acc@k (primary) — range: [0, 1]
  • Accuracy at rank k: the fraction of test queries for which the ground-truth positive document appears in the top-k retrieved results. Computed for k=1, 3, and 5.

Input / output format

Input: Query: structured metadata block of a legislative act. Document: substantive legislative text extracted from the corresponding PDF.

Output: Ranked list of documents based on cosine similarity scores between query and document embeddings.

Scoring recipe

def compute_acc_at_k(retrieved_docs, gold_doc, k):
    return 1.0 if gold_doc in retrieved_docs[:k] else 0.0

def evaluate(dataset, model):
    correct = 0
    for query, gold_doc in dataset:
        query_emb = model.encode(query)
        doc_embs = model.encode(dataset.documents)
        scores = cosine_similarity(query_emb, doc_embs)
        top_k_indices = argsort(scores, descending=True)[:k]
        top_k_docs = [dataset.documents[i] for i in top_k_indices]
        correct += compute_acc_at_k(top_k_docs, gold_doc, k)
    return correct / len(dataset)

Common pitfalls

• Documents are truncated (8–15% of corpus) with 40–50% of tokens removed, which may disproportionately affect retrieval for long acts.
• Training uses in-batch negatives, but evaluation retrieves against the full corpus; models may overfit to batch-level discrimination.
• Cross-lingual evaluation requires aligned splits; mismatched legislative acts across languages will break the protocol.

Evidence (verbatim from paper)

The data is split into 60% training, 20% validation, and 20% test sets, independently for each language or language pair, such that the same underlying legislative acts are assigned to the same split across languages, with each split containing the corresponding translations of those acts. Performance is measured using Acc@k for 1/3/5 on test queries evaluated against the test document collection

Citation

@misc{ahmadi2026lemur,
  title={LEMUR: A Corpus for Robust Fine-Tuning of Multilingual Law Embedding Models for Retrieval},
  author={Baba Ahmadi et al. (2026)},
  year={2026},
  note={arXiv:2602.09570}
}

• arXiv: 2602.09570