Rel Eval

rel-eval

Evaluating Relational Reasoning in LLMs with REL — Fesser et al. (2026) (arXiv:2604.12176, 2026)

What this evaluates

This benchmark probes an LLM's ability to perform high-arity relational reasoning and multi-constraint integration across scientific domains. It isolates the difficulty of jointly binding independent entities to satisfy a relation, independent of prompt length or in-context learning.

Datasets

• REL — total ?; splits: test (-1); repo https://github.com/ada-f/relational_reasoning

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly answered multiple-choice questions. For REL-A, models choose 1 of 8 options, yielding a trivial accuracy of 12.5%.
• exact match — range: [0, 1]
  • All-or-nothing scoring where the model's prediction must exactly match the ground-truth set of taxa (REL-B) or canonical SMILES string(s) (REL-C).
• task completion rate — range: [0, 1]
  • Proportion of instances where the model successfully completes the task requirements, used as a proxy for accuracy in REL-B and REL-C.
• recall — range: [0, 1]
  • TP / (TP + FN), measuring the fraction of true positive entities (e.g., missing isomers) correctly identified by the model.
• precision — range: [0, 1]
  • TP / (TP + FP), measuring the fraction of predicted entities that are actually correct.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).

Input / output format

Input: REL-A: RPMs formatted as row arrays separated by pipes, with a '?' marking the missing value, plus 8 answer choices. REL-B: Biological sequences with motifs, asking to detect homoplasy and identify taxa. REL-C: Canonicalized SMILES strings of molecules, asking to identify isomers or substructures.

Output: REL-A: Selection of one of the 8 provided answer choices. REL-B: Boolean flag for homoplasy presence plus an exact set of homoplastic taxa names. REL-C: Canonical SMILES string(s) for the predicted molecule(s).

Scoring recipe

def score(predictions, gold, task):
    if task == 'REL-A':
        return 1.0 if predictions == gold else 0.0
    elif task in ['REL-B', 'REL-C']:
        return 1.0 if predictions == gold else 0.0
    # For REL-C3 recall/precision/F1:
    tp = len(set(predictions) & set(gold))
    fn = len(set(gold) - set(predictions))
    fp = len(set(predictions) - set(gold))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return prec, rec, f1

Common pitfalls

• Confusing Relational Complexity (RC) with input size or prompt length; RC specifically measures the minimum number of independent entities that must be jointly bound, not the number of tokens or rows.
• For REL-B, failing to exactly match the set of homoplastic taxa; partial credit is not given, and all other outcomes are counted as incorrect.
• For REL-C, not canonicalizing predicted SMILES strings before comparison, which would incorrectly penalize chemically equivalent representations.

Evidence (verbatim from paper)

Model responses are evaluated by canonicalizing both predicted and ground-truth SMILES strings and comparing the canonical forms for exact match, ensuring that chemically equivalent SMILES representations are treated as correct.

Citation

@misc{fesser2026rel,
  title={Evaluating Relational Reasoning in LLMs with REL},
  author={Fesser et al. (2026)},
  year={2026},
  note={arXiv:2604.12176}
}

• arXiv: 2604.12176