Ontourl Eval

ontourl-eval

OntoURL: A Benchmark for Evaluating Large Language Models on Symbolic Ontological Understanding, Reasoning and Learning — Xiao Zhang et al. (2025) (arXiv:2505.11031, 2025)

What this evaluates

This benchmark evaluates large language models' ability to understand, reason over, and learn from symbolic ontologies. It probes hierarchical classification, logical inference, and structured knowledge construction across multiple domains and concept depths.

Datasets

• OntoURL — total 57303; splits: test (57303)

Metrics

• Accuracy (primary) — range: percent
  • Proportion of correctly answered multiple-choice and true/false questions across understanding (U1-U5) and reasoning (R1-R5) tasks.
• BERTScore F1 — range: [0, 1]
  • F1 score computed using BERT embeddings to compare generated class definitions against gold references (Task L1).
• F1 score — range: [0, 1]
  • Standard precision-recall F1 score applied to extracted triples or tuples for structured output tasks (L2-L5), after applying regular expressions to filter irrelevant text.

Input / output format

Input: Task instructions, questions, and answer options (for multiple-choice/true-false). In few-shot settings, 2 or 4 carefully selected examples demonstrating the expected reasoning pattern and output format are appended.

Output: Selected answer option for MC/TF tasks; natural language text for definition generation; syntactically valid triples or tuples for structured construction tasks.

Scoring recipe

def score_accuracy(preds, gold):
    return sum(1 for p, g in zip(preds, gold) if p == g) / len(gold)

def score_bertscore_f1(preds, gold):
    return bertscore_f1(preds, gold)

def score_f1_triples(preds, gold):
    pred_triples = extract_regex(preds)
    tp = len(set(pred_triples) & set(gold))
    fp = len(set(pred_triples) - set(gold))
    fn = len(set(gold) - set(pred_triples))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Models often output irrelevant text alongside valid triples/tuples; regex extraction is required to isolate the structured output before scoring.
• Chain-of-Thought prompting yields mixed results: it improves learning tasks but can significantly degrade performance on understanding and reasoning tasks due to domain-specific inference patterns.
• Few-shot prompting effectiveness scales with model size; smaller models may suffer from context length limitations when processing longer prompts.

Evidence (verbatim from paper)

As shown in Table[1], we use task-appropriate metrics: Accuracy for multiple-choice and true/false questions (tasks U1-U5, R1-R5), BERTScore F1 Zhang et al. ([2020]) for text generation (task L1), and F1 score for structured outputs such as triples or tuples (tasks L2-L5). We apply regular expressions to extract valid triples or tuples from the model’s responses to mitigate the impact of irrelevant text.

Citation

@misc{zhang2025ontourl,
  title={OntoURL: A Benchmark for Evaluating Large Language Models on Symbolic Ontological Understanding, Reasoning and Learning},
  author={Xiao Zhang et al. (2025)},
  year={2025},
  note={arXiv:2505.11031}
}

• arXiv: 2505.11031