Ontology Subsumption Inference Eval

ontology-subsumption-inference-eval

Language Model Analysis for Ontology Subsumption Inference — He et al. (2023) (arXiv:2302.06761, 2023)

What this evaluates

Evaluates large language models' ability to perform ontology subsumption inference by framing it as a binary natural language inference task. The model must predict whether a hypothesis concept subsumes a premise concept based on verbalized OWL axioms.

Datasets

• biMNLI — total ?; splits: train (-1), dev (-1), test (-1)
• Schema.org (Atomic SI) — total ?; splits: train (-1), dev (-1), test (-1)
• DOID (Atomic SI) — total ?; splits: train (-1), dev (-1), test (-1)
• FoodOn (Atomic SI) — total ?; splits: train (-1), dev (-1), test (-1)
• GO (Atomic SI) — total ?; splits: train (-1), dev (-1), test (-1)
• FoodOn (Complex SI) — total ?; splits: train (-1), dev (-1), test (-1)
• GO (Complex SI) — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• accuracy (primary) — range: percent
  • Standard classification accuracy: the proportion of correctly predicted labels ('positive' or 'negative') out of the total number of instances. Reported as mean accuracy with standard deviation across random seeds and template/label combinations.

Input / output format

Input: Verbalized subsumption axioms formatted as premise and hypothesis pairs, wrapped in prompt templates (T1 or T2) containing a token. For K-shot settings, K positive and K negative examples are sampled from training/dev sets and appended to the prompt.

Output: A binary classification prediction ('positive' or 'negative') derived by selecting the class whose label words (L1-L3) have the highest masked-token probability.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Zero-shot performance heavily depends on the specific prompt template and label word combination used, leading to high variance across seeds.
• K-shot refers to the number of samples per label (positive/negative), not total samples, which affects few-shot scaling analysis.
• Complex SI tasks (nested/recursive concepts) are significantly harder than Atomic SI, and domain-specific pretraining does not guarantee better performance on complex reasoning.

Evidence (verbatim from paper)

For both LMs in Table 3, we report results of $K$ -shot settings with $K \in {0,4,32,128}$ . We additionally present the results of the fully supervised setting for roberta-large as the oracle. For each setting, we report the averaged accuracy and standard deviation (where applicable).

Citation

@misc{he2023language,
  title={Language Model Analysis for Ontology Subsumption Inference},
  author={He et al. (2023)},
  year={2023},
  note={arXiv:2302.06761}
}

• arXiv: 2302.06761