Caligraph Owl Reasoning Eval

caligraph-owl-reasoning-eval

The CaLiGraph Ontology as a Challenge for OWL Reasoners — Heist et al. (2021) (arXiv:2110.05028, 2021)

What this evaluates

Evaluates the scalability and logical correctness of OWL2 EL reasoners when processing large ontologies with complex owl:hasValue restrictions. It measures whether systems can successfully materialize subclass hierarchies and infer individual and literal assertions, and how long they take under memory and time constraints.

Datasets

• CaLiGraph — total 54914982; splits: clg_10 (35), clg_10e2 (510), clg_10e3 (43120), clg_10e4 (297266), clg_10e5 (4641400), clg_full (54914982)

Metrics

• Inferrable Assertions (primary) — range: count
  • Count of correctly inferred assertions across three categories: transitive type assertions, individual type assertions, and literal assertions. A reasoner passes if it infers all expected assertions for a given subset.
• Processing Time — range: hours
  • Wall-clock time required for the reasoner to complete ontology materialization and assertion inference. Experiments enforce a hard timeout of 72 hours.

Input / output format

Input: OWL2 EL ontology files (T-box and A-box) containing class hierarchies, object/datatype properties, owl:hasValue restrictions, and named individuals. Restrictions with explicit URIs/labels are transformed to blank nodes before import via OWL API.

Output: Set of inferred RDF assertions, including subclass relations, individual type assertions, and literal assertions derived from owl:hasValue restrictions.

Scoring recipe

for each reasoner, dataset in datasets:
  start = now()
  reasoner.load(dataset)
  reasoner.materialize()
  elapsed = now() - start
  if elapsed > 72*3600: return {status: 'timeout', time: elapsed}
  inferred = reasoner.get_inferred_assertions()
  expected = dataset.get_expected_assertions()
  correct = len(inferred & expected)
  return {time: elapsed, correct_assertions: correct}

Common pitfalls

• Disjointness axioms were explicitly removed from larger-scale datasets to prevent combinatorial explosion; evaluating them on the full ontology without removal will cause memory exhaustion.
• OWL API does not natively support restrictions with explicit URIs and labels; they must be converted to blank nodes before importing, otherwise the reasoner will fail or misinterpret the structure.
• Reasoners often handle individual-valued owl:hasValue restrictions correctly but fail on literal-valued ones, so capability checks must be split by restriction type.

Evidence (verbatim from paper)

Table 3 Overview of the different CaLiGraph subsets used for the evaluation... Inferrable Assertions... The reasoner performances are given in Fig. 4. We conducted experiments on a system with 32 Intel Xeon CPUs and 200GB RAM, and we set a timeout of 72h. Overall, ELK was the only system capable of processing all datasets (but not returning any of the inferrable individual and literal assertions, as discussed above).

Citation

@misc{heist2021caligraph,
  title={The CaLiGraph Ontology as a Challenge for OWL Reasoners},
  author={Heist et al. (2021)},
  year={2021},
  note={arXiv:2110.05028}
}

• arXiv: 2110.05028