qhjqhj00 ilp-system-comparison-eval
A Critical Review of Inductive Logic Programming Techniques for Explainable AI — Zhang et al. (2021) (arXiv:2112.15319, 2021)
What this evaluates
Evaluates the predictive accuracy and learning efficiency of various Inductive Logic Programming (ILP) systems across synthetic grid-world tasks, scalability tests, and standard logical reasoning benchmarks. The protocol measures how well each system generalizes from positive and negative examples to learn correct logic programs under varying domain sizes and example counts.
Datasets
- Robot — total 200; splits: train (200)
- Robot2 — total 200; splits: train (200)
- Member — total ?; splits: train (-1)
- Benchmark ILP Problems — total 200; splits: train (200)
Metrics
accuracy(primary) — range: percent- Percentage of correctly classified examples by the learned logic program. If the system fails to learn a solution within the timeout, it receives a default score of 50%.
learning time— range: seconds- Wall-clock time in seconds required to learn a solution, measured per task and averaged over 10 runs.
Input / output format
Input: A set of positive and negative examples (facts) defining a target predicate, along with background knowledge. The number of examples and domain size are varied per experiment.
Output: A learned logic program consisting of Horn clauses defining the target predicate.
Scoring recipe
if timeout_exceeded:
return 50.0
else:
predictions = evaluate_learned_program(test_examples)
accuracy = sum(predictions == gold) / len(gold) * 100
return accuracy
Common pitfalls
- Systems that exceed the 60-second timeout are automatically assigned a default accuracy of 50%, which can skew comparisons.
- Metagol's performance is highly sensitive to the order of input examples, requiring them to be provided in increasing sizes.
- Comparisons are inherently biased because each ILP system is optimized for distinct tasks and uses different search strategies.
Evidence (verbatim from paper)
For each example, we use the same strategy as in the robot problem: We generate 100 positive and 100 negative examples. The default accuracy is therefore 50%. We repeat each experiment ten times to get the mean and the standard error.
Citation
@misc{zhang2021critical,
title={A Critical Review of Inductive Logic Programming Techniques for Explainable AI},
author={Zhang et al. (2021)},
year={2021},
note={arXiv:2112.15319}
}
- arXiv: 2112.15319