Maggn Rule Extraction Eval

maggn-rule-extraction-eval

Sound Logical Explanations for Mean Aggregation Graph Neural Networks — Morris et al. (2025) (arXiv:2511.11593, 2025)

What this evaluates

Evaluates the predictive performance of mean-aggregation GNNs with non-negative weights on link prediction and node classification tasks, while assessing the soundness, monotonicity, and logical complexity of the extracted explanatory rules.

Datasets

• WN18RRv1 — total ?; splits: train (-1), val (-1), test (-1)
• FB237v1 — total ?; splits: train (-1), val (-1), test (-1)
• NELLv1 — total ?; splits: train (-1), val (-1), test (-1)
• LUBM — total ?; splits: train (-1), test (-1)
• LogInfer-WN-hier — total ?; splits: train (-1), val (-1), test (-1)
• LogInfer-WN-sym — total ?; splits: train (-1), val (-1), test (-1)
• LogInfer-WN-hier_nmhier — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy (primary) — range: percent
  • Accuracy = (True Positives + True Negatives) / Total Instances. Used to select the optimal classification threshold across 108 values on the validation set.
• precision — range: percent
  • Precision = True Positives / (True Positives + False Positives). Measures the proportion of predicted positive links/nodes that are actually positive.
• recall — range: percent
  • Recall = True Positives / (True Positives + False Negatives). Measures the proportion of actual positive links/nodes that are correctly predicted.
• F1 score — range: percent
  • F1 = 2 * (Precision * Recall) / (Precision + Recall). Harmonic mean of precision and recall.

Input / output format

Input: Graph-structured data with entities and relations for link prediction or node classification. For LogInfer datasets, 10% of input facts are randomly set aside per epoch as ground truth positive targets.

Output: Binary predictions or probabilities for links/nodes, and extracted monotonic first-order logic rules (e.g., A(X) ∧ B(Y) → C(X,Y) or ⊤ ⊑ Concept) that soundly explain the model's predictions.

Scoring recipe

def compute_metrics(preds, gold, threshold):
    preds_bin = (preds >= threshold).astype(int)
    tp = np.sum((preds_bin == 1) & (gold == 1))
    fp = np.sum((preds_bin == 1) & (gold == 0))
    fn = np.sum((preds_bin == 0) & (gold == 1))
    tn = np.sum((preds_bin == 0) & (gold == 0))
    acc = (tp + tn) / len(gold)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return acc, prec, rec, f1

Common pitfalls

• Threshold is not fixed at 0.5; it is selected on the validation set across 108 values to maximize accuracy.
• Non-negative weight constraint is applied via post-step clamping, which may regularize the model but differs from standard unconstrained training.
• Rule extraction search space is truncated by dataset-specific limits on body concept counts (1 to 15), meaning longer rules are never evaluated.

Evidence (verbatim from paper)

For all trained models, we compute standard classification metrics, such as precision, recall, accuracy, and F1 score. For each model, we choose the classification threshold by computing the accuracy on the validation set across a range of 108 thresholds between 0 and 1, selecting the one which maximises accuracy.

Citation

@misc{morris2025soundlogical,
  title={Sound Logical Explanations for Mean Aggregation Graph Neural Networks},
  author={Morris et al. (2025)},
  year={2025},
  note={arXiv:2511.11593}
}

• arXiv: 2511.11593