Graphlog Eval

graphlog-eval

Evaluating Logical Generalization in Graph Neural Networks — Sinha et al. (2020) (arXiv:2003.06560, 2020)

What this evaluates

Evaluates the ability of Graph Neural Networks to induce, compose, and generalize logical rules across synthetic knowledge graphs. It probes relational reasoning, multi-task learning capacity, and catastrophic forgetting in continual learning settings.

Datasets

• GraphLog — total 57; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted logical relations on the test split. Reported as mean ± standard deviation across worlds or runs.

Input / output format

Input: Synthetic knowledge graphs (worlds) containing entities and relations, structured for GNN message passing.

Output: Predicted logical relations for query triples.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Models may exploit dataset-specific biases if evaluated on too few worlds; GraphLog uses 57 distinct worlds to mitigate this.
• Training on more than 20 diverse worlds causes capacity saturation and degrades multi-task performance.
• Catastrophic forgetting occurs rapidly in continual learning unless the representation function is shared across worlds.

Evidence (verbatim from paper)

In Figure 6: We evaluate the performance of all the models in a continual learning setup. The blue curve shows the accuracy on the current world and the orange curve shows the mean accuracy on all the previously seen worlds. As the model trains on new worlds, its performance on the previously seen worlds degrades rapidly.

Citation

@misc{sinha2020graphlog,
  title={Evaluating Logical Generalization in Graph Neural Networks},
  author={Sinha et al. (2020)},
  year={2020},
  note={arXiv:2003.06560}
}

• arXiv: 2003.06560