Cfkgr Eval

cfkgr-eval

Counterfactual Reasoning with Knowledge Graph Embeddings — Zellinger et al. (2024) (arXiv:2403.06936, 2024)

What this evaluates

Evaluates a model's ability to perform counterfactual reasoning on knowledge graphs by determining whether a target triple remains plausible after a hypothetical scenario is introduced, while also assessing knowledge retention for unaffected facts.

Datasets

• CFKGR-CoDEx-S — total ?; splits: test (-1); repo https://github.com/LenaZellinger/counterfactual_KGR
• CFKGR-CoDEx-M — total ?; splits: test (-1); repo https://github.com/LenaZellinger/counterfactual_KGR
• CFKGR-CoDEx-L — total ?; splits: test (-1); repo https://github.com/LenaZellinger/counterfactual_KGR
• CFKGR-CoDEx-M* — total 1159; splits: test (1159); repo https://github.com/LenaZellinger/counterfactual_KGR

Metrics

• Overall F1-score (primary) — range: [0, 100]
  • Harmonic mean of precision and recall across all test instances. The paper reports this metric as a percentage (0-100).
• F1 (E) — range: [0, 100]
  • F1 score computed against expected (rule-mined) labels for changed and unchanged facts.
• F1 (H) — range: [0, 100]
  • F1 score computed against human-annotated labels for changed and unchanged facts.
• Rule-wise accuracy — range: [0, 100]
  • Average accuracy per logical inference rule used to generate test triples, filtered to rules covering at least five test instances.

Input / output format

Input: A knowledge graph context containing a hypothetical scenario (e.g., a modified fact or added edge) and a target triple to evaluate. For LLMs, formatted as a natural language prompt asking if the triple is still plausible given the scenario.

Output: Binary classification label (plausible/true or false/untrue) or a continuous score used with relation-specific decision thresholds.

Scoring recipe

def compute_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1 * 100  # Paper reports as percentage

Common pitfalls

• Models may ignore the hypothetical scenario and simply predict based on the original knowledge graph facts (factual reasoning instead of counterfactual).
• ChatGPT often misinterprets the task by attempting to infer the target triple from the context rather than evaluating whether it still holds given the hypothetical change.
• Threshold tuning for CoDEx-L lacks hard negative triples, leading to suboptimal decision boundaries and lower reported performance.

Evidence (verbatim from paper)

First, we evaluate pre-trained KGEs, COULDD, and ChatGPT on our CFKGR datasets with expected labels to assess whether the methods can apply inference rules found by a rule mining system in hypothetical scenarios. In our second set of experiments, we evaluate on human-labeled data to check whether the methods also capture human reasoning... COULDD achieves the best results in terms of overall F1-score on all datasets.

Citation

@misc{zellinger2024counterfactual,
  title={Counterfactual Reasoning with Knowledge Graph Embeddings},
  author={Zellinger et al. (2024)},
  year={2024},
  note={arXiv:2403.06936}
}

• arXiv: 2403.06936