Kacc Eval

kacc-eval

KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion — Zhou et al. (2020) (arXiv:2004.13631, 2020)

What this evaluates

Evaluates models' capabilities in knowledge abstraction, concretization, and completion within entity-concept knowledge graphs. It specifically probes multi-hop reasoning through hierarchical relations and cross-view knowledge transfer between entities and concepts.

Datasets

• KACC — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/thunlp/KACC

Metrics

• MRR — range: [0, 1]
  • Mean Reciprocal Rank: the average of 1/rank for the correct answer across all queries.
• Hits@1 — range: [0, 1]
  • The fraction of queries where the correct answer is ranked first in the candidate list.
• Hits@10 (primary) — range: [0, 1]
  • The fraction of queries where the correct answer is ranked within the top 10 predictions. Used as the headline metric for the overall KACC score.

Input / output format

Input: A knowledge triple with one missing element (head, relation, ?) or (?, relation, tail), drawn from entity-level or concept-level graphs, sometimes requiring multi-hop reasoning.

Output: A ranked list of candidate entities or a relevance score for each candidate in the knowledge graph.

Scoring recipe

def compute_kacc_overall(predictions, golds, tasks):
    valid_tasks = [t for t in tasks if t not in ['CGC-Single', 'EGC-Single']]
    hits10_scores = []
    for task in valid_tasks:
        correct = sum(1 for pred, gold in zip(predictions[task], golds[task]) if gold in pred[:10])
        hits10_scores.append(correct / len(predictions[task]))
    return sum(hits10_scores) / len(hits10_scores)

Common pitfalls

• Multi-hop tasks (MKA, MKC) cause drastic performance degradation (>0.51 drop in MRR) compared to single-hop tasks, indicating composition rules are hard to learn.
• Models often bias toward entity triples over concept triples due to data imbalance, leading to poor performance on concept-level completion (CGC-Joint).
• Joint training settings do not consistently outperform single-graph settings for concept graphs, contrary to entity graph results.

Evidence (verbatim from paper)

Similar to GLUE (Wang et al., 2019a), we average Hits@10 scores of each method on all tasks (except CGC-Single and EGC-Single) to get final scores. We also compute the average scores for knowledge abstraction (KA), knowledge concretization (KCon), and knowledge completion (KCom).

Citation

@misc{zhou2020kacc,
  title={KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion},
  author={Zhou et al. (2020)},
  year={2020},
  note={arXiv:2004.13631}
}

• arXiv: 2004.13631