Semantic Kg Eval

semantic-kg-eval

Semantic-KG: Using Knowledge Graphs to Construct Benchmarks for Measuring Semantic Similarity — Wei et al. (2025) (arXiv:2511.19925, 2025)

What this evaluates

Evaluates the ability of semantic similarity methods to correctly classify pairs of natural language statements as semantically similar (label 1) or dissimilar (label 0). It specifically probes how well models handle controlled semantic variations (node and edge perturbations) across general and domain-specific knowledge domains.

Datasets

• Semantic-KG Benchmark — total ?; splits: val (-1), test (-1); repo https://github.com/QiyaoWei/semantic-kg

Metrics

• F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall for binary classification of statement pairs as similar (1) or dissimilar (0). Continuous scores from embedding or NLP methods are converted to binary labels using a threshold optimized on validation data.

Input / output format

Input: Pairs of natural language statements generated from knowledge graph subgraphs (positive pairs from the same subgraph, negative pairs from original vs. perturbed subgraphs).

Output: Binary label (0 or 1) indicating semantic similarity. For continuous methods, a score is computed and thresholded to a binary label.

Scoring recipe

best_f1 = 0
best_threshold = 0.5
for threshold in np.arange(0, 1, 0.01):
    preds = [1 if score >= threshold else 0 for score in val_scores]
    f1 = f1_score(val_labels, preds)
    if f1 > best_f1:
        best_f1 = f1
        best_threshold = threshold
test_preds = [1 if score >= best_threshold else 0 for score in test_scores]
final_f1 = f1_score(test_labels, test_preds)

Common pitfalls

• Threshold selection is performed on validation data to maximize F1, which can introduce validation leakage if the split is not strictly held out.
• Continuous scores from embedding or NLP methods must be thresholded to binary labels for F1 calculation, making results highly sensitive to the chosen threshold.
• Aggregate F1 scores mask significant performance disparities across perturbation types (node vs. edge) and domains, requiring stratified reporting.

Evidence (verbatim from paper)

To compute this threshold we split the data into validation and test data and find the threshold that maximizes the F1-score using the validation data. The test data is then used to report the final results.

Citation

@misc{wei2025semantickg,
  title={Semantic-KG: Using Knowledge Graphs to Construct Benchmarks for Measuring Semantic Similarity},
  author={Wei et al. (2025)},
  year={2025},
  note={arXiv:2511.19925}
}

• arXiv: 2511.19925