Hne Benchmark Eval

hne-benchmark-eval

Heterogeneous Network Representation Learning: A Unified Framework with Survey and Benchmark — Carl Yang et al. (2020) (arXiv:2004.00216, 2020)

What this evaluates

This benchmark evaluates the quality and robustness of heterogeneous network embedding (HNE) algorithms across diverse real-world graphs. It probes how well learned representations preserve multi-type structural and attribute information, measured via downstream node classification and link prediction tasks.

Datasets

• DBLP — total 1989077; splits: train (-1), test (-1); repo https://github.com/yangji9181/HNE
• Yelp — total 82465; splits: train (-1), test (-1); repo https://github.com/yangji9181/HNE
• Freebase — total 12164758; splits: train (-1), test (-1); repo https://github.com/yangji9181/HNE
• PubMed — total 63109; splits: train (-1), test (-1); repo https://github.com/yangji9181/HNE

Metrics

• macro-F1 (primary) — range: [0, 1]
  • F1 score computed per label and averaged across all labels.
• micro-F1 — range: [0, 1]
  • F1 score computed globally by counting total true positives, false negatives, and false positives across all nodes.
• AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, treating link prediction as a binary classification problem.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank, treating link prediction as a ranking/retrieval problem over candidate node pairs.

Input / output format

Input: Heterogeneous graph defined by node types, link types, adjacency matrices, and optional node attributes/labels. For evaluation, a fixed 80/20 split of links or labeled nodes is held out.

Output: Dense vector embeddings for each node in the graph. For classification: predicted class labels. For link prediction: binary scores or ranked lists of candidate node pairs.

Scoring recipe

# Node Classification
train_mask, test_mask = split_labels(0.8)
clf = LinearSVC().fit(embeddings[train_mask], labels[train_mask])
preds = clf.predict(embeddings[test_mask])
macro_f1 = f1_score(labels[test_mask], preds, average='macro')
micro_f1 = f1_score(labels[test_mask], preds, average='micro')

# Link Prediction
pos_train, pos_test = split_links(0.8)
neg_candidates = get_2hop_neighbors(pos_test)
pair_feats = hadamard(pos_train + neg_candidates)
link_clf = LinearSVC().fit(pair_feats, [1]*len(pos_train) + [0]*len(neg_candidates))
scores = link_clf.predict_proba(pair_feats)[:, 1]
auc = roc_auc_score([1]*len(pos_test) + [0]*len(neg_candidates), scores)
ranks = rank_pos_in_candidates(scores, pos_test)
mrr = mean(1.0 / r for r in ranks)

Common pitfalls

• Exhaustive computation over all node pairs is computationally prohibitive; the protocol mandates using only two-hop neighbors as negative candidates for link prediction.
• Results must be averaged over five independent random splits of the 80/20 train/test partition to account for variance.
• Algorithms must explicitly handle heterogeneous edge types; collapsing the graph into a homogeneous structure invalidates the benchmark's evaluation of multi-type preservation.

Evidence (verbatim from paper)

For node classification, we then train a separate linear Support Vector Machine (LinearSVC) based on the learned embeddings on 80% of the labeled nodes and predict on the remaining 20%. We repeat the process for five times and compute the average scores regarding macro-F1 (across all labels) and micro-F1 (across all nodes). For link prediction, we use the Hadamard function to construct feature vectors for node pairs, train a two-class LinearSVC on the 80% training links and evaluate towards the 20% held out links. We also repeat the process for five times and compute the two metrics of AUC (area under the ROC curve) and MRR (mean reciprocal rank).

Citation

@misc{yang2020hne,
  title={Heterogeneous Network Representation Learning: A Unified Framework with Survey and Benchmark},
  author={Carl Yang et al. (2020)},
  year={2020},
  note={arXiv:2004.00216}
}

• arXiv: 2004.00216