# Hne Benchmark Eval

> This benchmark evaluates the quality and robustness of heterogeneous network embedding (HNE) algorithms across diverse real-world graphs. It probes how well learned representations preserve multi-type structural and attribute information, measured via downstream node classification and link prediction tasks. Use when the user wants to benchmark on DBLP, Yelp, Freebase, PubMed, or asks about evaluating this task. Reports macro-F1.

- Skill: `qhjqhj00/hne-benchmark-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/hne-benchmark-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/hne-benchmark-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/hne-benchmark-eval

---


# hne-benchmark-eval

> Heterogeneous Network Representation Learning: A Unified Framework with Survey and Benchmark — Carl Yang et al. (2020) (arXiv:2004.00216, 2020)

## What this evaluates

This benchmark evaluates the quality and robustness of heterogeneous network embedding (HNE) algorithms across diverse real-world graphs. It probes how well learned representations preserve multi-type structural and attribute information, measured via downstream node classification and link prediction tasks.

## Datasets

- **DBLP** — total 1989077; splits: train (-1), test (-1); repo https://github.com/yangji9181/HNE
- **Yelp** — total 82465; splits: train (-1), test (-1); repo https://github.com/yangji9181/HNE
- **Freebase** — total 12164758; splits: train (-1), test (-1); repo https://github.com/yangji9181/HNE
- **PubMed** — total 63109; splits: train (-1), test (-1); repo https://github.com/yangji9181/HNE

## Metrics

- `macro-F1` **(primary)** — range: [0, 1]
  - F1 score computed per label and averaged across all labels.
- `micro-F1` — range: [0, 1]
  - F1 score computed globally by counting total true positives, false negatives, and false positives across all nodes.
- `AUC` — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, treating link prediction as a binary classification problem.
- `MRR` — range: [0, 1]
  - Mean Reciprocal Rank, treating link prediction as a ranking/retrieval problem over candidate node pairs.

## Input / output format

**Input**: Heterogeneous graph defined by node types, link types, adjacency matrices, and optional node attributes/labels. For evaluation, a fixed 80/20 split of links or labeled nodes is held out.

**Output**: Dense vector embeddings for each node in the graph. For classification: predicted class labels. For link prediction: binary scores or ranked lists of candidate node pairs.

## Scoring recipe

```python
# Node Classification
train_mask, test_mask = split_labels(0.8)
clf = LinearSVC().fit(embeddings[train_mask], labels[train_mask])
preds = clf.predict(embeddings[test_mask])
macro_f1 = f1_score(labels[test_mask], preds, average='macro')
micro_f1 = f1_score(labels[test_mask], preds, average='micro')

# Link Prediction
pos_train, pos_test = split_links(0.8)
neg_candidates = get_2hop_neighbors(pos_test)
pair_feats = hadamard(pos_train + neg_candidates)
link_clf = LinearSVC().fit(pair_feats, [1]*len(pos_train) + [0]*len(neg_candidates))
scores = link_clf.predict_proba(pair_feats)[:, 1]
auc = roc_auc_score([1]*len(pos_test) + [0]*len(neg_candidates), scores)
ranks = rank_pos_in_candidates(scores, pos_test)
mrr = mean(1.0 / r for r in ranks)
```

## Common pitfalls

- Exhaustive computation over all node pairs is computationally prohibitive; the protocol mandates using only two-hop neighbors as negative candidates for link prediction.
- Results must be averaged over five independent random splits of the 80/20 train/test partition to account for variance.
- Algorithms must explicitly handle heterogeneous edge types; collapsing the graph into a homogeneous structure invalidates the benchmark's evaluation of multi-type preservation.

## Evidence (verbatim from paper)

> For node classification, we then train a separate linear Support Vector Machine (LinearSVC) based on the learned embeddings on 80% of the labeled nodes and predict on the remaining 20%. We repeat the process for five times and compute the average scores regarding macro-F1 (across all labels) and micro-F1 (across all nodes). For link prediction, we use the Hadamard function to construct feature vectors for node pairs, train a two-class LinearSVC on the 80% training links and evaluate towards the 20% held out links. We also repeat the process for five times and compute the two metrics of AUC (area under the ROC curve) and MRR (mean reciprocal rank).

## Citation

```bibtex
@misc{yang2020hne,
  title={Heterogeneous Network Representation Learning: A Unified Framework with Survey and Benchmark},
  author={Carl Yang et al. (2020)},
  year={2020},
  note={arXiv:2004.00216}
}
```

- arXiv: 2004.00216