graphrec-eval
Graph Neural Networks for Social Recommendation — Fan et al. (2019) (arXiv:1902.07243, 2019)
What this evaluates
Evaluates the predictive accuracy of social recommendation models by forecasting user-item ratings. It jointly leverages user-item interaction graphs and user-user social graphs to learn co-embeddings, testing the model's ability to integrate heterogeneous social tie strengths and opinion signals into rating prediction.
Datasets
Metrics
MAE — range: other
- Mean Absolute Error: (1/N) Σ |r_ui - r̂_ui|. Lower values indicate better predictive accuracy.
RMSE (primary) — range: other
- Root Mean Square Error: √((1/N) Σ (r_ui - r̂_ui)²). Lower values indicate better predictive accuracy.
Input / output format
Input: User ID, Item ID, social graph adjacency, and opinion ratings.
Output: Predicted continuous rating score.
Scoring recipe
def compute_metrics(preds, gold):
n = len(gold)
mae = sum(abs(p - g) for p, g in zip(preds, gold)) / n
rmse = (sum((p - g)**2 for p, g in zip(preds, gold)) / n) ** 0.5
return {'MAE': mae, 'RMSE': rmse}
Common pitfalls
- Metrics are error-based (lower is better), not accuracy-based.
- Evaluation splits vary by training ratio (60% or 80%), which changes the test set size and may affect comparability.
- Some baselines (e.g., GCMC+SN) preprocess social graphs with node2vec before feeding to the model, which differs from using raw adjacency matrices.
Evidence (verbatim from paper)
In order to evaluate the quality of the recommendation algorithms, two popular metrics are adopted to evaluate the predictive accuracy, namely Mean Absolute Error (MAE) and Root Mean Square Error (RMSE). Smaller values of MAE and RMSE indicate better predictive accuracy. Note that small improvement in RMSE or MAE terms can have a significant impact on the quality of the top-few recommendations. For each dataset, we used x% as a training set to learning parameters, (1-x%)/2 as a validation set to tune hyper-parameters, and (1-x%)/2 as a testing set for the final performance comparison, where x was varied as {80%,60%}.
Citation
@misc{fan2019graphrec,
title={Graph Neural Networks for Social Recommendation},
author={Fan et al. (2019)},
year={2019},
note={arXiv:1902.07243}
}
1---2name: graphrec-eval3description: Evaluates the predictive accuracy of social recommendation models by forecasting user-item ratings. It jointly leverages user-item interaction graphs and user-user social graphs to learn co-embeddings, testing the model's ability to integrate heterogeneous social tie strengths and opinion signals into rating prediction. Use when the user wants to benchmark on Ciao, Epinions, or asks about evaluating this task. Reports RMSE.4---56# graphrec-eval78> Graph Neural Networks for Social Recommendation — Fan et al. (2019) (arXiv:1902.07243, 2019)910## What this evaluates1112Evaluates the predictive accuracy of social recommendation models by forecasting user-item ratings. It jointly leverages user-item interaction graphs and user-user social graphs to learn co-embeddings, testing the model's ability to integrate heterogeneous social tie strengths and opinion signals into rating prediction.1314## Datasets1516- **Ciao** — total 283319; splits: train (-1), val (-1), test (-1); repo https://github.com/wenqifan03/GraphRec-WWW1917- **Epinions** — total 764352; splits: train (-1), val (-1), test (-1); repo https://github.com/wenqifan03/GraphRec-WWW191819## Metrics2021- `MAE` — range: other22 - Mean Absolute Error: (1/N) Σ |r_ui - r̂_ui|. Lower values indicate better predictive accuracy.23- `RMSE` **(primary)** — range: other24 - Root Mean Square Error: √((1/N) Σ (r_ui - r̂_ui)²). Lower values indicate better predictive accuracy.2526## Input / output format2728**Input**: User ID, Item ID, social graph adjacency, and opinion ratings.2930**Output**: Predicted continuous rating score.3132## Scoring recipe3334```python35def compute_metrics(preds, gold):36 n = len(gold)37 mae = sum(abs(p - g) for p, g in zip(preds, gold)) / n38 rmse = (sum((p - g)**2 for p, g in zip(preds, gold)) / n) ** 0.539 return {'MAE': mae, 'RMSE': rmse}40```4142## Common pitfalls4344- Metrics are error-based (lower is better), not accuracy-based.45- Evaluation splits vary by training ratio (60% or 80%), which changes the test set size and may affect comparability.46- Some baselines (e.g., GCMC+SN) preprocess social graphs with node2vec before feeding to the model, which differs from using raw adjacency matrices.4748## Evidence (verbatim from paper)4950> In order to evaluate the quality of the recommendation algorithms, two popular metrics are adopted to evaluate the predictive accuracy, namely Mean Absolute Error (MAE) and Root Mean Square Error (RMSE). Smaller values of MAE and RMSE indicate better predictive accuracy. Note that small improvement in RMSE or MAE terms can have a significant impact on the quality of the top-few recommendations. For each dataset, we used x% as a training set to learning parameters, (1-x%)/2 as a validation set to tune hyper-parameters, and (1-x%)/2 as a testing set for the final performance comparison, where x was varied as {80%,60%}.5152## Citation5354```bibtex55@misc{fan2019graphrec,56 title={Graph Neural Networks for Social Recommendation},57 author={Fan et al. (2019)},58 year={2019},59 note={arXiv:1902.07243}60}61```6263- arXiv: 1902.07243
qhjqhj00