Yelp Rating Prediction Eval

yelp-rating-prediction-eval

Yelp Dataset Challenge: Review Rating Prediction — Asghar (2016) (arXiv:1605.05362, 2016)

What this evaluates

Predicts a 1-to-5 star rating for a restaurant review based on its text content. It probes a model's ability to capture sentiment, domain-specific linguistic patterns, and fine-grained textual features for multi-class classification.

Datasets

• Yelp Dataset — total ?; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of reviews where the predicted star rating matches the ground truth rating.
• RMSE — range: other
  • Square root of the average squared difference between predicted and actual star ratings.

Input / output format

Input: Preprocessed text review (lowercased, punctuation and stop words removed).

Output: Predicted star rating as an integer class label from {1, 2, 3, 4, 5}.

Scoring recipe

def compute_metrics(preds, gold):
    n = len(gold)
    accuracy = sum(1 for p, g in zip(preds, gold) if p == g) / n
    rmse = (sum((p - g) ** 2 for p, g in zip(preds, gold)) / n) ** 0.5
    return {'accuracy': accuracy, 'rmse': rmse}

Common pitfalls

• The paper performs 3-fold cross-validation only on the 80% training set for hyperparameter tuning (e.g., SVM C parameter), not for final evaluation on the test set.
• Feature extraction (n-gram dictionaries, TF-IDF, LSI matrices) must be fitted exclusively on the training data to prevent data leakage when scoring the test set.
• Preprocessing steps (lowercasing, stop-word removal, punctuation stripping) are explicitly required before feature extraction and must be applied identically to test reviews.

Evidence (verbatim from paper)

We use 80% of the dataset for training, and 20% for testing. For each of the sixteen prediction systems, we perform 3-fold cross validation on the training set and compute two metrics, Root Mean Squared Error (RMSE) and accuracy, for the training fold as well as the validation fold.

Citation

@misc{asghar2016yelp,
  title={Yelp Dataset Challenge: Review Rating Prediction},
  author={Asghar (2016)},
  year={2016},
  note={arXiv:1605.05362}
}

• arXiv: 1605.05362