Sentence Representation Eval

sentence-representation-eval

Sentence Bottleneck Autoencoders from Transformer Language Models — Montero et al. (2021) (arXiv:2109.00055, 2021)

What this evaluates

Evaluates the quality of sentence-level representations learned by a transformer-based autoencoder across semantic similarity, single- and multi-sentence classification, and controlled text generation. It probes the model's ability to capture semantic meaning, classify sentiment/acceptability, and reconstruct or modify text via vector arithmetic.

Datasets

• Semantic Textual Similarity (STS) — total ?; splits: test (-1), dev (-1)
• GLUE benchmark — total ?; splits: dev (-1)
• Yelp reviews — total ?; splits: train (-1)

Metrics

• Spearman's rank correlation (primary) — range: [-1, 1]
  • Computes the rank correlation between predicted similarity scores and human-annotated gold scores. Higher values indicate better alignment with human judgments.
• Accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted class labels out of the total number of instances. Reported as median over three random seeds.
• Self-BLEU — range: [0, 1]
  • Measures the n-gram overlap between the generated sentence and the original input sentence, computed using standard BLEU scoring.

Input / output format

Input: Pairs of sentences for similarity tasks; single sentences for classification; single sentences for generation/style transfer.

Output: Similarity scores or ranks; class labels; generated sentences.

Scoring recipe

from scipy.stats import spearmanr

# STS: Spearman correlation
spearman_corr, _ = spearmanr(gold_scores, pred_scores)

# Classification: Accuracy
correct = sum(1 for g, p in zip(gold_labels, pred_labels) if g == p)
acc = correct / len(gold_labels)

# Generation: Self-BLEU & Classifier Accuracy
self_bleu = compute_self_bleu(input_text, generated_text)
gen_acc = sentiment_classifier.predict(generated_text) == target_sentiment

Common pitfalls

• STS evaluation uses Spearman's rank correlation, not Pearson correlation or exact-match accuracy.
• Classification results report median accuracy over three random seeds, not a single run.
• Generation accuracy is measured by an external sentiment classifier, not human judgment or standard BLEU.

Evidence (verbatim from paper)

We report Spearman's rank correlation on the test set and the model sizes are reported in terms of trained parameter size.

Citation

@misc{montero2021sentence,
  title={Sentence Bottleneck Autoencoders from Transformer Language Models},
  author={Montero et al. (2021)},
  year={2021},
  note={arXiv:2109.00055}
}

• arXiv: 2109.00055