Ptb Wikitext2 Lm Eval

ptb-wikitext2-lm-eval

Pointer Sentinel Mixture Models — Merity et al. (2016) (arXiv:1609.07843, 2016)

What this evaluates

Evaluates next-token prediction accuracy and long-range dependency modeling in language models, with a specific focus on handling rare and out-of-vocabulary words without expanding vocabulary size.

Datasets

• Penn Treebank — total ?; splits: train (-1), val (-1), test (-1)
• WikiText-2 — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• perplexity (primary) — range: [0, inf)
  • Exponential of the negative average log-likelihood of the ground truth tokens: exp(-1/N * sum(log p(x_i))). Lower values indicate better language modeling performance.

Input / output format

Input: A sequence of tokens (words/subwords) from the dataset.

Output: A probability distribution over the fixed vocabulary and pointer targets for the next token.

Scoring recipe

def compute_perplexity(log_probs):
    # log_probs: list of log probabilities for each ground truth token
    avg_log_prob = sum(log_probs) / len(log_probs)
    return math.exp(-avg_log_prob)

Common pitfalls

• Truncated BPTT must use k1=1, k2=L to ensure the pointer component receives full backpropagation windows; standard k1=k2 splits cause uneven gradient flow.
• Test-time Monte Carlo dropout averaging (1000 forward passes) is required for fair comparison with variational LSTM baselines but drastically increases inference cost.
• Vocabulary cutoffs differ across datasets; WikiText-2 contains many OOV words relative to PTB's cutoff, which heavily impacts perplexity scores for models without pointer mechanisms.

Evidence (verbatim from paper)

Table 2 compares the pointer sentinel-LSTM to a variety of other models on the Penn Treebank dataset. The pointer sentinel-LSTM achieves the lowest perplexity, followed by the recent Recurrent Highway Networks (Zilly et al., 2016). The medium pointer sentinel-LSTM model also achieves lower perplexity than the large LSTM models. We halve the learning rate when validation perplexity is worse than the previous iteration, stopping training when validation perplexity fails to improve for three epochs or when 64 epochs are reached.

Citation

@misc{merity2016pointer,
  title={Pointer Sentinel Mixture Models},
  author={Merity et al. (2016)},
  year={2016},
  note={arXiv:1609.07843}
}

• arXiv: 1609.07843