Evo Rnn Eval

evo-rnn-eval

Quantifying Long Range Dependence in Language and User Behavior to improve RNNs — Belletti et al. (2019) (arXiv:1905.09414, 2019)

What this evaluates

Evaluates the ability of Evolutive RNNs (EvoRNNs) to model long-range dependencies in sequential data. It compares EvoRNNs against standard RNN baselines on language modeling and sequential recommendation tasks, emphasizing both predictive accuracy and computational efficiency.

Datasets

• LM1B (Billion word dataset) — total ?; splits: test (-1)
• Sequential recommendation dataset — total ?; splits: train (-1), test (-1)

Metrics

• MAP@20 (primary) — range: [0, 1]
  • Mean Average Precision at rank 20. For each user sequence, compute the average precision of the top-20 predicted items relative to the ground-truth next item. Then average this score across all sequences in the test set.

Input / output format

Input: Language modeling: sequence of 128 words. Sequential recommendation: sequence of embedded item IDs representing a user's historical interactions.

Output: Language modeling: probability distribution over the vocabulary for the next 4 words. Sequential recommendation: predicted item ID via a softmax layer.

Scoring recipe

def compute_map_at_20(predictions, gold):
    # predictions: list of lists of top-20 predicted item IDs per query
    # gold: list of ground-truth next item IDs per query
    precisions = []
    for pred_list, true_item in zip(predictions, gold):
        hits = 0
        for rank, item in enumerate(pred_list, 1):
            if item == true_item:
                hits += 1
                precisions.append(hits / rank)
                break
        if hits == 0:
            precisions.append(0.0)
    return sum(precisions) / len(precisions)

Common pitfalls

• The recommendation dataset uses a strict temporal split (last 7 days for training, last 2 days for testing). Random shuffling would cause severe data leakage.
• EvoRNNs dynamically allocate hidden units across sub-sequences, meaning computational cost (add/multiply operations) varies significantly from standard fixed-size RNNs and must be evaluated alongside predictive metrics.
• The language modeling task modifies the standard LM1B benchmark by predicting the last 4 words in a 128-word sequence, which changes the evaluation window compared to standard next-token prediction.

Evidence (verbatim from paper)

We report the Mean-average-precision-at-20 (MAP@20) as the main performance metric. ... The Billion word data set is a standard benchmark for language modeling aimed at predicting the next word in a text. We slightly modify the benchmark to create a long range prediction task involving longer sequences. Sequences of 128 words are considered and the model’s task is now to predict the last 4 words.

Citation

@misc{belletti2019quantifying,
  title={Quantifying Long Range Dependence in Language and User Behavior to improve RNNs},
  author={Belletti et al. (2019)},
  year={2019},
  note={arXiv:1905.09414}
}

• arXiv: 1905.09414