Long Sequence Modeling Eval

long-sequence-modeling-eval

What Makes Convolutional Models Great on Long Sequence Modeling? — Yuhong Li et al. (arXiv:2210.09298, 2022)

What this evaluates

Evaluates the capability of sequence modeling architectures to capture long-range dependencies across text, audio, and image modalities, as well as their computational efficiency and compatibility with standard Transformer and CNN backbones.

Datasets

• Long Range Arena (LRA) — total ?; splits: test (-1)
• Speech Commands (SC) — total ?; splits: test (-1)
• WikiText-103 — total ?; splits: test (-1)
• GLUE — total ?; splits: test (-1)
• ImageNet-1k — total ?; splits: train (-1), val (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted class labels over the total number of instances in the evaluation set.
• perplexity — range: [0, inf)
  • Exponential of the average negative log-likelihood of the ground-truth tokens in the sequence.
• top-1 accuracy — range: [0, 1]
  • Fraction of instances where the highest-probability predicted class matches the ground truth label.

Input / output format

Input: Variable-length sequences (text tokens, raw audio waveforms, or image patches) depending on the specific dataset and task.

Output: Class labels for classification tasks; next-token probability distributions for language modeling.

Scoring recipe

def compute_metric(predictions, gold, metric_name):
    if metric_name == 'accuracy':
        return sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    elif metric_name == 'perplexity':
        return math.exp(-sum(log_probs) / len(log_probs))
    elif metric_name == 'top-1 accuracy':
        return sum(p.argmax(axis=1) == g for p, g in zip(predictions, gold)) / len(gold)

Common pitfalls

• LRA tasks require models to process sequences up to 16K tokens without truncation, testing true long-range dependency capture rather than local context.
• Speech Commands raw waveform classification lacks strong non-SSM baselines, making comparisons against MFCC-based methods potentially uneven.
• GLUE evaluation in this work excludes tasks with fewer than 5K training samples to avoid fine-tuning instability, which may skew average scores.

Evidence (verbatim from paper)

SGConv achieves a 1% improvement in average accuracy upon well-tuned S4 variants introduced in Gu et al. (2022b).

Citation

@misc{li2022whatmakesconvolutional,
  title={What Makes Convolutional Models Great on Long Sequence Modeling?},
  author={Yuhong Li et al.},
  year={2022},
  note={arXiv:2210.09298}
}

• arXiv: 2210.09298