Hrt Multi Scale Eval

hrt-multi-scale-eval

Hierarchical Resolution Transformers: A Wavelet-Inspired Architecture for Multi-Scale Language Understanding — Ayan Sar et al. (2025) (arXiv:2509.20581, 2025)

What this evaluates

Evaluates a wavelet-inspired multi-resolution transformer across five linguistic granularities, from character morphology to discourse reasoning. It probes the model's capacity for hierarchical composition, long-range dependency modeling up to 16K tokens, and computational efficiency compared to standard transformers.

Datasets

• WikiMorpho — total 2100000; splits: unspecified (-1)
• IMDB-BYTE — total 50000; splits: unspecified (-1)
• WordNet Hypernymy (WN-Hyper) — total 150000; splits: unspecified (-1)
• SentEval Word Similarity Suite — total 12000; splits: unspecified (-1)
• GLUE Benchmark — total 850000; splits: unspecified (-1)
• SuperGLUE — total ?; splits: unspecified (-1)
• Long Range Arena (LRA) — total ?; splits: unspecified (-1)
• WikiText-103 — total 103000000; splits: unspecified (-1)
• DiscoEval Benchmark — total ?; splits: unspecified (-1)
• NarrativeQA — total 46000; splits: unspecified (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted labels divided by the total number of instances. Used for classification tasks across most benchmarks.
• F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall, typically computed with macro or weighted averaging depending on the task's class distribution.
• Perplexity (PPL) (primary) — range: [1, inf)
  • Exponential of the average negative log-likelihood of the ground truth tokens: exp(-1/N * sum(log p(x_i))). Used for generative language modeling.
• Normalized Efficiency Score (NES) (primary) — range: [0, 1]
  • Accuracy divided by a unit of computational cost (e.g., memory footprint or training time per epoch). Designed to capture the efficiency-performance trade-off.
• Pearson/Spearman Correlation — range: [-1, 1]
  • Linear (Pearson) or rank-based (Spearman) correlation coefficient between predicted similarity scores and reference human judgments.
• Exact Match (EM) — range: [0, 1]
  • Binary score (1 or 0) indicating whether the model's generated answer exactly matches the reference answer string.

Input / output format

Input: Character-level byte sequences, tokenized words/pairs, sentences, or long documents up to 16,384 tokens, depending on the benchmark.

Output: Class labels, similarity scores, token probabilities, or exact text spans for QA.

Scoring recipe

def compute_metrics(predictions, golds, logits=None, cost_units=None):
    acc = sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    f1 = f1_score(golds, predictions, average='weighted')
    ppl = math.exp(-sum(math.log(p[g]) for p, g in zip(logits, golds)) / len(golds)) if logits else None
    nes = acc / cost_units if cost_units else None
    return {'accuracy': acc, 'f1': f1, 'perplexity': ppl, 'nes': nes}

Common pitfalls

• Failing to distinguish between byte-level (IMDB-BYTE) and token-level inputs, which requires different preprocessing pipelines and affects character-module evaluation.
• Not adhering to benchmark-specific max sequence lengths (4,096 for GLUE/SuperGLUE vs. 16,384 for LRA), leading to unfair truncation or padding.
• Calculating NES without normalizing computational cost consistently across hardware configurations or reporting the specific cost unit used.

Evidence (verbatim from paper)

To evaluate rigorously the performance of the proposed Hierarchical Resolution Transformer (HRT), we apply a rich set of evaluation metrics that depend on the nature of each benchmark task. In the case of natural language understanding tasks like GLUE or SuperGLUE, we present natural language understanding ones, such as the standard task-specific measures of accuracy, F1-score, and Matthews Correlation Coefficient (MCC), so that they are comparable to established baselines. In the case of generative and language modeling (e.g., WikiText-103), we use perplexity (PPL) as our main measure of model performance, which estimates the model’s predictive control of sequential tokens. In the long-context reasoning and efficiency-oriented benchmarks or the Long Range Arena (LRA), we quantify the classification accuracy, as well as computational measures such as memory footprint, training time per epoch, and inference latency. In order to capture the efficiency-performance trade-off, we also introduce a normalized efficiency score (NES), which is accuracy/unit of computational cost and hence gives a unitary consideration of the accuracy and resource usage.

Citation

@misc{sar2025hierarchical,
  title={Hierarchical Resolution Transformers: A Wavelet-Inspired Architecture for Multi-Scale Language Understanding},
  author={Ayan Sar et al. (2025)},
  year={2025},
  note={arXiv:2509.20581}
}

• arXiv: 2509.20581