Embbert Q Eval

embbert-q-eval

EmbBERT-Q: Breaking Memory Barriers in Embedded NLP — Bravin et al. (2025) (arXiv:2502.10001, 2025)

What this evaluates

Evaluates tiny language models and baselines on resource-constrained embedded devices by measuring performance across classification and regression tasks under strict memory limits (≤2 MB). It probes the trade-off between model compression, hardware compatibility, and NLP task accuracy.

Datasets

• TinyNLP — total ?; splits: train (-1), val (-1), test (-1)
• GLUE — total ?; splits: train (-1), val (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted class labels out of the total number of instances.
• GLUE Average Score (primary) — range: [0, 1]
  • Task-specific metrics averaged across all GLUE datasets following the standard protocol: SCC for STSB, MCC for CoLA, F1 for QQP/MRPC, and Accuracy for remaining tasks.

Input / output format

Input: Tokenized text sequences (single sentences or sentence pairs) using a custom BPE tokenizer, typically truncated or padded to a fixed maximum length (e.g., 256 or 512 tokens).

Output: Discrete class labels for classification tasks, or continuous floating-point scores for regression tasks.

Scoring recipe

def compute_metrics(predictions, gold_labels, task):
    if task == 'STSB':
        return spearman_corr(gold_labels, predictions)
    elif task == 'CoLA':
        return matthews_corrcoef(gold_labels, predictions)
    elif task in ('QQP', 'MRPC'):
        return f1_score(gold_labels, predictions, average='macro')
    else:
        return accuracy(gold_labels, predictions)

Common pitfalls

• GLUE official test labels are not publicly released; the paper uses the validation set as the test set, which may yield higher scores than official test evaluations.
• Memory footprint calculations explicitly exclude task-specific output layers, meaning reported sizes (e.g., 781 KB) are lower bounds for actual deployment.
• Datasets without official splits are randomly partitioned (90/10 train/test, then 10% of train for val), introducing variance that requires multiple random seeds (5 runs) to mitigate.

Evidence (verbatim from paper)

For the sake of simplicity, in the experimental results reported in Sec. 5, as evaluation metrics we focus on Accuracy for the TinyNLP benchmark, and on the metric used for computing the average Score in each dataset in the GLUE benchmark: SCC for STSB, MCC for CoLA, F1 score for QQP and MRPC, Accuracy for the remaining GLUE tasks.

Citation

@misc{bravin2025embbertq,
  title={EmbBERT-Q: Breaking Memory Barriers in Embedded NLP},
  author={Bravin et al. (2025)},
  year={2025},
  note={arXiv:2502.10001}
}

• arXiv: 2502.10001