Tape Eval

tape-eval

TAPE: Assessing Few-shot Russian Language Understanding — Taktasheva et al. (2022) (arXiv:2210.12813, 2022)

What this evaluates

Evaluates few-shot and zero-shot Russian language understanding across six tasks probing logical reasoning, multi-hop inference, commonsense knowledge, and ethical judgment. It also measures model robustness against linguistic adversarial perturbations like typos, deletions, and modality changes.

Datasets

• TAPE — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted labels or multiple-choice answers. Calculated as the number of matches between predictions and gold labels divided by total instances.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall, reported as weighted average for imbalanced classification tasks.
• ASR — range: percent
  • Attack Success Rate, measuring the percentage drop in model performance after applying adversarial perturbations compared to the clean baseline.

Input / output format

Input: Russian text passages or questions, optionally preceded by k-shot demonstrations (k ∈ {1, 4, 8}) formatted as prompt examples.

Output: Predicted class labels for classification/multiple-choice tasks; free-form generated text for QA tasks.

Scoring recipe

# Classification/Multiple-Choice
acc = sum(pred == gold for pred, gold in zip(predictions, golds)) / len(golds)
f1 = f1_score(golds, predictions, average='weighted')
# QA Generation
exact_match = sum(gen.strip() == gold.strip() for gen, gold in zip(generated_preds, golds)) / len(golds)
# Robustness (ASR)
asr = ((p_clean - p_perturbed) / p_clean) * 100

Common pitfalls

• Few-shot demonstrations can cause performance degradation or constant predictions on imbalanced tasks (e.g., models predict the negative label ~97% of the time in zero-shot).
• Standard exact-match QA metrics fail to capture semantically correct but non-literal generated answers, requiring manual semantic analysis.
• Model performance exhibits strong length bias, dropping significantly on longer inputs due to context window limitations.

Evidence (verbatim from paper)

Table 4 shows the ASR scores for each perturbation and k-shot setting averaged over the RuWorldTree and RuOpenBookQA tasks, where the model performance exceeds the random baseline. We observe that the models are more susceptible to simple spelling-based perturbations (BUTTERFINGERS), token deletion (EDA_DELETE) and modality changes (EMOJIFY).

Citation

@misc{taktasheva2022tape,
  title={TAPE: Assessing Few-shot Russian Language Understanding},
  author={Taktasheva et al. (2022)},
  year={2022},
  note={arXiv:2210.12813}
}

• arXiv: 2210.12813