Franken Adapter Eval

franken-adapter-eval

Franken-Adapter: Cross-Lingual Adaptation of LLMs by Embedding Surgery — Fan Jiang et al. (arXiv:2502.08037, 2025)

What this evaluates

Evaluates the cross-lingual adaptation and zero-shot/few-shot transfer capabilities of decoder-only LLMs across reading comprehension, topic classification, machine translation, mathematical reasoning, and summarization tasks in Southeast Asian, African, and Indic languages.

Datasets

• BeleBele — total ?; splits: test (-1)
• Sib-200 — total ?; splits: test (-1)
• Flores-200 — total ?; splits: test (-1)
• GSM8K-NTL — total ?; splits: test (-1)
• IndicGenBench — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted labels or options across all instances. Calculated as correct predictions divided by total predictions.
• ChrF++ (primary) — range: percent
  • Character n-gram F-score extended with character n-gram precision and recall, typically with beta=2. Measures character-level overlap between generated and reference translations.
• Token-level F1 — range: [0, 1]
  • Harmonic mean of token-level precision and recall for cross-lingual question answering. Averages F1 scores across all tokens in the prediction and reference.
• ChrF — range: percent
  • Character n-gram F-score used for summarization evaluation. Measures character-level overlap between generated summaries and references.

Input / output format

Input: Text prompt containing task instructions and up to five demonstration examples (few-shot) or just the task instruction (zero-shot), followed by the target language query or source text.

Output: Model-generated text (for generation tasks) or selected option letter/text (for multiple-choice/classification tasks).

Scoring recipe

def compute_metrics(predictions, golds, task_type):
    if task_type in ['classification', 'qa', 'math_reasoning']:
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    elif task_type == 'mt':
        return chrf_score(predictions, golds, beta=2)  # ChrF++
    elif task_type == 'summarization':
        return chrf_score(predictions, golds, beta=2)  # ChrF
    elif task_type == 'qa_f1':
        return f1_score(predictions, golds, average='token')

Common pitfalls

• Generation tasks often show performance degradation compared to classification due to auto-regressive error propagation accumulating over time steps.
• English proficiency regression is minimal but must be explicitly monitored using a separate compiled English benchmark, as cross-lingual gains do not guarantee English preservation.
• Tokenizer fertility reduction improves low-resource performance but requires careful 'Prune-with-Extension' construction to avoid parameter bloat and latency issues.

Evidence (verbatim from paper)

For LLMs after language adaptation, we adopt the five-shot prompting strategy. In contrast, Franken-Adapter is evaluated in a zero-shot setting, given it has been instruction tuned. We also evaluate Franken-Adapter using a compiled English benchmark (Appendix[B]) to examine potential regressions in general English ability. BeleBele (Bandarkar et al., [2024]) is a multiple-choice reading comprehension (MRC) dataset with 122 languages. ... We follow the original paper to use the Accuracy metric and sample five-shot prompts from the English training dataset. ... Flores-200 (Goyal et al., [2022]) is a machine translation dataset with 200 languages. ... evaluate ... using ChrF++ and sample five-shot prompts ... GSM8K-NTL ... use the Accuracy metric and fixed five-shot prompts. IndicGenBench ... zero-shot prompting. The token-level F1 and ChrF scores are reported inXorQA-In andXSum-In, respectively.

Citation

@misc{jiang2025frankenadapter,
  title={Franken-Adapter: Cross-Lingual Adaptation of LLMs by Embedding Surgery},
  author={Fan Jiang et al.},
  year={2025},
  note={arXiv:2502.08037}
}

• arXiv: 2502.08037