Persian Instruction Following Eval

persian-instruction-following-eval

Empowering Persian LLMs for Instruction Following: A Novel Dataset and Training Approach — Mokhtarabadi et al. (2024) (arXiv:2407.11186, 2024)

What this evaluates

Evaluates the instruction-following capability of Persian large language models across multiple NLP tasks, including paraphrasing, sentiment analysis, and textual entailment.

Datasets

• parsinlu queryparaphrasing — total ?; splits: test (-1)
• Digikala SentimentAnalysis — total ?; splits: test (-1)
• FarsTail — total ?; splits: test (-1)
• ParsinluEntailment — total ?; splits: test (-1)

Metrics

• ROUGE-L F1 (primary) — range: percent
  • Computes the F1 score for the longest common subsequence (LCS) of words between the predicted and reference text. F1 is the harmonic mean of precision (LCS length / predicted length) and recall (LCS length / reference length).

Input / output format

Input: Instruction prompt with task-specific context or query (exact format not specified in the provided section).

Output: Generated text response to the instruction (exact format not specified in the provided section).

Scoring recipe

def compute_rouge_l_f1(predictions, references):
    f1_scores = []
    for p, r in zip(predictions, references):
        lcs = longest_common_subsequence(p, r)
        prec = len(lcs) / len(p) if len(p) > 0 else 0
        rec = len(lcs) / len(r) if len(r) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores) * 100

Common pitfalls

• ROUGE-L is a surface-form metric that may poorly correlate with semantic correctness, especially for entailment tasks.
• The evaluation table reports ROUGE-L F1, but some baseline models might report precision or recall separately, causing direct comparison issues if not standardized.
• Held-in versus held-out split definitions are not explicitly detailed in the provided text, making reproducibility of generalization claims difficult.

Evidence (verbatim from paper)

These models are evaluated using ROUGE-L F1 scores. As illustrated in Table[2], Co-CoLA performs comparably well to the Aya model, despite having fewer parameters and being trained on less instruction data and significantly outperforms all other models, indicating the effectiveness of Co-CoLA.

Citation

@misc{mokhtarabadi2024farsinstruct,
  title={Empowering Persian LLMs for Instruction Following: A Novel Dataset and Training Approach},
  author={Mokhtarabadi et al. (2024)},
  year={2024},
  note={arXiv:2407.11186}
}

• arXiv: 2407.11186