Few Shot Nlg Eval

few-shot-nlg-eval

Scaled Prompt-Tuning for Few-Shot Natural Language Generation — Ting Hu et al. (2023) (arXiv:2309.06759, 2023)

What this evaluates

Evaluates parameter-efficient fine-tuning methods for few-shot natural language generation from structured data (knowledge graphs and semantic representations) to text. It probes the model's ability to adapt to data-scarce regimes while preserving generation fluency and factual alignment with the source structure.

Datasets

• WebNLG 2020 — total ?; splits: train (-1), dev (-1), test (-1)
• E2E — total ?; splits: train (-1), dev (-1), test (-1)
• DART — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• BLEU (primary) — range: [0, 1]
  • Standard n-gram overlap metric between generated text and reference. Higher is better.
• METEOR — range: [0, 1]
  • Metric for Evaluation of Translation with Explicit ORdering. Higher is better.
• TER — range: [0, 1]
  • Translation Edit Ratio: measures the number of edits needed to change the hypothesis to the reference. Lower is better.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence metric. Higher is better.

Input / output format

Input: Linearized structured data (triples or slot-value pairs) with delimiter tokens (, , for KGs; , for MRs).

Output: Natural language sentence.

Scoring recipe

def compute_bleu(hypotheses, references):
    # hypotheses and references are lists of strings
    # Uses standard n-gram precision with brevity penalty
    return nltk.translate.bleu_score.corpus_bleu(references, hypotheses)

Common pitfalls

• TER is lower the better, unlike other metrics which are higher the better.
• Results are averaged over 9 runs (3 seeds × 3 sampling iterations) per few-shot scenario.
• Prefix-Tuning degrades significantly in extremely few-shot regimes on WebNLG and DART.

Evidence (verbatim from paper)

Experiments are conducted on three NLG datasets: WebNLG 2020, E2E, and DART. Since they contain a large number of instances, we sample subsets of instances from each dataset for few-shot tuning. The sampling process is implemented three times for each few-shot scenario. ... For each dataset, we employ the metrics provided in the benchmark for evaluation. WebNLG 2020 applies BLEU, METEOR, chrF++, TER, BERTScore, and BLEURT. E2E uses BLEU,NIST, METEOR, ROUGE-L, and CIDEr. DART employs BLEU, METEOR, TER, BERTScore, MoverScore, and BLEURT. ... Considering the training instability of few-shot learning, we conduct experiments three times with distinct random seeds on each sampled subset. Eventually, we showcase the average evaluation results of nine experiments in each few-shot case below if not specially claimed.

Citation

@misc{hu2023scaled,
  title={Scaled Prompt-Tuning for Few-Shot Natural Language Generation},
  author={Ting Hu et al. (2023)},
  year={2023},
  note={arXiv:2309.06759}
}

• arXiv: 2309.06759