Few Shot Bot Eval

few-shot-bot-eval

Few-Shot Bot: Prompt-Based Learning for Dialogue Systems — Andrea Madotto et al. (arXiv:2110.08118, 2021)

What this evaluates

Evaluates prompt-based large language models on 15 diverse dialogue tasks, including response generation, conversational parsing, and skill selection, using a few-shot learning setup without fine-tuning. The protocol tests the model's ability to dynamically select the most appropriate task prompt based on dialogue history and generate accurate responses or parses.

Datasets

• Persona Chat — total ?; splits: test (-1)
• Empathetic Dialogues (ED) — total ?; splits: test (-1)
• Wizard of Wikipedia (WoW) — total ?; splits: test (-1)
• Image Chat (IC) — total ?; splits: test (-1)
• Wizard of Internet (WIT) — total ?; splits: test (-1)
• Controlled Generation (CG-IC) — total ?; splits: test (-1)
• Multi-Session Chat (MSC) — total ?; splits: test (-1)
• DailyDialogue (DD) — total ?; splits: test (-1)
• Stanford Multidomain Dialogue (SMD) — total ?; splits: test (-1)
• DialKG — total ?; splits: test (-1)

Metrics

• perplexity (primary) — range: other
  • Computed over the prompt and candidate response to dynamically select the most appropriate skill for a given dialogue history. Lower perplexity indicates a better prompt-skill match. (Note: Specific metrics for response generation and conversational parsing are not detailed in the provided text.)

Input / output format

Input: Dialogue history/context paired with a few-shot prompt template containing task-specific examples for each candidate skill.

Output: Task-specific output: natural language response, parsed dialogue act/intent, or selected skill identifier.

Scoring recipe

# Pseudo-code for skill selection and task evaluation
# Note: Generation/parsing metrics are not specified in the provided text.

def evaluate_skill_selection(dialogue_history, candidate_skills, model):
    best_skill = None
    min_ppl = float('inf')
    for skill in candidate_skills:
        prompt = build_prompt(dialogue_history, skill, few_shot_examples)
        ppl = model.compute_perplexity(prompt)
        if ppl < min_ppl:
            min_ppl = ppl
            best_skill = skill
    return best_skill, min_ppl

def evaluate_generation_or_parsing(predictions, gold):
    # Task-specific metrics (e.g., BLEU, ROUGE, Accuracy) not detailed in provided text
    # Placeholder for standard evaluation
    return compute_task_metric(predictions, gold)

Common pitfalls

• Prompt ordering and example selection significantly impact few-shot performance and can introduce bias.
• Perplexity-based skill selection may not align perfectly with human judgment or downstream task accuracy.
• Zero-shot/few-shot LLMs often show high variance across random seeds, temperature settings, or prompt formatting.

Evidence (verbatim from paper)

Prompt-based few-shot learning enables large language models to achieve performance comparable to fully trained state-of-the-art dialogue models across 15 diverse tasks—including knowledge-grounded response generation, dialogue parsing, and stylistic control—without any fine-tuning. A novel perplexity-based few-shot skill selector dynamically chooses the most appropriate prompt for a given dialogue history, enabling an end-to-end, zero-training chatbot (Few-Shot Bot) that autonomously selects skills, retrieves external knowledge, and generates human-like responses using only a few examples per task.

Citation

@misc{madotto2021fewshotbot,
  title={Few-Shot Bot: Prompt-Based Learning for Dialogue Systems},
  author={Andrea Madotto et al.},
  year={2021},
  note={arXiv:2110.08118}
}

• arXiv: 2110.08118