Negativeprompt Eval

negativeprompt-eval

NegativePrompt: Leveraging Psychology for Large Language Models Enhancement via Negative Emotional Stimuli — Xu Wang et al. (2024) (arXiv:2405.02814, 2024)

What this evaluates

Evaluates LLM instruction-following and reasoning capabilities under zero-shot and few-shot settings by appending psychologically grounded negative emotional stimuli to prompts. It probes task accuracy, complex reasoning on beyond-capability tasks, and the truthfulness and informativeness of generated responses.

Datasets

• Instruction Induction — total 24; splits: test (24)
• BIG-Bench (curated subset) — total 21; splits: test (21)
• TruthfulQA — total 817; splits: test (817); HF truthfulqa/truthful_qa

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard exact-match accuracy averaged across the 24 Instruction Induction tasks.
• normalized preferred metric (primary) — range: percent
  • Normalized score where 100 equals human expert performance and 0 equals random guessing. Computed as (model_acc - random_acc) / (human_acc - random_acc) * 100.
• truthfulness — range: percent
  • Percentage of responses classified as 'true' by the GPT-judge binary classifier.
• informativeness — range: percent
  • Percentage of responses classified as 'informative' by the GPT-info binary classifier.

Input / output format

Input: Original task prompt with negative emotional stimuli appended. For few-shot settings, five randomly selected input-output demonstrations are added after the prompt.

Output: Model-generated text response.

Scoring recipe

def compute_metrics(predictions, golds, random_acc, human_acc):
    # Accuracy (Instruction Induction)
    acc = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(golds)
    # Normalized BIG-Bench metric
    model_acc = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(golds)
    bb_norm = ((model_acc - random_acc) / (human_acc - random_acc)) * 100
    # Truthfulness & Informativeness (TruthfulQA)
    true_count = sum(1 for p in predictions if GPT_judge(p) == 'true')
    info_count = sum(1 for p in predictions if GPT_info(p) == 'informative')
    truth_pct = true_count / len(predictions)
    info_pct = info_count / len(predictions)
    return acc, bb_norm, truth_pct, info_pct

Common pitfalls

• Applying few-shot demonstrations to BIG-Bench tasks (the paper explicitly uses zero-shot only for BIG-Bench).
• Confusing the '+Ours(avg)' and '+Ours(max)' aggregation strategies: avg averages performance across 10 stimuli first, then across tasks; max selects the best stimulus per task before averaging across tasks.
• Using default ChatGPT settings instead of the specified gpt-3.5-turbo with temperature 0.7.

Evidence (verbatim from paper)

For the Instruction Induction tasks, accuracy is the primary evaluation metric. In contrast, for the BIG-Bench tasks, we employ the normalized preferred metric as defined in Srivastava et al. ([2022]). According to this metric, a score of 100 is equated to the performance level of human experts, while a score of 0 aligns with random guessing. It’s critical to note that if an model’s performance on multiple-choice tasks falls below the threshold of random guessing, it may receive a score lower than 0.

Citation

@misc{wang2024negativeprompt,
  title={NegativePrompt: Leveraging Psychology for Large Language Models Enhancement via Negative Emotional Stimuli},
  author={Xu Wang et al. (2024)},
  year={2024},
  note={arXiv:2405.02814}
}

• arXiv: 2405.02814