qhjqhj00 ruozhiba-eval
When Emotional Stimuli meet Prompt Designing: An Auto-Prompt Graphical Paradigm — Chengqian Ma et al. (2024) (arXiv:2404.10500, 2024)
What this evaluates
Evaluates large language models' ability to solve complex, logic-heavy Chinese natural language puzzles and perform multi-step reasoning on diverse benchmark tasks. It probes the model's capacity for progressive reasoning, self-verification, and adaptability to structured prompt frameworks without manual tuning.
Datasets
- Ruozhiba — total 240; splits: test (240)
- BIG-Bench-Hard — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Calculated as the ratio of correctly answered instances to the total number of instances. Correctness is determined by an LLM-based judge rather than exact string matching.
Input / output format
Input: Text prompts or questions, often augmented with auto-generated emotional and framework prompts, fed to the LLM.
Output: Natural language response from the LLM, which is subsequently evaluated by an LLM-based judge for correctness.
Scoring recipe
def compute_accuracy(predictions, gold_answers):
correct_count = 0
for pred, gold in zip(predictions, gold_answers):
# Use an LLM judge to assess if the prediction matches the gold answer
if llm_judge_correctness(pred, gold):
correct_count += 1
return (correct_count / len(predictions)) * 100
Common pitfalls
- Relying on exact string matching for evaluation, which the authors explicitly warn against due to formatting inconsistencies and LLM explanations.
- Assuming the LLM judge is infallible; the paper notes that correctness determination relies on the judgment capabilities of the LLM itself, which may vary.
Evidence (verbatim from paper)
As shown in [table 1], Our framework achieved an accuracy of 62.08% on the Ruozhiba dataset. In fact, the Ruozhiba dataset poses a significant challenge to any natural language processing system due to its unique linguistic phenomena.
Citation
@misc{ma2024autopromptgraphical,
title={When Emotional Stimuli meet Prompt Designing: An Auto-Prompt Graphical Paradigm},
author={Chengqian Ma et al. (2024)},
year={2024},
note={arXiv:2404.10500}
}
- arXiv: 2404.10500