Qoc Eval

qoc-eval

Unearthing Large Scale Domain-Specific Knowledge from Public Corpora — Fei et al. (2024) (arXiv:2401.14624, 2024)

What this evaluates

Evaluates mathematical reasoning, knowledge-oriented language understanding, and challenging reasoning capabilities of LLMs fine-tuned on domain-specific corpora.

Datasets

• MATH — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• AGIEval — total ?; splits: test (-1)
• BIG-Bench Hard — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1] or percent
  • Standard benchmark metrics: exact-match or pass@1 for mathematical tasks (MATH, GSM8K), and multiple-choice accuracy for knowledge/reasoning tasks (MMLU, AGIEval, BIG-Bench Hard).

Input / output format

Input: Natural language questions or prompts from standard benchmarks.

Output: Model-generated text responses.

Scoring recipe

For each dataset instance:
  pred = model.generate(prompt)
  gold = extract_answer(gold_text)
  score = 1 if pred == gold else 0
  metric_value = mean(scores) * 100

Common pitfalls

• Prompting strategy (zero-shot vs few-shot) and decoding parameters are not specified in the provided text.
• Answer extraction rules vary across benchmarks and are deferred to Appendix B.
• Scores in Table 3 are presented as raw numbers without explicit percentage signs, requiring careful interpretation.

Evidence (verbatim from paper)

For the selection of evaluation datasets, we opted for three distinct capabilities to assess both Llama2-QoC and Mistral-QoC. These encompassed mathematical reasoning datasets such as Math, GSM8K, knowledge-oriented language understanding datasets including MMLU, AGIEval, and challenging reasoning tasks BIG-Bench hard.

Citation

@misc{fei2024unearthing,
  title={Unearthing Large Scale Domain-Specific Knowledge from Public Corpora},
  author={Fei et al. (2024)},
  year={2024},
  note={arXiv:2401.14624}
}

• arXiv: 2401.14624