mkurman |
| leaderboard | MMLU, ARC, HellaSwag, TruthfulQA, Winogrande, GSM8K | Open LLM Leaderboard suite |
| mmlu | 57 subjects (STEM, humanities, social sciences) | World knowledge + reasoning |
| gsm8k | Grade-school math word problems | Mathematical reasoning |
| hellaswag | Commonsense NLI | Commonsense reasoning |
| arc_challenge | Science exam questions | Scientific reasoning |
| truthfulqa | Adversarial questions | Truthfulness/hallucination |
| humaneval | Python code generation | Coding ability |
| bigbench | 200+ BIG-Bench tasks | Broad capability assessment |
| ifeval | Instruction-following | Instruction adherence |
Run leaderboard suite:
lm-eval run \
--model hf \
--model_args pretrained=your-model \
--tasks leaderboard \
--device cuda:0 \
--batch_size auto
4. Python API
from lm_eval import simple_evaluate
results = simple_evaluate(
model="hf",
model_args={"pretrained": "meta-llama/Llama-3.2-1B"},
tasks=["mmlu", "hellaswag", "gsm8k"],
device="cuda:0",
batch_size="auto",
limit=100, # Optional: limit samples per task
)
# Access results
for task, metrics in results["results"].items():
print(f"{task}: {metrics}")
# Formatted table
print(results["configs"])
print(results["samples"]) # Per-sample outputs
5. Evaluating LoRA / PEFT Adapters
lm-eval run \
--model hf \
--model_args pretrained=meta-llama/Llama-3.2-1B,peft=/path/to/lora_adapter \
--tasks mmlu \
--device cuda:0
In Python:
from lm_eval import simple_evaluate
results = simple_evaluate(
model="hf",
model_args={
"pretrained": "meta-llama/Llama-3.2-1B",
"peft": "/path/to/lora_adapter",
},
tasks=["mmlu"],
)
6. Multi-GPU Evaluation
Data-parallel (model fits on single GPU):
accelerate launch -m lm_eval \
--model hf \
--model_args pretrained=model-name \
--tasks lambada_openai,arc_easy \
--batch_size 16
Model-parallel (model too large for one GPU):
lm-eval run \
--model hf \
--model_args pretrained=model-name,parallelize=True \
--tasks mmlu \
--batch_size 8
Both (data + model parallel):
accelerate launch --multi_gpu --num_processes 4 \
-m lm_eval \
--model hf \
--model_args pretrained=model-name,parallelize=True \
--tasks mmlu \
--batch_size 8
7. API Model Evaluation
OpenAI-compatible API:
export OPENAI_API_KEY=your-key
lm-eval run \
--model openai-completions \
--model_args model=gpt-4o,base_url=https://api.openai.com/v1/completions \
--tasks mmlu \
--batch_size 32
Local server (vLLM served):
lm-eval run \
--model local-completions \
--model_args model=local-model,base_url=http://localhost:8000/v1/completions \
--tasks mmlu
8. Custom Tasks (YAML Config)
Create a custom task at lm_eval/tasks/my_task/my_task.yaml:
task: my_custom_task
dataset_path: my-dataset
dataset_name: default
output_type: multiple_choice
training_split: train
validation_split: validation
doc_to_text: "Question: {{question}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:"
doc_to_target: "{{answer}}"
doc_to_choice: "{{choices}}"
metric_list:
- metric: acc
Run it:
lm-eval run --model hf --model_args pretrained=model-name --tasks my_custom_task
9. Few-Shot Configuration
lm-eval run \
--model hf \
--model_args pretrained=model-name \
--tasks mmlu \
--num_fewshot 5 \
--fewshot_random_seed 42
10. Logging and Output Formats
# JSON output
lm-eval run --model hf --model_args pretrained=model-name \
--tasks mmlu --output_path results/
# W&B logging
lm-eval run --model hf --model_args pretrained=model-name \
--tasks mmlu --wandb_args project=eval-runs
Key Evaluation Tips
- Use
--batch_size auto— automatic batch size detection maximizes throughput - Always set a
--seedfor reproducibility across evaluation runs - Prefer vLLM backend for models >7B — 5-10x faster than HF
- Use
--limit 100during development to test task setup quickly - mmlu uses 5-shot by default, while most tasks are 0-shot
- Install model backends separately — base package is lightweight by design
- GGUF eval requires explicit tokenizer path — skip this and loading may hang