How do I install the Eval Model skill?

Run npx skillmds@latest add mybigday/eval-model in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Eval Model safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Eval Model?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Eval Model free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Eval Model?

mybigday (@mybigday) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Eval Model by mybigday — Agent Skill

Name: Eval Model
Author: mybigday

Eval Model

Evaluate a trained or downloaded language model with `lm-eval-harness` standard tasks (arc, hellaswag, gsm8k, mmlu, truthfulqa, ifeval, ...). Triggered when the user wants to benchmark, eval, or compare a model — pre- or post-training.

mybigday c9cfb3f 1.4 KB Updated Sep 17, 2026

File contents

eval-model — lm-eval-harness wrapper

Wrapper: scripts/eval_lm.py. Requires pip install 'lm-eval>=0.4.5' (or bash bootstrap.sh --eval).

Quick eval

python scripts/eval_lm.py \
    --model HuggingFaceTB/SmolLM2-360M-Instruct \
    --tasks arc_easy,hellaswag,piqa \
    --num-fewshot 0 \
    --batch-size auto

For local checkpoints, pass the directory (outputs/sft-default) or any Hub repo id. lm-eval handles loading.

Task picks by training goal

Goal	Tasks
Instruct-quality smoke	`arc_easy,hellaswag,piqa,winogrande`
Math reasoning	`gsm8k,minerva_math`
Knowledge	`mmlu`
Truthfulness	`truthfulqa_mc2`
Instruction following	`ifeval`
Code	`humaneval` (needs sandbox)

Reporting

After the run, surface:

Task → score → baseline comparison. Use the published numbers from the paper that motivated the recipe (the research-recipe output should include these).
The model URL on the Hub.
The Trackio / wandb run URL for training, if applicable.

Do not just dump the JSON. Pick the 3-5 numbers the user actually cares about and put them in a small table.

Frequently asked questions