Hugging Face Evaluation

Hugging Face Evaluation Skill

All paths are relative to this SKILL.md's directory. cd here or use full paths before running scripts.

PR Safety — Check Before Creating

Before using --create-pr, check for existing open PRs to avoid duplicating work for maintainers:

uv run scripts/evaluation_manager.py get-prs --repo-id "username/model-name"

If open PRs exist:

1. Warn the user and show existing PR URLs
2. Only proceed if the user explicitly confirms creating another PR

Core Workflow

Use --help for the latest workflow guidance:

uv run scripts/evaluation_manager.py --help

Standard flow:

1. get-prs → check for existing open PRs
2. inspect-tables → find table numbers/columns
3. extract-readme --table N → prints YAML (preview by default)
4. Add --apply (push) or --create-pr to write changes

Prerequisites

• Use uv run (PEP 723 header auto-installs deps)
• Set HF_TOKEN env var with write-access token
• For Artificial Analysis: set AA_API_KEY env var
• .env is loaded automatically if python-dotenv is installed

Commands Reference

Inspect tables (start here):

uv run scripts/evaluation_manager.py inspect-tables --repo-id "username/model-name"

Extract from README:

uv run scripts/evaluation_manager.py extract-readme \
  --repo-id "username/model-name" \
  --table N \
  [--model-column-index N] \
  [--model-name-override "Exact Column Header"] \
  [--task-type "text-generation"] \
  [--dataset-name "Custom Benchmarks"] \
  [--apply | --create-pr]

Import from Artificial Analysis:

AA_API_KEY=... uv run scripts/evaluation_manager.py import-aa \
  --creator-slug "creator-name" \
  --model-name "model-slug" \
  --repo-id "username/model-name" \
  [--create-pr]

View / Validate:

uv run scripts/evaluation_manager.py show --repo-id "username/model-name"
uv run scripts/evaluation_manager.py validate --repo-id "username/model-name"

Run evaluation job (inference providers):

hf jobs uv run scripts/inspect_eval_uv.py \
  --flavor "cpu-basic|t4-small|a10g-small" \
  --secret HF_TOKEN=$HF_TOKEN \
  -- --model "model-id" --task "task-name"

Run vLLM evaluation (custom models):

# lighteval with vLLM
uv run scripts/lighteval_vllm_uv.py \
  --model meta-llama/Llama-3.2-1B \
  --tasks "leaderboard|mmlu|5"

# inspect-ai with vLLM
uv run scripts/inspect_vllm_uv.py \
  --model meta-llama/Llama-3.2-1B \
  --task mmlu

# Helper script (auto hardware selection)
uv run scripts/run_vllm_eval_job.py \
  --model "model-id" \
  --task "leaderboard|mmlu|5" \
  --framework lighteval

vLLM Evaluation

Requires uv installed and sufficient GPU memory. Verify with nvidia-smi.

Two frameworks available:

• lighteval — HuggingFace's library; task format: suite|task|num_fewshot (e.g. leaderboard|mmlu|5)
• inspect-ai — UK AISI framework; tasks: mmlu, gsm8k, hellaswag, arc_challenge, truthfulqa, winogrande, humaneval

Key flags:

• --backend accelerate or --backend hf for non-vLLM inference
• --use-chat-template for instruction-tuned models
• --trust-remote-code for models with custom code (e.g. Phi-2, Qwen)
• --tensor-parallel-size N for multi-GPU

Hardware sizing: <3B → t4-small, 3-13B → a10g-small, 13-34B → a10g-large, 34B+ → a100-large

Submit via HF Jobs by prefixing with hf jobs uv run and adding --flavor/--secrets.

Model-Index Format

model-index:
  - name: Model Name
    results:
      - task:
          type: text-generation
        dataset:
          name: Benchmark Dataset
          type: benchmark_type
        metrics:
          - name: MMLU
            type: mmlu
            value: 85.2
        source:
          name: Source Name
          url: https://source-url.com

• Use plain text for model name (no markdown formatting)
• URLs belong only in source.url

Best Practices

1. Run get-prs before creating any PR
2. Start with inspect-tables to see table structure
3. Preview YAML output before using --apply or --create-pr
4. Prefer --model-column-index over --model-name-override; if using override, match column header text exactly
5. For multi-table READMEs, specify --table N
6. Use --create-pr when updating models you don't own
7. One model per repo — only add the main model's results

References

• CLI examples and detailed usage — full Method 1-4 walkthroughs, common patterns, model name matching
• Troubleshooting — error messages and solutions