Hugging Face Evaluation

Hugging Face Evaluation Skill

⚡ Quick Setup

# Install uv (if needed)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Required: Write-access HF token
export HF_TOKEN="hf_your_token_here"

# Optional: only needed for Artificial Analysis import
export AA_API_KEY="your_aa_api_key"
# Or use a .env file — loaded automatically if python-dotenv is installed

All scripts use PEP 723 inline dependencies — uv run auto-installs on first run.

⚠️ CRITICAL: Check for Existing PRs Before Creating Any New One

uv run scripts/evaluation_manager.py get-prs --repo-id "{your-username}/model-name"

If open PRs exist: do not create a new one. Show the user the existing PR URLs and wait for explicit confirmation before proceeding. This prevents duplicate PRs on repos you don't own.

Method Selection

Method 1: Extract from README

Parses markdown evaluation tables in an existing README and converts them to model-index YAML.

# Step 1: Inspect tables — get table numbers and column hints
uv run scripts/evaluation_manager.py inspect-tables --repo-id "{your-username}/model-name"

# Step 2: Preview YAML (prints to stdout by default — review before applying)
uv run scripts/evaluation_manager.py extract-readme \
  --repo-id "{your-username}/model-name" \
  --table 1 \
  [--model-column-index N]           # preferred: index from inspect-tables output
  [--model-name-override "Exact Header"]  # fallback: exact column header text
  [--task-type "text-generation"]    # sets task.type in model-index
  [--dataset-name "Custom Benchmarks"]

# Step 3a: Push directly (if you own the repo)
uv run scripts/evaluation_manager.py extract-readme \
  --repo-id "{your-username}/model-name" --table 1 --apply

# Step 3b: Open a PR (if you don't own the repo — run get-prs first)
uv run scripts/evaluation_manager.py extract-readme \
  --repo-id "other-username/model-name" --table 1 --create-pr

Column matching notes:

• Prefer --model-column-index (integer from inspect-tables output)
• For --model-name-override, the value must be the exact column header text
• For transposed tables (models as rows), ensure only one row is extracted
• Matching is fuzzy-normalized: "OLMo-3-32B" → tokens {olmo, 3, 32b} — exact token match required

Method 2: Import from Artificial Analysis

Fetches benchmark scores from the Artificial Analysis API and adds them to a model card.

# Preview / dry run
AA_API_KEY="your-key" uv run scripts/evaluation_manager.py import-aa \
  --creator-slug "anthropic" \
  --model-name "claude-sonnet-4" \
  --repo-id "{your-username}/model-name"

# Create a PR (always check get-prs first)
uv run scripts/evaluation_manager.py import-aa \
  --creator-slug "anthropic" \
  --model-name "claude-sonnet-4" \
  --repo-id "{your-username}/model-name" \
  --create-pr

Method 3: Run Evaluation on HF Jobs (Inference Providers)

Submits an eval job to HF infrastructure using inspect-ai. No GPU required for CPU tasks.

# CPU task
HF_TOKEN=$HF_TOKEN \
hf jobs uv run scripts/inspect_eval_uv.py \
  --flavor cpu-basic \
  --secret HF_TOKEN=$HF_TOKEN \
  -- --model "meta-llama/Llama-2-7b-hf" --task "mmlu"

# GPU task (A10G)
HF_TOKEN=$HF_TOKEN \
hf jobs uv run scripts/inspect_eval_uv.py \
  --flavor a10g-small \
  --secret HF_TOKEN=$HF_TOKEN \
  -- --model "meta-llama/Llama-2-7b-hf" --task "gsm8k"

# Or use the Python helper
uv run scripts/run_eval_job.py \
  --model "meta-llama/Llama-2-7b-hf" \
  --task "mmlu" \
  --hardware "t4-small"

Method 4: Run Custom Model Evaluation with vLLM

For any HF model on local GPU or HF Jobs GPU. Requires nvidia-smi and uv.

Before running: verify nvidia-smi shows an available GPU.

Option A: lighteval (HF's evaluation library)

# Local GPU
uv run scripts/lighteval_vllm_uv.py \
  --model meta-llama/Llama-3.2-1B \
  --tasks "leaderboard|mmlu|5"

# Multiple tasks
uv run scripts/lighteval_vllm_uv.py \
  --model meta-llama/Llama-3.2-1B \
  --tasks "leaderboard|mmlu|5,leaderboard|gsm8k|5"

# Instruction-tuned model
uv run scripts/lighteval_vllm_uv.py \
  --model meta-llama/Llama-3.2-1B-Instruct \
  --tasks "leaderboard|mmlu|5" --use-chat-template

# accelerate backend (fallback if vLLM not supported)
uv run scripts/lighteval_vllm_uv.py \
  --model meta-llama/Llama-3.2-1B \
  --tasks "leaderboard|mmlu|5" --backend accelerate

# Via HF Jobs
hf jobs uv run scripts/lighteval_vllm_uv.py \
  --flavor a10g-small \
  --secret HF_TOKEN=$HF_TOKEN \
  -- --model meta-llama/Llama-3.2-1B --tasks "leaderboard|mmlu|5"

lighteval task format: suite|task|num_fewshot — e.g. leaderboard|mmlu|5, lighteval|hellaswag|0, bigbench|abstract_narrative_understanding|0. Full task list: https://github.com/huggingface/lighteval/blob/main/examples/tasks/all_tasks.txt

Option B: inspect-ai (UK AI Safety Institute framework)

# Local GPU
uv run scripts/inspect_vllm_uv.py \
  --model meta-llama/Llama-3.2-1B --task mmlu

# HF Transformers backend (for architectures vLLM doesn't support)
uv run scripts/inspect_vllm_uv.py \
  --model meta-llama/Llama-3.2-1B --task mmlu --backend hf

# Multi-GPU
uv run scripts/inspect_vllm_uv.py \
  --model meta-llama/Llama-3.2-70B --task mmlu --tensor-parallel-size 4

# Via HF Jobs
hf jobs uv run scripts/inspect_vllm_uv.py \
  --flavor a10g-small \
  --secret HF_TOKEN=$HF_TOKEN \
  -- --model meta-llama/Llama-3.2-1B --task mmlu

Available inspect-ai tasks: mmlu, gsm8k, hellaswag, arc_challenge, truthfulqa, winogrande, humaneval

Option C: Helper Script (auto hardware selection)

uv run scripts/run_vllm_eval_job.py \
  --model meta-llama/Llama-3.2-1B \
  --task "leaderboard|mmlu|5" \
  --framework lighteval   # or: inspect

# Explicit hardware + tensor parallelism
uv run scripts/run_vllm_eval_job.py \
  --model meta-llama/Llama-3.2-70B \
  --task mmlu \
  --framework inspect \
  --hardware a100-large \
  --tensor-parallel-size 4

Hardware guide:

model-index Format Reference

model-index:
  - name: Model Name        # plain text — no markdown formatting
    results:
      - task:
          type: text-generation
        dataset:
          name: Benchmark Dataset
          type: benchmark_type
        metrics:
          - name: MMLU
            type: mmlu
            value: 85.2
        source:
          name: Source Name
          url: https://source-url.com   # URLs only in source.url

Common Workflows

Update your own model from its README:

uv run scripts/evaluation_manager.py inspect-tables --repo-id "{your-username}/model"
uv run scripts/evaluation_manager.py extract-readme \
  --repo-id "{your-username}/model" --table 1 --task-type "text-generation"
# Review YAML, then:
uv run scripts/evaluation_manager.py extract-readme \
  --repo-id "{your-username}/model" --table 1 --apply

Submit a PR to someone else's model:

# 1. Check PRs first — abort if any exist
uv run scripts/evaluation_manager.py get-prs --repo-id "other-username/model"

# 2. Only if no open PRs:
uv run scripts/evaluation_manager.py extract-readme \
  --repo-id "other-username/model" --table 1 --create-pr

Import Artificial Analysis benchmarks:

uv run scripts/evaluation_manager.py get-prs --repo-id "{your-username}/model"
AA_API_KEY=... uv run scripts/evaluation_manager.py import-aa \
  --creator-slug "anthropic" --model-name "claude-sonnet-4" \
  --repo-id "{your-username}/model" --create-pr

Validate / inspect current model-index:

uv run scripts/evaluation_manager.py show     --repo-id "{your-username}/model"
uv run scripts/evaluation_manager.py validate --repo-id "{your-username}/model"

Troubleshooting