# Skill Regression > Regression testing framework for AgentSkills. Analyzes a target skill, runs script-layer assertions and AI-layer semantic scoring, and outputs a Markdown report. Supports two backends: OpenAI-compatible LLM (default, works anywhere) or OpenClaw cron-based agent trigger (auto-detected if openclaw CLI present). Use when the user asks to "test a skill", "regression test xxx skill", "run skill QA", or "audit my skill against test cases". Requires SR_LLM_API_KEY env var or interactive setup; supports .env files. - Skill: `sahit-sai/skill-regression` (Agent Skill, multi-file: 16 files) - Install (CLI): `npx skillmds@latest add sahit-sai/skill-regression` - Raw SKILL.md: https://api.skillmd.com/api/skills/sahit-sai/skill-regression/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: sahit-sai (https://skillmd.com/u/sahit-sai) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/sahit-sai/skill-regression --- # Skill Regression - **Version**: 1.0.3 - **License**: MIT - **Author**: Evan Song · [github.com/Songhonglei](https://github.com/Songhonglei) - **Repository**: https://github.com/Songhonglei/build-better-skills/tree/main/skills/skill-regression - **Part of**: [`build-better-skills`](https://github.com/Songhonglei/build-better-skills) suite — see [Stages](https://github.com/Songhonglei/build-better-skills#stages) for the lifecycle map. ## What it does Given a target skill directory, this skill: 1. Normalizes test file naming (TEST.md is canonical; legacy Test.md/TESTS.md auto-renamed) 2. Loads or LLM-generates structured test cases 3. Runs script-layer assertions (subprocess + regex/contains/exact) 4. Runs AI-layer semantic scoring (LLM-as-judge, configurable threshold) 5. Outputs a Markdown report (with optional upload hook) Two execution backends, auto-detected: | Backend | When | What it tests | |---------|------|---------------| | `api` (default outside OpenClaw) | LLM API + system prompt | Whether SKILL.md is well-written enough for LLMs to follow correctly | | `openclaw` (default inside OpenClaw) | Cron probe job → real agent → jsonl polling | End-to-end agent + skill integration | ## Quick start ```bash # 1. First time: configure your LLM python3 scripts/setup.py # 2. Run regression on a skill bash scripts/run_regression.sh /path/to/my-skill # 3. Detailed mode (full input/output per case) bash scripts/run_regression.sh /path/to/my-skill --detail # 4. Re-run only failed cases from a previous run bash scripts/run_regression.sh /path/to/my-skill \ --rerun ~/.skill-regression-space/output/my-skill/20260622_001500 ``` ## Configuration (SR_* env vars, .env supported) Required for `api` backend: - `SR_LLM_API_KEY` — OpenAI-compatible API key (e.g. `sk-...`) - `SR_LLM_BASE_URL` — LLM endpoint (default: `https://api.openai.com/v1`) - `SR_LLM_MODEL` — Model name (default: `gpt-4o-mini`) Additional for `openclaw` backend: - `SR_TARGET_AGENT` — Target agent name in OpenClaw (default: `main`) Config resolution priority (high → low): 1. CLI args (e.g. `--backend api`) 2. Process env vars (`export SR_LLM_API_KEY=...`) 3. Project-local `./.env` file (auto-added to `.gitignore`, chmod 600) 4. Global `~/.config/skill-regression/.env` (chmod 600) 5. Missing required keys → interactive onboarding (Style A) if TTY, else hint and exit 2 (Style B) Defaults are NOT applied unless user explicitly accepts them during onboarding. ## Setup commands ```bash python3 scripts/setup.py # Interactive onboarding python3 scripts/setup.py --show # Show current config (secrets masked) python3 scripts/setup.py --reset # Wipe and re-onboard ``` ## Options ``` Usage: bash scripts/run_regression.sh [options] --backend Test backend (default: auto-detect) --space-dir Workspace root (default: ~/.skill-regression-space) --threshold AI semantic score pass threshold (default: 7) --timeout Per-case AI-layer timeout (default: 120) --cases Auto-generated normal cases (default: 3, range 2-20) --error-cases Auto-generated error cases (default: 2, range 1-5) --rerun Re-run only failures from a previous output dir --skip-agent Script layer only (no LLM/agent layer) --no-ai-suggestions Skip LLM-generated improvement suggestions --detail Detailed report (full input/output per case) --setup Run interactive onboarding -h, --help Show this help ``` ## TEST.md format (placed in target skill dir) ````markdown # TEST.md ## Test Cases ### Case 1: basic ```yaml id: case_001 name: Basic functionality type: normal # normal | error | edge trigger: "user message to simulate" script_cmd: "python3 {SKILL_DIR}/scripts/foo.py" expected_output: "Success" # keyword/regex; empty = no script assertion expected_output_mode: contains # contains | regex | exact expected_agent_response: "describe expected agent response semantics" skip_agent: false # true = script layer only ``` ```` Path placeholders in `script_cmd`: | Placeholder | Resolves to | |---|---| | `{SKILL_DIR}` | absolute path of target skill | | `{TESTRES_DIR}` | `~/.skill-regression-space/testres/` (long-term test resources) | | `{WORK_DIR}` | `~/.skill-regression-space/output//` (this run output) | ## Directory structure ``` ~/.skill-regression-space/ ├── testres// Test resources (fixtures, helper scripts) — long-term └── output/// ├── cases.json ├── results/ │ ├── script_results.json │ └── agent_results.json └── report.md ``` ## Report upload (optional hook) Set the env var **SR_REPORT_UPLOAD_HOOK** (pointing to your upload script) to push the generated report somewhere: ```bash # Your upload.sh receives: $1=report-path, $2=skill-name, $3=date # Must print the resulting URL to stdout on success SR_REPORT_UPLOAD_HOOK=~/bin/gist-upload.sh \ bash scripts/run_regression.sh /path/to/my-skill ``` ## Semantic scoring rubric The LLM-as-judge returns 0-10: | Score | Meaning | |-------|---------| | 9-10 | Fully matches expectations, accurate, well-formatted | | 7-8 | Acceptable with minor issues | | 5-6 | Partial match, core deficiencies | | 3-4 | Major deviation, only some content matches | | 0-2 | Wrong or unparseable | ⚠️ The judging LLM and the simulating LLM are typically the same family — treat scores as directional, not absolute. Always inspect detailed mode reports for high-stakes calls. ## Internal modules | File | Purpose | |------|---------| | `scripts/_lib_config.py` | Config loader (.env layers, onboarding) | | `scripts/_lib_llm.py` | OpenAI-compatible LLM client (chat/score/infer) | | `scripts/setup.py` | Interactive onboarding entry | | `scripts/run_regression.sh` | Main pipeline orchestrator | | `scripts/normalize_testfile.py` | TEST.md naming normalization | | `scripts/analyze_skill.py` | Load TEST.md or LLM-generate cases | | `scripts/run_script_tests.py` | Script-layer assertions | | `scripts/run_agent_tests.py` | AI-layer (dual backend) | | `scripts/generate_report.py` | Merge results + Markdown report | | `scripts/merge_results.py` | --rerun result merging | | `references/TEST_template.md` | TEST.md template reference | | `references/report_structure.md` | Report structure reference | ## Dependencies - **Required**: Python 3.8+, `python3`, `bash`, `pip install pyyaml` - **Optional (for `openclaw` backend)**: `openclaw` CLI No external Python packages beyond `pyyaml`. Network calls use stdlib `urllib`. ## Notes & limitations - `api` backend tests "would an LLM follow this SKILL.md correctly?" — useful for skill documentation quality but does NOT test true agent + skill interaction. - `openclaw` backend requires running inside an OpenClaw runtime and creates a namespaced cron job (`skill-regression-probe--`) per target agent. - The normalize_testfile script will rename Test.md/TESTS.md → TEST.md in the target skill dir. Use `--dry-run` to preview. - Auto-generated placeholder cases are marked `skip_agent=true` to avoid false-positive AI-layer scores; users should fill in a real TEST.md ASAP.