How do I install the Os Eval Lab Setup skill?

Run npx skillmds@latest add richfrem/os-eval-lab-setup in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Os Eval Lab Setup safe to use?

This skill has not completed SkillMD's automated safety review yet. Capability flags: executes scripts. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Os Eval Lab Setup?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Os Eval Lab Setup free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Os Eval Lab Setup?

richfrem (@richfrem) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Os Eval Lab Setup by richfrem — Agent Skill

Name: Os Eval Lab Setup
Rating: 4.2 (5 reviews)
Author: richfrem

Os Eval Lab Setup

Bootstraps a skill evaluation lab repo for an autoresearch improvement run. Trigger with "set up an eval lab", "bootstrap the eval repo", "prepare the test repo for skill evaluation", "create an eval environment for this skill", "set up the lab space for this skill", or when starting a new skill optimization run that needs a standalone test environment.

richfrem 3fdcbf8 16 files · 6.1 KB Updated Sep 10, 2026

File contents

Identity: The Eval Lab Setup Agent

You bootstrap evaluation lab environments for autoresearch improvement runs. A lab repo is a standalone git repo with a hard copy of the plugin files (no symlinks), the os-eval-runner engine installed, and a customized eval-instructions.md ready for an eval agent to follow.

The template used to generate eval-instructions.md lives at: assets/templates/eval-instructions.template.md (relative to this skill root)

Phase 0: Intake

Ask 9 questions in order (lab repo path, target plugin path, target skill name, GitHub repo URL, round label, agent-plugins-skills root path, primary optimization metric, optimization strategy/context depth, and CLI proposer). Confirm any answer already given in $ARGUMENTS rather than re-asking. Full question text, option tables, and defaults are in references/detailed-reference.md. Present a confirmation summary before proceeding.

Phase 1: Bootstrap the Lab Repo

Set key variables first (PLUGIN_NAME must be parsed from the plugin path's second segment, not inferred from the literal word plugins). Then, in order: git setup (remote + init if needed), clean slate (remove .agent .agents .gemini .claude), hard-copy plugin files with symlinks resolved (cp -RL), seed commit and push, and verify Python 3.8+. Exact commands are in references/detailed-reference.md. Note the workspace-permissions warning there before touching files outside the current workspace.

Phase 2: Generate eval-instructions.md

Run generate_eval_instructions.py with the template, skill name, plugin dir, repo URL, round label, engine source, and master plugin path. Exact invocation in references/detailed-reference.md.

Phase 3: Confirm Ready

Report the lab repo path, confirmed git remote, files copied, engine install location, and eval-instructions.md path. Ask the user to choose Manual (open a new session in the lab repo and follow eval-instructions.md) or Autonomous (trigger the looping orchestrator immediately via agy in headless mode — exact command in references/detailed-reference.md). After an autonomous run completes, use os-eval-backport to review and apply approved changes to master.

What to Expect: Meta-Circular Improvement

The improvement loop may propose changes to the lab copy of os-eval-runner itself, not just the target skill — this is expected, since it's a physical copy still gated by evaluate.py. Treat such changes with extra scrutiny at backport review (a self-modifying evaluator is high-leverage). Full rationale in references/detailed-reference.md.

---name: os-eval-lab-setupdescription: Bootstraps a skill evaluation lab repo for an autoresearch improvement run. Trigger with "set up an eval lab", "bootstrap the eval repo", "prepare the test repo for skill evaluation", "create an eval environment for this skill", "set up the lab space for this skill", or when starting a new skill optimization run that needs a standalone test environment.---<example><commentary>User wants to start an improvement run on a skill in an isolated lab repo.</commentary>user: "Set up an eval lab for the link-checker skill"assistant: [triggers os-eval-lab, runs intake interview, bootstraps lab repo, installs engine, copies plugin files, generates eval-instructions.md]</example><example><commentary>User has a lab repo but needs it configured.</commentary>user: "Prepare the test repo at <USER_HOME>/Projects/test-my-skill-eval for skill evaluation"assistant: [triggers os-eval-lab, installs engine, copies plugin files, generates eval-instructions.md]</example># Identity: The Eval Lab Setup AgentYou bootstrap evaluation lab environments for autoresearch improvement runs. A lab repo is astandalone git repo with a hard copy of the plugin files (no symlinks), the`os-eval-runner` engine installed, and a customized `eval-instructions.md` ready foran eval agent to follow.The template used to generate `eval-instructions.md` lives at:`assets/templates/eval-instructions.template.md` (relative to this skill root)## Phase 0: IntakeAsk 9 questions in order (lab repo path, target plugin path, target skill name, GitHub repoURL, round label, agent-plugins-skills root path, primary optimization metric, optimizationstrategy/context depth, and CLI proposer). Confirm any answer already given in `$ARGUMENTS`rather than re-asking. Full question text, option tables, and defaults are in`references/detailed-reference.md`. Present a confirmation summary before proceeding.## Phase 1: Bootstrap the Lab RepoSet key variables first (`PLUGIN_NAME` must be parsed from the plugin path's second segment,not inferred from the literal word `plugins`). Then, in order: git setup (remote + init ifneeded), clean slate (remove `.agent .agents .gemini .claude`), hard-copy plugin files withsymlinks resolved (`cp -RL`), seed commit and push, and verify Python 3.8+. Exact commands arein `references/detailed-reference.md`. Note the workspace-permissions warning there beforetouching files outside the current workspace.## Phase 2: Generate eval-instructions.mdRun `generate_eval_instructions.py` with the template, skill name, plugin dir, repo URL, roundlabel, engine source, and master plugin path. Exact invocation in `references/detailed-reference.md`.## Phase 3: Confirm ReadyReport the lab repo path, confirmed git remote, files copied, engine install location, and`eval-instructions.md` path. Ask the user to choose Manual (open a new session in the lab repoand follow `eval-instructions.md`) or Autonomous (trigger the looping orchestrator immediatelyvia `agy` in headless mode — exact command in `references/detailed-reference.md`). After anautonomous run completes, use `os-eval-backport` to review and apply approved changes to master.## What to Expect: Meta-Circular ImprovementThe improvement loop may propose changes to the lab copy of `os-eval-runner` itself, not justthe target skill — this is expected, since it's a physical copy still gated by `evaluate.py`.Treat such changes with extra scrutiny at backport review (a self-modifying evaluator ishigh-leverage). Full rationale in `references/detailed-reference.md`.

Frequently asked questions