Pmstack Run Eval

Execute an evaluation YAML produced by pmstack-eval against a real target system, running n_trials per task and reporting both pass@k and pass^k. Implements the running side of Anthropic's eval framework. HARD-STOPS with educative errors if no target is configured, no tasks are present, or required env vars are missing — never simulates results. Use when the user asks to "run", "execute", or "score" an eval, or wants real metrics from an existing eval design. Requires a runtime that can execute Python and the `claude` CLI (Claude Code, sandboxed code-exec on web, or local terminal).

RyanAlberts Updated 9 repo stars
The funnel of an AI experience for a dental booking assistant. 100 reviewed conversations move left to right through five stages: understand the request, hand o

File contents

RyanAlberts/pmstack commit cc63565d77

Frequently asked questions

npx get-skill add ryanalberts/pmstack-run-eval