How do I install the Genai Prompt Eval skill?

Run npx skillmds@latest add timothywarner-org/genai-prompt-eval in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Genai Prompt Eval safe to use?

This skill has not completed SkillMD's automated safety review yet. Independent scanners report: SkillSpector: PASS, Skill Scanner: PASS. Capability flags: executes scripts. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Genai Prompt Eval?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Genai Prompt Eval free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Genai Prompt Eval?

timothywarner-org (@timothywarner-org) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Genai Prompt Eval by timothywarner-org

Name: Genai Prompt Eval
Rating: 4.2 (5 reviews)
Author: timothywarner-org

Genai Prompt Eval

Score a Python generative-AI app's outputs on groundedness, relevance, coherence, and safety before it ships. Use when building a prompt-eval harness, writing eval cases for a GenAI feature, gating a deploy on quality thresholds, or measuring whether model answers stay grounded and on-topic. Triggers on "evaluate my prompts", "run evals", "groundedness score", "eval cases", "quality gate for the model", "is the answer grounded".

File contents

Score GenAI outputs before shipping

This skill measures whether a generative-AI feature produces answers that are grounded, relevant, coherent, and safe. It runs a set of eval cases through the model, scores each output on those four dimensions, and reports pass or fail against thresholds. Pair it with the azure-ai-deploy skill: evals are Gate 1 of that deploy checklist.

When to reach for this

A GenAI feature is changing and you need a regression signal on answer quality.
A deploy gate requires proof that outputs meet a quality bar.
You want a repeatable eval corpus that reflects real enterprise questions, not toy prompts.

Workflow

1. Read the dimensions

Read resources/references/EVAL-DIMENSIONS.md. It defines groundedness, relevance, coherence, and safety, states what each one measures, and gives a pass signal for each.

2. Build the eval corpus

Start from resources/templates/eval_cases.jsonl. Each line is one case: an input prompt, optional context the answer must stay grounded to, and expected_criteria describing a passing answer. Add cases that mirror the questions real users send.

3. Run the harness

uv run python ${CLAUDE_SKILL_DIR}/resources/scripts/run_eval.py \
  --cases ${CLAUDE_SKILL_DIR}/resources/templates/eval_cases.jsonl \
  --threshold 0.8

The script loads the cases, calls the model for each, scores the output on the four dimensions, prints a per-case and aggregate report, and exits non-zero when the aggregate score falls below the threshold. That non-zero exit fails a CI or deploy step.

4. Read the report and act

Cases below threshold name the failing dimension. Fix the prompt, the retrieval context, or the guardrail, then re-run.
Record the aggregate score as the new baseline so the next run detects regressions.

Conventions

uv manages Python, not pip. Run scripts with uv run.
No hardcoded secrets. The scoring model client reads its endpoint and deployment from env vars.
Realistic cases only. Eval inputs are enterprise scenarios, never placeholder prompts.
Deterministic scoring where possible. Prefer a low temperature on any model-graded dimension so scores are stable across runs.

---name: genai-prompt-evaldescription: Score a Python generative-AI app's outputs on groundedness, relevance, coherence, and safety before it ships. Use when building a prompt-eval harness, writing eval cases for a GenAI feature, gating a deploy on quality thresholds, or measuring whether model answers stay grounded and on-topic. Triggers on "evaluate my prompts", "run evals", "groundedness score", "eval cases", "quality gate for the model", "is the answer grounded".---# Score GenAI outputs before shippingThis skill measures whether a generative-AI feature produces answers that are **grounded**, **relevant**, **coherent**, and **safe**. It runs a set of eval cases through the model, scores each output on those four dimensions, and reports pass or fail against thresholds. Pair it with the `azure-ai-deploy` skill: evals are Gate 1 of that deploy checklist.## When to reach for this- A GenAI feature is changing and you need a regression signal on answer quality.- A deploy gate requires proof that outputs meet a quality bar.- You want a repeatable eval corpus that reflects real enterprise questions, not toy prompts.## Workflow### 1. Read the dimensionsRead `resources/references/EVAL-DIMENSIONS.md`. It defines groundedness, relevance, coherence, and safety, states what each one measures, and gives a pass signal for each.### 2. Build the eval corpusStart from `resources/templates/eval_cases.jsonl`. Each line is one case: an `input` prompt, optional `context` the answer must stay grounded to, and `expected_criteria` describing a passing answer. Add cases that mirror the questions real users send.### 3. Run the harness```bashuv run python ${CLAUDE_SKILL_DIR}/resources/scripts/run_eval.py \ --cases ${CLAUDE_SKILL_DIR}/resources/templates/eval_cases.jsonl \ --threshold 0.8```The script loads the cases, calls the model for each, scores the output on the four dimensions, prints a per-case and aggregate report, and exits non-zero when the aggregate score falls below the threshold. That non-zero exit fails a CI or deploy step.### 4. Read the report and act- Cases below threshold name the failing dimension. Fix the prompt, the retrieval context, or the guardrail, then re-run.- Record the aggregate score as the new baseline so the next run detects regressions.## Conventions- **uv** manages Python, not pip. Run scripts with `uv run`.- **No hardcoded secrets.** The scoring model client reads its endpoint and deployment from env vars.- **Realistic cases only.** Eval inputs are enterprise scenarios, never placeholder prompts.- **Deterministic scoring where possible.** Prefer a low temperature on any model-graded dimension so scores are stable across runs.

Frequently asked questions