Engineering Manager, Agent Prompts & Evals
When to Use
- Build or scale a prompt + eval engineering function (central or embedded)
- Prioritize golden set, harness, and judge work with PM and risk
- Define release policy for prompt/tool changes (gates, waivers, rollback)
- Hire and level prompt engineers, eval engineers, and tech leads
- Resolve capacity conflicts between new agents and eval debt
- Report eval health (pass rate, slice regressions, judge drift) to leadership
- Partner with
ai-lead-ops on production incidents tied to prompts
When NOT to Use
- Author system prompts, tool schemas, or eval cases →
prompt-engineer-agent-prompts-evals
- General prompt patterns (CoT, few-shot) →
prompt-engineer
- Full RAG/agent application code →
ai-engineer
- Vertical AI product roadmap and GTM launches →
engineering-manager-vertical-ai-products
- Adversarial red-team execution →
ai-redteam
- Org-wide data or analytics management →
data-manager
Related skills
| Need |
Skill |
| IC prompt/eval implementation |
prompt-engineer-agent-prompts-evals |
| Vertical AI EM (broader) |
engineering-manager-vertical-ai-products |
| AI ops and rollouts |
ai-lead-ops |
| Risk tier and policy |
ai-risk-governance |
| Token cost programs |
ai-token-improvement-plan-engineer |
Core Workflows
1. Org design
Central eval platform vs embedded prompt owners; ratios and interfaces.
See references/team_org_prompt_eval.md.
2. Roadmap and prioritization
Eval debt, new agent coverage, judge calibration, CI investment.
See references/roadmap_prioritization.md.
3. Release governance
Prompt semver gates, waivers, coordination with risk and ops.
See references/release_governance.md.
4. Stakeholder partnerships
PM, applied architect, risk, red-team, platform eng.
See references/stakeholder_partnerships.md.
5. Hiring and development
Levels for prompt/eval specialists.
See references/hiring_development.md.
6. Team metrics
Pass rate, coverage, time-to-golden-case, incident linkage.
See references/team_metrics_accountability.md.
Output standards
- Roadmap items name eval slice, prompt surface, and risk tier
- No prompt prod change without documented baseline vs candidate
- Waivers require risk sign-off for Tier 1–2 customer-facing agents
- Escalations include trade-offs (scope, date, headcount)
When to load references
- Org →
references/team_org_prompt_eval.md
- Roadmap →
references/roadmap_prioritization.md
- Release →
references/release_governance.md
- Stakeholders →
references/stakeholder_partnerships.md
- People →
references/hiring_development.md
- KPIs →
references/team_metrics_accountability.md
1---2name: engineering-manager-agent-prompts-evals3description: Guides engineering managers leading teams that own agent prompts, tool schemas, golden eval suites, judge programs, and prompt regression CI—org design, hiring, roadmap with PM/risk, release governance, and team KPIs for eval quality and prompt stability. Use when managing prompt/eval engineers, prioritizing eval backlogs, staffing harness and judge programs, or governing prompt releases—not for writing prompts or harnesses (prompt-engineer-agent-prompts-evals), vertical AI squad management (engineering-manager-vertical-ai-products), or AI platform SRE (ai-lead-ops).4---56# Engineering Manager, Agent Prompts & Evals78## When to Use910- Build or scale a **prompt + eval** engineering function (central or embedded)11- Prioritize **golden set**, harness, and judge work with PM and risk12- Define **release policy** for prompt/tool changes (gates, waivers, rollback)13- **Hire and level** prompt engineers, eval engineers, and tech leads14- Resolve capacity conflicts between **new agents** and **eval debt**15- Report **eval health** (pass rate, slice regressions, judge drift) to leadership16- Partner with `ai-lead-ops` on production incidents tied to prompts1718## When NOT to Use1920- Author system prompts, tool schemas, or eval cases → `prompt-engineer-agent-prompts-evals`21- General prompt patterns (CoT, few-shot) → `prompt-engineer`22- Full RAG/agent application code → `ai-engineer`23- Vertical AI product roadmap and GTM launches → `engineering-manager-vertical-ai-products`24- Adversarial red-team execution → `ai-redteam`25- Org-wide data or analytics management → `data-manager`2627## Related skills2829| Need | Skill |30|---|---|31| IC prompt/eval implementation | `prompt-engineer-agent-prompts-evals` |32| Vertical AI EM (broader) | `engineering-manager-vertical-ai-products` |33| AI ops and rollouts | `ai-lead-ops` |34| Risk tier and policy | `ai-risk-governance` |35| Token cost programs | `ai-token-improvement-plan-engineer` |3637## Core Workflows3839### 1. Org design4041Central eval platform vs embedded prompt owners; ratios and interfaces.4243**See `references/team_org_prompt_eval.md`.**4445### 2. Roadmap and prioritization4647Eval debt, new agent coverage, judge calibration, CI investment.4849**See `references/roadmap_prioritization.md`.**5051### 3. Release governance5253Prompt semver gates, waivers, coordination with risk and ops.5455**See `references/release_governance.md`.**5657### 4. Stakeholder partnerships5859PM, applied architect, risk, red-team, platform eng.6061**See `references/stakeholder_partnerships.md`.**6263### 5. Hiring and development6465Levels for prompt/eval specialists.6667**See `references/hiring_development.md`.**6869### 6. Team metrics7071Pass rate, coverage, time-to-golden-case, incident linkage.7273**See `references/team_metrics_accountability.md`.**7475## Output standards7677- Roadmap items name **eval slice**, **prompt surface**, and **risk tier**78- No prompt prod change without documented **baseline vs candidate**79- Waivers require risk sign-off for Tier 1–2 customer-facing agents80- Escalations include trade-offs (scope, date, headcount)8182## When to load references8384- **Org** → `references/team_org_prompt_eval.md`85- **Roadmap** → `references/roadmap_prioritization.md`86- **Release** → `references/release_governance.md`87- **Stakeholders** → `references/stakeholder_partnerships.md`88- **People** → `references/hiring_development.md`89- **KPIs** → `references/team_metrics_accountability.md`
daemon-blockint-tech