AI Lead Ops
When to Use
- Standing up AI platform operations and production service reliability
- Defining SLAs/SLOs for LLM-powered features
- Running AI incident reviews and post-mortems
- Governing model, prompt, and index rollouts with tiered gates
- Tracking AI unit economics (cost per session, tokens per feature)
- Coordinating red-team and evaluation gates before releases
- Building team rituals and cadence across engineering, research, risk, and product
- Managing AI vendor relationships, contracts, and bake-offs
When NOT to Use
- Implementing memory stores or context packing code →
ai-memory-developer / ai-context-engineer
- Building RAG pipelines or agent tools →
ai-engineer
- Designing corporate AI policy or regulatory mapping →
ai-risk-governance
- General network penetration testing or enterprise security programs →
cybersecurity
- Structured token/cost improvement roadmaps with backlog →
ai-token-improvement-plan-engineer
- Commercial/enterprise AI solution architecture →
applied-ai-architect-commercial-enterprise
- Vertical AI product engineering managers and squad roadmaps →
engineering-manager-vertical-ai-products
Related skills
| Need |
Skill |
| Build RAG, agents, eval harnesses |
ai-engineer |
| Memory and context implementation |
ai-memory-developer, ai-context-engineer |
| Risk tiering and policies |
ai-risk-governance |
| Adversarial testing execution |
ai-redteam |
| CI/CD and platform incidents |
devops |
| Pipeline security |
devsecops |
| Token optimization roadmap and initiative backlog |
ai-token-improvement-plan-engineer |
| Commercial/enterprise AI architecture |
applied-ai-architect-commercial-enterprise |
| Skills portfolio governance |
ai-skill-manager |
| Safeguard inference platform |
ml-infrastructure-engineer-safeguards |
| Safety classifier research |
ml-research-engineer-safeguards |
Core Workflows
1. Operating model and cadence
| Ritual |
Frequency |
Outcomes |
| AI ops standup |
Daily |
Blockers, incidents, deploys |
| Model/prompt change review |
Per release |
Approvers, eval delta |
| Cost review |
Weekly |
Spend vs budget, top features |
| Risk & safety sync |
Bi-weekly |
Incidents, policy gaps |
| Quarterly capacity |
Quarterly |
Model roadmap, vendor contracts |
Define RACI: who owns model, prompt, index, eval suite, on-call.
See references/operating_model.md for roles and escalation.
2. Release governance
Production promotion checklist:
See references/release_governance.md for tiered gates and canary metrics.
3. SLOs, incidents, and observability
Example SLIs:
| SLI |
Notes |
| Availability |
Successful completion / total requests |
| Latency |
p95 end-to-end |
| Quality proxy |
Thumbs-down rate, escalation rate |
| Safety |
Policy violation rate post-deploy |
| Cost |
USD per successful session |
AI incident types: toxic output, PII leak in logs, retrieval cross-tenant leak, runaway agent loop, vendor outage.
See references/incidents_slos.md for severity matrix and post-incident template.
4. Cost and capacity
- Track tokens by model, feature, tenant
- Set budgets and alerts at 80/100/110%
- Optimize via routing, caching, context engineering (partner with
ai-context-engineer)
- Forecast from usage growth + model price changes
See references/cost_capacity.md for unit economics worksheet.
5. Vendor and eval program
- Maintain scorecard: quality, latency, safety, price, data terms
- Run structured bake-offs before annual renewals
- Own central eval harness ownership and dataset hygiene
See references/vendor_eval_program.md for RFP topics and eval program maturity.
When to load references
- Team cadence and RACI →
references/operating_model.md
- Releases and canaries →
references/release_governance.md
- SLOs and incidents →
references/incidents_slos.md
- Cost and capacity →
references/cost_capacity.md
- Vendors and eval ops →
references/vendor_eval_program.md
1---2name: ai-lead-ops3description: Guides AI ops leadership—LLM SRE, model/prompt releases, eval/incidents, cost/capacity, vendors, and cross-functional cadence. Use for AI platform ops, LLM SLAs, incidents, rollout governance, unit economics, red-team/eval gates, and team rituals—not memory (ai-memory-developer), context code (ai-context-engineer), security programs (cybersecurity), token roadmaps (ai-token-improvement-plan-engineer), solution architecture (applied-ai-architect-commercial-enterprise), skills portfolio (ai-skill-manager), or vertical AI product eng management (engineering-manager-vertical-ai-products). Prompt/eval team management and golden-set release policy: engineering-manager-agent-prompts-evals. Safeguard inference platform: ml-infrastructure-engineer-safeguards. Safeguard model research: ml-research-engineer-safeguards.4---56# AI Lead Ops78## When to Use910- Standing up AI platform operations and production service reliability11- Defining SLAs/SLOs for LLM-powered features12- Running AI incident reviews and post-mortems13- Governing model, prompt, and index rollouts with tiered gates14- Tracking AI unit economics (cost per session, tokens per feature)15- Coordinating red-team and evaluation gates before releases16- Building team rituals and cadence across engineering, research, risk, and product17- Managing AI vendor relationships, contracts, and bake-offs1819## When NOT to Use2021- Implementing memory stores or context packing code → `ai-memory-developer` / `ai-context-engineer`22- Building RAG pipelines or agent tools → `ai-engineer`23- Designing corporate AI policy or regulatory mapping → `ai-risk-governance`24- General network penetration testing or enterprise security programs → `cybersecurity`25- Structured token/cost improvement roadmaps with backlog → `ai-token-improvement-plan-engineer`26- Commercial/enterprise AI solution architecture → `applied-ai-architect-commercial-enterprise`27- Vertical AI product engineering managers and squad roadmaps → `engineering-manager-vertical-ai-products`2829## Related skills3031| Need | Skill |32|---|---|33| Build RAG, agents, eval harnesses | `ai-engineer` |34| Memory and context implementation | `ai-memory-developer`, `ai-context-engineer` |35| Risk tiering and policies | `ai-risk-governance` |36| Adversarial testing execution | `ai-redteam` |37| CI/CD and platform incidents | `devops` |38| Pipeline security | `devsecops` |39| Token optimization roadmap and initiative backlog | `ai-token-improvement-plan-engineer` |40| Commercial/enterprise AI architecture | `applied-ai-architect-commercial-enterprise` |41| Skills portfolio governance | `ai-skill-manager` |42| Safeguard inference platform | `ml-infrastructure-engineer-safeguards` |43| Safety classifier research | `ml-research-engineer-safeguards` |4445## Core Workflows4647### 1. Operating model and cadence4849| Ritual | Frequency | Outcomes |50|---|---|---|51| AI ops standup | Daily | Blockers, incidents, deploys |52| Model/prompt change review | Per release | Approvers, eval delta |53| Cost review | Weekly | Spend vs budget, top features |54| Risk & safety sync | Bi-weekly | Incidents, policy gaps |55| Quarterly capacity | Quarterly | Model roadmap, vendor contracts |5657Define RACI: who owns model, prompt, index, eval suite, on-call.5859**See `references/operating_model.md` for roles and escalation.**6061### 2. Release governance6263**Production promotion checklist:**6465- [ ] Eval regression passed on golden + safety set66- [ ] Red-team sign-off for tier-2+ use cases67- [ ] Model card / change log updated68- [ ] Canary with error and cost monitors69- [ ] Rollback procedure tested (previous prompt + model version pinned)70- [ ] Comms plan for customer-visible behavior change7172**See `references/release_governance.md` for tiered gates and canary metrics.**7374### 3. SLOs, incidents, and observability7576**Example SLIs:**7778| SLI | Notes |79|---|---|80| Availability | Successful completion / total requests |81| Latency | p95 end-to-end |82| Quality proxy | Thumbs-down rate, escalation rate |83| Safety | Policy violation rate post-deploy |84| Cost | USD per successful session |8586**AI incident types:** toxic output, PII leak in logs, retrieval cross-tenant leak, runaway agent loop, vendor outage.8788**See `references/incidents_slos.md` for severity matrix and post-incident template.**8990### 4. Cost and capacity9192- Track tokens by model, feature, tenant93- Set budgets and alerts at 80/100/110%94- Optimize via routing, caching, context engineering (partner with `ai-context-engineer`)95- Forecast from usage growth + model price changes9697**See `references/cost_capacity.md` for unit economics worksheet.**9899### 5. Vendor and eval program100101- Maintain scorecard: quality, latency, safety, price, data terms102- Run structured bake-offs before annual renewals103- Own central eval harness ownership and dataset hygiene104105**See `references/vendor_eval_program.md` for RFP topics and eval program maturity.**106107## When to load references108109- **Team cadence and RACI** → `references/operating_model.md`110- **Releases and canaries** → `references/release_governance.md`111- **SLOs and incidents** → `references/incidents_slos.md`112- **Cost and capacity** → `references/cost_capacity.md`113- **Vendors and eval ops** → `references/vendor_eval_program.md`
daemon-blockint-tech