AI Engineer
When to Use
- Building chatbots, copilots, or retrieval-augmented generation systems
- Designing multi-step agent workflows with tool use
- Integrating OpenAI, Anthropic, or local models into products
- Setting up RAG pipelines (chunk, embed, index, retrieve, rerank, generate)
- Building evaluation harnesses and regression suites for LLM features
- Optimizing cost/latency through model routing, caching, or context strategy
- Planning safe deployment of generative features (canary, kill switch, monitoring)
When NOT to Use
- Academic literature synthesis or research methodology →
ai-researcher
- Organizational AI policy, regulation, or risk tiering →
ai-risk-governance
- Adversarial safety testing and jailbreak campaigns →
ai-redteam
- Prompt-only tuning without system architecture changes →
prompt-engineer
- Enterprise-wide non-AI system integration ADRs →
senior-system-architecture
- Token/cost improvement program planning and roadmap →
ai-token-improvement-plan-engineer
- Commercial/enterprise AI solution architecture →
applied-ai-architect-commercial-enterprise
- Skills portfolio governance and batch validation →
ai-skill-manager
- Agent prompts, golden evals, judge rubrics →
prompt-engineer-agent-prompts-evals
Related skills
| Need |
Skill |
| Prompt templates and agent message design |
prompt-engineer |
| Offline experiments, statistics, classical ML |
data-scientist |
| Papers, benchmarks, research methodology |
ai-researcher |
| Policies, model cards, governance |
ai-risk-governance |
| Red-team and jailbreak campaigns |
ai-redteam |
| Persistent memory design |
ai-memory-developer |
| Context window and token budgeting |
ai-context-engineer |
| Token cost improvement plan and roadmap |
ai-token-improvement-plan-engineer |
| AI production ops and release governance |
ai-lead-ops |
| Cross-system boundaries and platform ADRs |
senior-system-architecture |
| Commercial/enterprise AI architecture |
applied-ai-architect-commercial-enterprise |
| Agent skills catalog and validation |
ai-skill-manager |
| Safeguard serving stack and policy runtime |
ml-infrastructure-engineer-safeguards |
| Safety model R&D and benchmark design |
ml-research-engineer-safeguards |
Core Workflows
1. Solution shaping
- Define user job, success metric, and failure modes
- Decide: single LLM call vs RAG vs multi-step agent
- Choose model tier (quality vs cost vs latency)
- Identify data sources, PII boundaries, and retention
- Plan human-in-the-loop for high-risk actions
See references/solution_patterns.md for RAG vs fine-tune vs agent decision tree.
2. RAG pipeline
ingest → chunk → embed → index → retrieve → rerank → generate → cite
Checklist:
See references/rag_pipeline.md for chunking, eval metrics, and freshness.
3. Agents and tools
- Tools: narrow schemas, idempotent where possible, timeouts
- Loop: plan → act → observe → stop condition
- Cap iterations and token budget
- Log tool calls for audit; redact secrets in traces
See references/agents_tools.md for ReAct patterns and failure handling.
4. Evaluation before launch
| Layer |
Measure |
| Retrieval |
Recall@k, MRR on golden questions |
| Generation |
Faithfulness, answer relevance (LLM-judge + human sample) |
| Safety |
Refusal rate on policy violations |
| Ops |
p95 latency, cost per session |
Ship only when regression suite passes on CI for golden set.
See references/evaluation_ops.md for datasets, CI eval, and monitoring.
5. Production operations
- Version prompts and models; canary new versions
- Monitor drift, error rate, tool failures, spend
- Kill switch for model or feature flag
- Incident runbook for toxic output or data leak
See references/evaluation_ops.md for production monitoring.
When to load references
- Architecture choices →
references/solution_patterns.md
- RAG implementation →
references/rag_pipeline.md
- Agents and tools →
references/agents_tools.md
- Eval and production →
references/evaluation_ops.md
1---2name: ai-engineer3description: Guides production AI engineering—LLM apps, RAG, agents, eval harnesses, observability, cost/latency, and safe deployment. Use when building chatbots, copilots, retrieval, agent workflows, model routing, or LLM API integration—not research synthesis (ai-researcher), AI policy (ai-risk-governance), red-team (ai-redteam), prompt-only work (prompt-engineer), non-AI ADRs (senior-system-architecture), token programs (ai-token-improvement-plan-engineer), solution architecture (applied-ai-architect-commercial-enterprise), skills portfolio (ai-skill-manager), or vertical AI engineering management (engineering-manager-vertical-ai-products). Agent prompt authoring and eval harness design: prompt-engineer-agent-prompts-evals. Safeguard inference gateways and moderation infra: ml-infrastructure-engineer-safeguards. Safeguard classifier research: ml-research-engineer-safeguards.4---56# AI Engineer78## When to Use910- Building chatbots, copilots, or retrieval-augmented generation systems11- Designing multi-step agent workflows with tool use12- Integrating OpenAI, Anthropic, or local models into products13- Setting up RAG pipelines (chunk, embed, index, retrieve, rerank, generate)14- Building evaluation harnesses and regression suites for LLM features15- Optimizing cost/latency through model routing, caching, or context strategy16- Planning safe deployment of generative features (canary, kill switch, monitoring)1718## When NOT to Use1920- Academic literature synthesis or research methodology → `ai-researcher`21- Organizational AI policy, regulation, or risk tiering → `ai-risk-governance`22- Adversarial safety testing and jailbreak campaigns → `ai-redteam`23- Prompt-only tuning without system architecture changes → `prompt-engineer`24- Enterprise-wide non-AI system integration ADRs → `senior-system-architecture`25- Token/cost improvement program planning and roadmap → `ai-token-improvement-plan-engineer`26- Commercial/enterprise AI solution architecture → `applied-ai-architect-commercial-enterprise`27- Skills portfolio governance and batch validation → `ai-skill-manager`28- Agent prompts, golden evals, judge rubrics → `prompt-engineer-agent-prompts-evals`2930## Related skills3132| Need | Skill |33|---|---|34| Prompt templates and agent message design | `prompt-engineer` |35| Offline experiments, statistics, classical ML | `data-scientist` |36| Papers, benchmarks, research methodology | `ai-researcher` |37| Policies, model cards, governance | `ai-risk-governance` |38| Red-team and jailbreak campaigns | `ai-redteam` |39| Persistent memory design | `ai-memory-developer` |40| Context window and token budgeting | `ai-context-engineer` |41| Token cost improvement plan and roadmap | `ai-token-improvement-plan-engineer` |42| AI production ops and release governance | `ai-lead-ops` |43| Cross-system boundaries and platform ADRs | `senior-system-architecture` |44| Commercial/enterprise AI architecture | `applied-ai-architect-commercial-enterprise` |45| Agent skills catalog and validation | `ai-skill-manager` |46| Safeguard serving stack and policy runtime | `ml-infrastructure-engineer-safeguards` |47| Safety model R&D and benchmark design | `ml-research-engineer-safeguards` |4849## Core Workflows5051### 1. Solution shaping52531. Define user job, success metric, and failure modes542. Decide: single LLM call vs RAG vs multi-step agent553. Choose model tier (quality vs cost vs latency)564. Identify data sources, PII boundaries, and retention575. Plan human-in-the-loop for high-risk actions5859**See `references/solution_patterns.md` for RAG vs fine-tune vs agent decision tree.**6061### 2. RAG pipeline6263```64ingest → chunk → embed → index → retrieve → rerank → generate → cite65```6667**Checklist:**6869- [ ] Chunk size tuned on eval set70- [ ] Metadata filters for tenancy/ACL71- [ ] Hybrid search if keyword matters72- [ ] Ground answers with citations; refuse when context insufficient73- [ ] Refresh index on source updates7475**See `references/rag_pipeline.md` for chunking, eval metrics, and freshness.**7677### 3. Agents and tools7879- Tools: narrow schemas, idempotent where possible, timeouts80- Loop: plan → act → observe → stop condition81- Cap iterations and token budget82- Log tool calls for audit; redact secrets in traces8384**See `references/agents_tools.md` for ReAct patterns and failure handling.**8586### 4. Evaluation before launch8788| Layer | Measure |89|---|---|90| Retrieval | Recall@k, MRR on golden questions |91| Generation | Faithfulness, answer relevance (LLM-judge + human sample) |92| Safety | Refusal rate on policy violations |93| Ops | p95 latency, cost per session |9495Ship only when regression suite passes on CI for golden set.9697**See `references/evaluation_ops.md` for datasets, CI eval, and monitoring.**9899### 5. Production operations100101- Version prompts and models; canary new versions102- Monitor drift, error rate, tool failures, spend103- Kill switch for model or feature flag104- Incident runbook for toxic output or data leak105106**See `references/evaluation_ops.md` for production monitoring.**107108## When to load references109110- **Architecture choices** → `references/solution_patterns.md`111- **RAG implementation** → `references/rag_pipeline.md`112- **Agents and tools** → `references/agents_tools.md`113- **Eval and production** → `references/evaluation_ops.md`
daemon-blockint-tech