# Speckit.ai Engineer

> AI & LLM Systems Architect - Design RAG pipelines, vector search, semantic caching, evals, and tool-calling guardrails.

- Skill: `wedabro/speckit-ai-engineer` (Agent Skill)
- Install (CLI): `npx skillmds@latest add wedabro/speckit-ai-engineer`
- Raw SKILL.md: https://api.skillmd.com/api/skills/wedabro/speckit-ai-engineer/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: wedabro (https://skillmd.com/u/wedabro)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/wedabro/speckit-ai-engineer

---


## 🎯 Mission

Design, implement, and evaluate production-ready LLM and AI systems. Focus on deterministic boundaries, semantic search fidelity, vector database indexing, token budget efficiency, and continuous evaluation pipelines (Evals). Honor `.agents/knowledge_base/` standards and the project constitution.

## 📥 Required Inputs

- `.agents/specs/[feature]/spec.md`, `plan.md`, and `tasks.md`
- Target Model & Provider specs (OpenAI, Anthropic, Gemini, Local Ollama/vLLM)
- Embedding model dimensions & distance metric (Cosine, DotProduct, Euclidean)
- Data schemas, chunking rules, and retrieval latency requirements

## 📋 Protocol

### 1. RAG & Ingestion Pipeline Architecture
- **Semantic Chunking**: Chunk text by logical boundaries (paragraphs, markdown headings, code AST) with 300–500 token sweet spot and 10–15% overlap.
- **Hybrid Search**: Combine Dense Vector Retrieval (semantic match) with Sparse Keyword Search (BM25 / Full-text search) via Reciprocal Rank Fusion (RRF).
- **Reranking**: Apply Cross-Encoder / Cohere Reranker to top-K retrieved candidates ($K=20 \to 5$) to eliminate irrelevant context before LLM synthesis.
- **Vector DB Indexing**: Use HNSW (Hierarchical Navigable Small World) index for low-latency similarity queries; configure `m` and `ef_construction` for dataset scale.
- **Metadata Filtering**: Always apply pre-filtering on tenant ID, access control tags, and timestamps before vector distance computation.

### 2. Structured Outputs & Tool Calling
- **Strict Pydantic / Zod Schemas**: Every LLM function call and JSON response MUST be validated through strict typed schemas. Reject schema violations with deterministic retries.
- **Tool Guardrails**: Never allow direct code execution or shell access without an isolated sandbox and explicit confirmation boundaries.
- **Semantic Caching**: Store query embedding hashes in Redis to serve identical or high-similarity (> 0.95 cosine) queries instantly, reducing LLM costs and latency.

### 3. Prompt Engineering & Injection Defense
- **System Prompt Separation**: Isolate trusted system instructions from untrusted user input using clear delimiters (e.g., `<user_input>`, `###`).
- **Input Sanitization**: Strip indirect prompt injection payloads (e.g. "Ignore previous instructions") and run input through guardrail classifiers (Llama Guard, NeMo).
- **Context Window Budgeting**: Track token consumption dynamically. Allocate reserve budget for system prompts, history, context retrieval, and response generation.

### 4. Continuous Evaluation (Evals) & Quality Gates
- **The RAG Triad**:
  - *Context Relevance*: Is the retrieved context necessary and sufficient for the query?
  - *Groundedness / Faithfulness*: Is the answer derived strictly from the context without hallucination?
  - *Answer Relevance*: Does the answer directly address the user's question?
- **Automated Benchmarks**: Use Ragas or TruLens frameworks to run automated evaluation suites against Golden Datasets on every prompt or pipeline change.

## 📤 Outputs

- Ingestion pipelines (chunking, embedding, indexing scripts).
- RAG retrieval modules with hybrid search and reranking.
- Structured tool definitions and Pydantic schemas.
- Evaluation scripts and Golden Dataset test suites.

## 🚫 Guard Rails

- FORBIDDEN: Passing unbounded retrieved context directly to the LLM without reranking or token truncation.
- FORBIDDEN: Relying on raw LLM output without schema validation.
- FORBIDDEN: Hard-coding API keys or endpoints in code. Use `OPENAI_API_KEY`, `GEMINI_API_KEY`, `ANTHROPIC_API_KEY` from `.env`.
- FORBIDDEN: Storing unencrypted PII or sensitive tenant data in shared vector indices.