11 AI Architecture Spec

AI Architecture Spec Skill

Use When

• One or more production AI features have approved value, data and evaluation requirements.

Do Not Use When

• Do not use for projects without AI or to select a provider/model without current verification and measurable criteria.

Required Inputs

Workflow

1. Read the named inputs and confirm their approval, version and unresolved decisions.
2. Apply the decision rules below before drafting; stop on a missing authority, unsafe assumption or unresolved scope driver.
3. Produce the AI Architecture Specification and ADR seeds through the existing domain procedure and load only the references needed for the chosen branch.
4. Trace each material statement in the AI Architecture Specification and ADR seeds to an input, decision or explicitly qualified assumption.
5. Verify the observable acceptance conditions, record unassessed checks, and hand the artefacts to their named consumers.
6. If validation fails, recover by correcting the source decision or artefact and rerun the affected check; do not weaken the acceptance condition.

Outputs

Evidence Produced

Capability and permission boundaries

Read-only is the default for analysis, review, evaluation and planning. Read and search access to authorised project artefacts are required. Editing is limited to an explicitly requested project deliverable. Execution may run document, syntax or validation checks. Network access is used only for facts that require current verification. Do not publish, spend, change production, approve policy, or claim certification without explicit authority.

Degraded mode

If any required capability is unavailable, return the narrowest useful qualified AI Architecture Specification and ADR seeds draft plus a gap register showing the missing item, affected sections, risk and owner. Never convert an unassessed check into a pass.

Decision Rules

Quality Standards

• Preserve repository terminology and trace every material choice to project context.
• Use deterministic acceptance conditions; replace vague quality claims with an observable check, threshold or named approval.
• Cover error, empty, edge, recovery and operational cases relevant to this skill.
• Verify standards, citations, APIs and package names before relying on them; qualify what cannot be checked.
• Stop release for a failed safety, security, legal, financial, accessibility or data-integrity gate.

Anti-Patterns

• Letting each service call providers directly. Fix: route through the governed model gateway.
• Choosing RAG because AI is required. Fix: compare direct call, retrieval, fine-tune and non-AI alternatives.
• Sharing embeddings across tenants without policy. Fix: enforce namespace and retrieval filters.
• Treating offline eval as sufficient. Fix: add production monitoring and feedback controls.
• Pinning a model name forever. Fix: specify capability criteria, version evidence and replacement tests.

References

• Architecture patterns
• Architecture template
• Agent runtime cross-link
• AI architecture oversight and drift

Overview

The AI-distinctive architecture artefact. Sits alongside the multi-tenancy spec and the generic HLD. Captures the model gateway, vector store, eval harness, prompt registry, observability bus, and the multi-tenant AI security boundaries.

Core Instructions

Step 1: Read context

Read HLD, multi-tenancy spec, AI feature PRD spec, AI data spec. Identify in-scope AI features, models, patterns, and the tenant boundaries.

Step 2: Declare the AI plane

Include the problem/non-AI alternative, human and affected-non-user map, model/system/input/output boundaries, human control, consent/notice, drift signals, staged promotion, rollback and re-promotion criteria. Use the oversight reference before selecting a runtime pattern.

The AI plane is a sub-set of the application plane plus a small set of dedicated control-plane services:

• Model Gateway (control plane) — single egress for model-provider calls; carries auth, tenant-id propagation, per-tenant rate limit, cost meter, request/response log, content-filter, fallback routing.
• Prompt Registry (control plane) — versioned prompts, change-control, regression-test attachment.
• Vector Store (application plane, per-tenant or namespaced) — embedding-backed retrieval.
• Eval Harness Runner (control plane) — runs eval suites against new prompt/model versions in CI.
• Observability Bus — token use, latency, fallback rate, abstention, citation rate, judge-LLM score, cost per tenant.

Diagram with Mermaid; place every AI service.

Step 3: Map each AI feature to a pattern

For each AI feature select the pattern:

State the verdict per feature with rejected alternatives.

Step 4: Specify the Model Gateway

The gateway is the sole egress to model providers. Capture:

• Supported providers and models (primary + fallback per feature).
• Authentication and credential rotation.
• Tenant-id propagation as a guarded claim.
• Per-tenant and per-feature rate limit and cost ceiling.
• Request/response log retention.
• Content-filter chain (input and output).
• Fallback routing rule (model-down, cost-overrun, latency-overrun, content-filter-trip).
• Idempotency keys for retries.

Step 5: Specify the Vector Store

For each retrieval index: store technology, partitioning model (per-tenant index / namespace / metadata-filter), embedding model + version, dimensions, ANN parameters, freshness, encryption posture, key management.

Step 6: Specify the Prompt Registry

Versioned, tagged, changes proposed via PR with regression eval attached. State the registry source-of-truth, deploy pipeline, rollback procedure.

Step 7: Specify the Eval Harness in architecture terms

The eval harness is a first-class production system, not a notebook. State: dataset store, judge-LLM, CI gate hook, scheduled regression, alerting on score drop.

Step 8: Specify observability

AI-specific signals: tokens in/out per request, model latency per provider, fallback rate, abstention rate, citation rate, judge-LLM score, cost per tenant per feature, content-filter trips, red-team alerts.

Step 9: Specify security boundaries

• Prompt injection surface (untrusted text in retrieved docs, in user input, in tool outputs).
• Sandboxing of tool execution.
• Egress allow-list at the gateway.
• Secrets handling: never in prompts; tool-side fetch via tenant-id claim.
• Cross-tenant retrieval prohibition enforced at the gateway.

Step 10: Emit ADR seeds

ADR seeds: model choice per feature, RAG-vs-fine-tune, vector store choice, eval threshold, abstain policy, content filter, fallback policy.

Step 11: Write the spec

AI_Architecture_Spec.md sections: 1) AI Plane Diagram, 2) Feature-to-Pattern Map, 3) Model Gateway, 4) Vector Store, 5) Prompt Registry, 6) Eval Harness, 7) Observability, 8) Security Boundaries, 9) ADR Seed Index, 10) Traceability.

Standards

• AWS Well-Architected ML/AI Lens
• OWASP LLM Top 10
• NIST AI RMF MAP / MEASURE
• ISO/IEC 42001

For human-owned decisions, governed memory, correction, drift, and fallback, load human agency and system boundaries.