# Fastapi AI Patterns > Production patterns and gotchas for building FastAPI services, especially AI/ML/LLM serving. Use when you build, review, or debug a FastAPI app: choosing def vs async def, loading models in lifespan, preventing event-loop blocking, setting response_model to stop data leaks, enforcing object-level authz in queries, wrapping LLMs with streaming + Pydantic validation loops, or designing RAG/embedding endpoints. - Skill: `daviddwlee84/fastapi-ai-patterns` (Agent Skill, multi-file: 9 files) - Install (CLI): `npx skillmds@latest add daviddwlee84/fastapi-ai-patterns` - Raw SKILL.md: https://api.skillmd.com/api/skills/daviddwlee84/fastapi-ai-patterns/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: daviddwlee84 (https://skillmd.com/u/daviddwlee84) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/daviddwlee84/fastapi-ai-patterns --- # FastAPI AI Patterns Opinionated, production-calibrated guidance for FastAPI services — with the focus on what makes **AI/ML/LLM serving** different from an ordinary CRUD API. It is a knowledge skill: it teaches the decisions and the traps, then routes you to a per-topic reference for depth. Stock agents already write basic FastAPI well (routes, Pydantic models, simple CRUD). This skill spends its tokens on the parts they get **wrong by default**: the concurrency model, the lifespan/singleton lifecycle, the security boundaries, and the discipline that turns a stochastic LLM into a typed component. > Inspired by *FastAPI for AI Engineers: From First Endpoint to Production-Scale > AI Systems* (AI Engineering Insider, 2026). All content here is re-expressed in > original wording; consult the book for its worked examples and case studies. ## When to use - Building a FastAPI service that serves ML models, embeddings, RAG, or wraps an LLM - Reviewing or debugging a FastAPI app for production-readiness (latency, leaks, auth) - Deciding `def` vs `async def` for a handler, or diagnosing a "p99 exploded under load" incident - Designing inference, streaming-chat, embedding, or document-Q&A endpoints - Hardening an endpoint: response shape, authz, rate limiting, input/output guardrails ## When NOT to use - You want a full project skeleton generated → use `fastapi-ai-scaffold` - You are drilling interview questions on these topics → use `fastapi-ai-interview-prep` - Pure frontend/Next.js work, or a non-FastAPI Python web stack (Flask/Django) → different skills ## Core mental model: the endpoint is a typed contract A FastAPI endpoint signature **is** the API contract. Parameter annotations declare what the request must contain; the return annotation or `response_model` declares what the response will contain. The framework enforces both at runtime and publishes them as OpenAPI. Everything else — validation, docs, the 422 error body, SDK generation — is derived from that one declaration. Two consequences worth internalizing: 1. **The annotation is the single source of truth.** Don't duplicate validation logic that a Pydantic field constraint already expresses. 2. **FastAPI is a thin layer over Starlette + Pydantic.** When you debug, you are often reading Starlette (routing, middleware, `BackgroundTasks`) or Pydantic (validation, coercion) source. Knowing which layer owns a behavior is the difference between guessing and fixing. ## The single most important decision: `def` vs `async def` FastAPI runs `async def` handlers on **one event loop**; it dispatches plain `def` handlers to a **threadpool** (~40 threads by default). Choose by the workload, not by habit: | Workload | Correct handler | Why | |---|---|---| | Async-capable I/O (`httpx`, `asyncpg`, async redis) | `async def` + `await` | Loop interleaves thousands of in-flight waits | | Blocking-only library (`requests`, classic ORM, `boto3`) | plain `def` | Threadpool absorbs the block; loop stays free | | Light CPU (< a few ms) | either | Negligible either way | | Heavy CPU (model inference, parsing, crypto) | plain `def`, or offload (`to_thread`/process pool/queue) | Never run multi-ms CPU on the loop | The deadliest FastAPI production bug: an `async def` handler that makes a **blocking** call inside it (`requests.get`, `time.sleep`, a sync model `.predict()`, a CPU-heavy parse). It passes every single-request test, then under concurrency it freezes the entire loop — and p99 explodes for **every** endpoint at once, including `/health`. See `references/async-and-external.md`. ## Top gotchas (read before you write the handler) These are the corrections an agent most often needs. They are environment facts that defy reasonable assumptions, so they live here, not in a reference. - **`async def` + blocking call = whole-process stall.** One non-cooperating coroutine starves every other request on the loop. If a call isn't truly async, use plain `def` or offload it. Detect with a loop-lag watchdog. - **Load ML models once, in the `lifespan` handler — never per request, never at import time.** Per-request loading re-reads weights on every call (and OOMs under concurrency); import-time loading makes tests, linters, and CLI tools pay the cost (or crash without a GPU). Lifespan loading runs once per worker before traffic and integrates with readiness probes. With N workers you hold N copies — size worker count by memory, or front a dedicated inference server. - **Every response needs a `response_model` (or explicit return type).** Returning an ORM object directly serializes *all* its columns — that is how `hashed_password` ends up in a JSON response. Separate request / storage / response schemas; the response schema is a structural leak-prevention boundary. - **Object-level authorization must be enforced in the query, not by review.** BOLA (broken object-level authorization) tops the OWASP API list because `GET /orders/{id}` that trusts the path id leaks other users' data. Filter by owner in the data layer (`WHERE owner_id = current_user.id`), so an unauthorized fetch returns 404 by construction — don't rely on a reviewer remembering. - **Echo `model_version` (and embedding model version) in every AI response.** When quality regresses the first question is "which model produced this?", and canary rollouts serve multiple versions at once. Vectors from different embedding models are incomparable — a model upgrade means re-embedding the corpus. - **Treat LLM output as untrusted input: generate → validate with Pydantic → retry with the error appended → fall back.** This loop is the single most useful LLM-engineering pattern: it converts a stochastic text generator into a typed component with an explicit failure mode. One retry fixes most cases; on the second failure raise a typed error to a defined fallback (human review, cheaper deterministic path, or 502) — never pass malformed data downstream. - **In RAG, enforce access control in retrieval (SQL/filter), never in the prompt.** Never let the model see chunks the user can't access and "ask it" to respect permissions — prompt injection defeats that. The authorization boundary lives outside the model. - **Per-process singletons (`httpx.AsyncClient`, DB engine, Redis client, model handle) belong in `lifespan`, injected via a dependency — not constructed per request.** A per-request `httpx` client pays TCP+TLS handshakes every call and leaks connections; a lifespan-scoped client reuses keepalive connections and enforces global limits. - **Wrap streaming generators in `try/finally`.** Clients disconnect mid-stream constantly; token metering, logging, and upstream stream cancellation must run even then, or you keep paying for abandoned generations and lose usage data. - **Keep `/health` trivially cheap; use a separate `/ready` for dependency readiness.** A health check that does real work (or shares the event loop with heavy handlers) will fail under load and trigger restart storms. Liveness and readiness are different questions and need different probes. - **Retry only idempotent operations on transient errors, with backoff + jitter and a total deadline below your caller's timeout.** Naive retries amplify load exactly when an upstream is weakest (retry storm); synchronized retries are a self-inflicted thundering herd. Add a circuit breaker for repeated failure. - **Validate JWTs with a pinned algorithm; hash passwords with bcrypt/argon2, not SHA-256.** Accepting the token's own `alg` header enables the `alg=none` forgery; fast hashes make password cracking cheap. See `references/security.md`. ## How to use the references Load the one reference that matches the decision in front of you — don't preload them all. Each is self-contained with patterns, gotchas, and capacity/cost rules restated in original wording. - **`references/api-design.md`** — Read when designing routes, status codes, pagination/versioning, or Pydantic v2 request/response schemas (Ch 1–3 scope: ASGI vs WSGI, REST semantics, idempotency, validation, leak prevention). - **`references/architecture-di.md`** — Read when structuring a project or wiring `Depends()` (Ch 4: DI resolution + caching, `yield` deps for DB sessions, router/service/repository layering, composition root, `dependency_overrides`). - **`references/database.md`** — Read when adding persistence (Ch 5: session-per- request, N+1 + `selectinload`/`joinedload`, SQLModel vs SQLAlchemy+Pydantic, Alembic, pool sizing, PgBouncer, `flush` vs `commit`). - **`references/security.md`** — Read when adding auth or hardening (Ch 6: password hashing, JWT + revocation, API keys, CORS, rate limiting, OWASP API risks, BOLA). - **`references/testing.md`** — Read when writing tests (Ch 7: `TestClient`, `dependency_overrides`, SQLite-in-memory tiers, authz matrix, time injection, coverage honesty, 422-body assertions). - **`references/async-and-external.md`** — Read when doing concurrency or external calls (Ch 8: event loop model, the full `def`/`async` table, `httpx` client + retries + circuit breaker, `BackgroundTasks` vs Celery, webhooks, uploads, streaming, WebSockets, backpressure). - **`references/ai-ml-serving.md`** — Read when serving models or LLMs (Ch 9: model loading, batch + dynamic micro-batching, LLM gateway + SSE streaming, embeddings + vector DB + RAG, guardrails, validation loop, versioning/monitoring/ HITL, LLM cost model). - **`references/deployment-observability.md`** — Read when deploying or operating (Ch 10: Uvicorn/Gunicorn workers, Docker, K8s probes, logs/metrics/traces, caching + stampede, load testing, graceful degradation, production checklist). ## Authoritative sources (link, don't paraphrase from memory) FastAPI's API surface and the surrounding ecosystem change between minor versions. When unsure of exact syntax or current behavior, fetch the docs: - FastAPI: - Pydantic v2: - Starlette: - SQLModel: - httpx: ## Gotchas The top, cross-cutting gotchas are in the section above (they apply before you even pick a topic). Topic-specific traps live in each reference's own `## Gotchas` section — for example, `flush()` vs `commit()` timing in `database.md`, the `alg=none` attack in `security.md`, and SSE proxy-buffering in `ai-ml-serving.md`. Load the reference for the area you're working in so you read its gotchas **before** hitting the situation, not after.