Production FastAPI Template: Rules
These rules define where code lives and how it executes in a unified FastAPI + RAG + Agentic AI backend. Apply them whenever you generate or modify code in a project built from this template.
Companion references. Read the relevant file before writing code:
| File | Read before writing… | Summarized in |
|---|---|---|
| ASYNC_EXECUTION.md | Any route, service, client, agent tool, worker or background job | Section 5 |
| PYDANTIC_STANDARDS.md | Any schema, settings class, serialization, LLM call with structured output, citation check, agent plan or tool argument model | Section 6 |
| API_CONVENTIONS.md | Any dependency, route signature, auth/permission check, OpenAPI metadata, DB model or query, migration, API test or lint setup | Section 7 |
| GUARDRAILS.md | Any user-facing LLM call, RAG answer, ingestion of untrusted documents, agent tool execution, gateway target or content-safety check | Section 8 |
| AI_EVALUATION.md | Any evaluation contract, metric, dataset, judge, runner, report or threshold; any new RAG stage, agent, tool or conversation feature that needs quality coverage | Section 9 |
| API_SECURITY.md | Any authentication, token, permission, tenant or ownership check; input/upload/outbound-URL handling; response headers, CORS, errors; rate limits; audit events; ECS/IAM/secrets; CI security; security tests | Section 10 |
| AGENT_SECURITY.md | Any agent, tool, tool registry entry, approval flow, memory write, MCP connector or LLM-output sink | Section 10 |
Status labels:
- 🟢 Convention. A proposed rule. Follow it by default.
- 🟡 Needs approval. An open team decision. Surface the options to the user and never pick one silently.
These rules don't choose agent frameworks, LLM providers, models, vector stores or a queue implementation. If a rule doesn't fit a real need, raise it with the user instead of silently working around it.
1. Core structure rules 🟢
- One application. FastAPI, RAG and Agentic AI are not separate apps. There is exactly one
src/package, one FastAPI entry point (src/main.py) and one shared infrastructure layer. - Never create a second entry point. Don't add a second
src/, a separate app folder (rag_app/,agents_service/,worker_app/) or an extraFastAPI()instance. - Background workers and Lambda functions are extra entry points into the same codebase. They are not separate applications, and they call the same service functions the API calls.
- One central API router. All routers mount through
src/api/v1/router.py. Domain, RAG and agent routers never mount themselves on the app. - Domain-oriented modules. Business code lives inside its domain package.
src/'s root holds only application-wide concerns. - No duplicated shared layers. Model clients go in
src/llm/, content-safety checks insrc/guardrails/and external-service clients insrc/providers/, each exactly once. RAG and agents consume them and never define their own copies (norag/llm_client.py, noagents/vector_db.py). - Provider-neutral naming. Name files by responsibility, not vendor (
llm/embeddings.py, notopenai_embeddings.py). Vendor specifics stay behindsrc/providers/orsrc/llm/. common/is for genuinely reusable, domain-agnostic code only. Never put business logic there.- Keep agents, reasoning and execution apart. Agent definitions go in
agents/, thinking incognition/and doing, scheduling and offloading inexecution/. - Tests mirror
src/. Quality evaluation (datasets, DeepEval/Ragas suites, deterministic metrics, runners, reports) lives in the top-levelevaluation/directory, separate fromtests/.src/evaluation/holds only framework-neutral contracts, the metric registry and adapters, with lazy framework imports (AI_EVALUATION.md §1). - Every Python directory under
src/has an__init__.py. Non-Python directories that must survive in Git (prompt folders, data, empty test folders) get a.gitkeep. - Explicit module imports across packages, e.g.
from src.auth import constants as auth_constants. - Reuse existing files before adding new ones. Extend
execution/background_worker.pyrather than creatingexecution/worker2.pyorrag/worker.py.
2. Canonical structure
<project-name>/
├── src/
│ ├── __init__.py
│ ├── main.py # the single FastAPI entry point (lifespan creates shared clients/limiters)
│ ├── config.py # application-wide configuration
│ ├── constants.py
│ ├── exceptions.py # base domain exception + handlers rendering ErrorResponse
│ ├── middleware.py
│ ├── database.py # shared metadata (naming convention), engine/session factory, get_db_session
│ ├── models.py # genuinely shared models only
│ ├── pagination.py
│ │
│ ├── api/
│ │ └── v1/
│ │ └── router.py # aggregates domain, RAG and agent routers
│ │
│ ├── auth/ # reference domain module
│ │ ├── router.py
│ │ ├── schemas.py # the domain's own Pydantic API/internal schemas
│ │ ├── models.py # persistence (ORM) models, not Pydantic
│ │ ├── dependencies.py # current_principal, require_<permission>, scope dependencies
│ │ ├── config.py # the domain's own pydantic-settings class (AUTH_ prefix: issuers, audiences, JWKS)
│ │ ├── constants.py # permission and scope names
│ │ ├── exceptions.py
│ │ ├── service.py
│ │ ├── tokens.py # JWT/JWKS validation, token types, claims → Principal
│ │ ├── permissions.py # RBAC/ABAC policy evaluation (default deny)
│ │ ├── security.py # password hashing, only if local accounts are approved 🟡
│ │ ├── audit.py # auth audit events (login, token rejected, session revoked)
│ │ └── utils.py
│ │
│ ├── example_domain/ # copy this layout for every new business domain
│ │ ├── router.py
│ │ ├── schemas.py
│ │ ├── models.py
│ │ ├── dependencies.py # valid_<entity>_id and other I/O-backed validation
│ │ ├── constants.py
│ │ ├── exceptions.py
│ │ ├── service.py
│ │ └── utils.py
│ │
│ ├── rag/
│ │ ├── routes.py # "routes", not "router": avoids clashing with rag/router/
│ │ ├── schemas.py # RAG API schemas + cross-stage contracts (RetrievedEvidence)
│ │ ├── service.py # container/lifecycle of long-lived RAG services
│ │ ├── dependencies.py
│ │ ├── config.py # RAG settings (RAG_ prefix)
│ │ ├── constants.py
│ │ ├── exceptions.py
│ │ ├── extraction.py # structure-preserving document extraction
│ │ ├── ingestion.py # coordinates processing and indexing (run by background jobs)
│ │ ├── semantic/
│ │ │ ├── metadata.py
│ │ │ ├── segmentation.py
│ │ │ └── chunks.py
│ │ ├── router/
│ │ │ ├── classifier.py
│ │ │ ├── fusion.py
│ │ │ └── evaluation.py
│ │ ├── preprocessing/
│ │ │ ├── query_expansion.py
│ │ │ └── hyde.py
│ │ ├── retrieval/
│ │ │ ├── semantic_search.py
│ │ │ ├── bm25.py
│ │ │ ├── rrf.py
│ │ │ ├── mmr.py
│ │ │ ├── evidence_filter.py
│ │ │ └── hybrid_retriever.py
│ │ ├── generation/
│ │ │ ├── context_builder.py
│ │ │ ├── citations.py
│ │ │ ├── streaming.py
│ │ │ ├── schemas.py # LLM-output contracts: GeneratedAnswer, Citation
│ │ │ ├── validation.py # application-level answer/citation validation
│ │ │ └── service.py
│ │ ├── chat/
│ │ │ ├── routes.py
│ │ │ ├── schemas.py
│ │ │ ├── service.py
│ │ │ └── storage.py
│ │ └── security/
│ │ ├── access_control.py # AccessScope model + vector-store/DB filter translation
│ │ ├── document_validation.py # ingestion file checks (type, signature, size, archives)
│ │ └── retrieval_policy.py # post-retrieval ACL re-check, trust labels, provenance
│ │
│ ├── agents/
│ │ ├── router.py
│ │ ├── schemas.py # AgentTask, ToolResult, AgentExecutionState
│ │ ├── structured_output.py # LLM-produced: ExecutionPlan, ToolCall union, FinalAgentResponse
│ │ ├── dependencies.py
│ │ ├── config.py # agent settings (AGENTS_ prefix)
│ │ ├── constants.py
│ │ ├── exceptions.py
│ │ ├── service.py
│ │ ├── base_agent.py
│ │ ├── autonomous_agent.py
│ │ ├── planner_agent.py
│ │ ├── agent_interface.py
│ │ ├── team_orchestrator.py
│ │ ├── step_handler.py
│ │ ├── task_manager.py
│ │ ├── tools/
│ │ │ ├── calculator.py
│ │ │ ├── file_manager.py
│ │ │ ├── search_tool.py
│ │ │ └── tool_registry.py
│ │ ├── workflows/
│ │ │ ├── code_review_chain.py
│ │ │ ├── research_chain.py
│ │ │ ├── multi_agent_workflow.yaml
│ │ │ └── workflow_executor.py
│ │ └── security/ # policies; enforced by execution/executor.py
│ │ ├── tool_permissions.py # default-deny tool authorization (user perms ∩ agent profile)
│ │ ├── execution_policy.py # step, tool-call, token, cost and time budgets
│ │ └── approval_policy.py # human approval bound to argument hash
│ │
│ ├── cognition/
│ │ ├── cognitive_loop.py
│ │ ├── decision_policy.py
│ │ ├── planner.py
│ │ ├── reasoner.py
│ │ ├── state_interpreter.py
│ │ └── memory/
│ │ ├── long_term_memory.py
│ │ ├── short_term_memory.py
│ │ └── memory_manager.py
│ │
│ ├── execution/
│ │ ├── action_resolver.py
│ │ ├── controller.py
│ │ ├── error_handler.py
│ │ ├── executor.py
│ │ ├── threadpool.py # thread-offload helpers and limiters (no custom executor yet)
│ │ ├── job_scheduler.py # enqueue/schedule background jobs
│ │ └── background_worker.py # queue consumer / job dispatch loop
│ │
│ ├── llm/ # shared by RAG and agents
│ │ ├── client.py
│ │ ├── embeddings.py
│ │ ├── reranker.py
│ │ ├── model_loader.py
│ │ ├── cache.py
│ │ ├── dependencies.py # accessors for lifespan-created model clients
│ │ ├── config.py # LLM settings (LLM_ prefix, SecretStr keys)
│ │ ├── schemas.py # structured-output modes, capabilities, failure kinds
│ │ ├── structured_output.py # generate → validate → bounded retry (provider-independent)
│ │ └── prompts/
│ │ ├── system/
│ │ ├── tasks/
│ │ └── templates/
│ │
│ ├── guardrails/ # shared content-safety layer, used by RAG, agents and workers
│ │ ├── schemas.py # GuardrailPhase, GuardrailOutcome, GuardrailFinding, GuardrailVerdict
│ │ ├── service.py # run a phase's checks, decide outcome, enforce / log-only, fail closed
│ │ ├── policies.py # which checks apply to which surface and phase
│ │ ├── config.py # GUARDRAILS_ settings (mode, fail mode, thresholds, policy version)
│ │ ├── dependencies.py
│ │ ├── exceptions.py # GuardrailBlocked, GuardrailUnavailable
│ │ └── validators/ # deterministic in-house checks (regex, lists, length)
│ │
│ ├── providers/ # external-service boundaries (MCP, vector DB, queues, Lambda, guardrails, …)
│ │ ├── guardrails/client.py # managed guardrail API / validator-library adapters
│ │ ├── mcp/client.py
│ │ ├── vector_store/client.py
│ │ └── external/client.py
│ │
│ ├── evaluation/ # runtime-safe evaluation contracts only (no deepeval/ragas at import)
│ │ ├── schemas.py # EvaluationSample, RetrievedContext, AgentTrajectory, EvaluationResult
│ │ ├── registry.py # metric key → framework, class, version, required fields, result kind
│ │ └── adapters/
│ │ ├── deepeval_adapter.py # contracts ↔ LLMTestCase / ConversationalTestCase (lazy import)
│ │ └── ragas_adapter.py # contracts ↔ Ragas collections inputs / samples (lazy import)
│ │
│ └── common/
│ ├── schemas/
│ │ ├── base.py # CustomModel, UtcDatetime, shared annotated types
│ │ ├── responses.py # ErrorResponse and shared response shapes
│ │ └── pagination.py # Page[T], PageParams (models only; logic in src/pagination.py)
│ ├── dependencies.py
│ ├── logging.py
│ ├── retry.py
│ ├── timers.py
│ ├── serialization.py
│ ├── validation.py
│ └── security/
│ ├── headers.py # security-headers ASGI middleware (registered in src/middleware.py)
│ ├── rate_limiting.py # distributed limiter interface + key builders (backend 🟡)
│ ├── request_validation.py # body size/depth, content type, filenames, outbound URL (SSRF) policy
│ └── audit_schemas.py # AuditEvent, AuditActor, AuditTarget
│
├── tests/
│ ├── conftest.py # async client (httpx + ASGITransport), dependency-override fixtures
│ ├── auth/
│ ├── rag/{ingestion,retrieval,generation}/
│ ├── agents/
│ ├── cognition/
│ ├── execution/
│ ├── guardrails/ # fake checkers; deny / suppress / fail-closed / log-only paths
│ ├── evaluation/ # test_schemas.py, test_adapters.py, test_deterministic.py (no LLM calls)
│ ├── security/
│ │ ├── authentication/ # JWT/JWKS validation, throttling
│ │ ├── authorization/ # BOLA, BFLA, BOPLA, cross-tenant, service-to-service
│ │ ├── api/ # headers, CORS, limits, uploads, SSRF, injection, CSRF, rate limits
│ │ ├── rag/ # unauthorized/cross-tenant retrieval, ACL revocation, citation integrity
│ │ └── agents/ # tool authorization, argument BOLA, approval bypass, budgets, injection
│ ├── integration/
│ ├── e2e/
│ └── fixtures/
├── evaluation/ # offline quality evaluation (never imported by src/)
│ ├── README.md
│ ├── config/{metrics.yaml,thresholds.yaml}
│ ├── datasets/{rag,agents,conversations,golden,guardrails}/
│ ├── deepeval/{rag,agents,conversations,custom}/
│ ├── ragas/{rag,agents,custom}/
│ ├── deterministic/{citations,retrieval,structured_output,tool_calls}.py
│ ├── tracing/{collectors,normalization}.py
│ ├── runners/{offline,regression,report}.py
│ ├── fixtures/
│ └── reports/ # generated, git-ignored 🟡
├── data/{documents,knowledge_bases,agent_state,fixtures}/
├── scripts/
├── docs/
│ ├── architecture/
│ ├── api/
│ ├── development/
│ ├── decisions/
│ └── standards/
│ ├── ASYNC_EXECUTION.md # copied from this skill
│ ├── PYDANTIC_STANDARDS.md # copied from this skill
│ ├── API_CONVENTIONS.md # copied from this skill
│ ├── GUARDRAILS.md # copied from this skill
│ ├── AI_EVALUATION.md # copied from this skill
│ ├── API_SECURITY.md # copied from this skill
│ └── AGENT_SECURITY.md # copied from this skill
├── .env.example
├── .gitignore
├── pyproject.toml
├── README.md
└── CLAUDE.md
__init__.py files are omitted above for brevity. Rule 11 still applies.
3. Where does new code go?
| You are adding… | Put it in | Not in |
|---|---|---|
| A new business capability (orders, billing…) | src/<domain>/ using the example_domain/ layout |
src/ root, common/ |
| An HTTP endpoint | The owning module's router.py (routes.py inside rag/), mounted via api/v1/router.py |
main.py |
| Request/response models | <module>/schemas.py, inheriting CustomModel |
src/models.py, direct BaseModel |
Shared base model, UtcDatetime, shared annotated types |
common/schemas/base.py |
Per-domain base classes |
| Error, message and page response shapes | common/schemas/{responses,pagination}.py |
Each domain redefining them |
| Settings for a domain or component | <module>/config.py (own prefix, cached getter) |
The global src/config.py |
| LLM-output schema for answer generation | rag/generation/schemas.py |
rag/schemas.py (API) |
| LLM-output schema for one RAG stage (metadata, classification, expansion) | rag/<stage>/schemas.py, created when the stage is implemented |
API schemas |
| Citation and answer checks against evidence | rag/generation/validation.py |
Pydantic validators |
| Agent plan, tool-call union, final agent response | agents/structured_output.py |
agents/schemas.py |
| A tool's argument model | That tool's module in agents/tools/ |
A central args file |
| Structured-output generation, validation and retry flow | llm/structured_output.py + llm/schemas.py |
Re-implemented in RAG or agents |
Evaluation contracts (EvaluationSample, EvaluationResult, …), metric registry |
src/evaluation/{schemas,registry}.py |
evaluation/, per-suite copies |
| Contract ↔ DeepEval / Ragas mapping | src/evaluation/adapters/ (lazy framework imports) |
Suites building test cases ad hoc |
| Metric suites, G-Eval/DAG/custom metrics, judge wrappers | evaluation/{deepeval,ragas}/ |
src/, tests/ |
| Deterministic metrics (retrieval P/R/MRR/NDCG, citations, schema, tool calls) | evaluation/deterministic/ |
LLM judges |
| Evaluation datasets, metric/threshold config, runners, reports | evaluation/{datasets,config,runners,reports}/ |
data/, tests/fixtures/ |
| Tests of the evaluation code | tests/evaluation/ |
evaluation/ |
| DB models used by one domain | <domain>/models.py |
src/models.py |
| Existence, ownership or uniqueness checks that need I/O | <domain>/dependencies.py (valid_<entity>_id) |
Pydantic validators, repeated in each route |
current_principal, require_<permission>, scope dependencies |
auth/dependencies.py |
Each domain parsing tokens |
JWT/JWKS validation, claims → Principal |
auth/tokens.py |
auth/service.py, routes |
| RBAC/ABAC policy evaluation | auth/permissions.py |
Routes, prompts |
| Password hashing (only if local accounts are approved 🟡) | auth/security.py |
common/ |
Auth audit events; shared AuditEvent schema |
auth/audit.py; common/security/audit_schemas.py |
Free-form log lines |
| Security headers, rate limiting, request/upload/outbound-URL validation | common/security/{headers,rate_limiting,request_validation}.py |
Per-domain copies |
| RAG access scope, document validation, post-retrieval policy | rag/security/ (scope resolved in rag/dependencies.py), applied as a store filter in retrieval |
Prompts, the LLM, evidence_filter.py |
| Agent tool permissions, execution budgets, approval policy | agents/security/ (context built in agents/dependencies.py), enforced by execution/executor.py |
Prompts, the planner, tools themselves |
| Security tests | tests/security/{authentication,authorization,api,rag,agents}/ |
evaluation/ |
| Domain exceptions | <domain>/exceptions.py, subclassing src/exceptions.py |
HTTPException in services |
| SQL queries | <domain>/service.py |
Routers, dependencies |
| DB naming convention, engine, session dependency | src/database.py |
Per-domain engines |
| Migrations 🟡 | alembic/ + alembic.ini at the root, only after approval |
src/, the app lifespan |
| Lint/format script 🟡 | scripts/lint.sh |
Ad-hoc commands in docs |
| Content-safety checks (input, context, tool, output) | Call src/guardrails/service.py at the checkpoints in GUARDRAILS.md §3 |
Checks embedded in rag/ or agents/, system prompts |
| Deterministic safety validators (regex, lists) | guardrails/validators/ |
common/validation.py |
| Managed guardrail or validator-library client | providers/guardrails/client.py |
guardrails/, llm/ |
| Gateway Cedar guardrail policies 🟡 | Infrastructure-as-code, outside src/ (location needs approval) |
src/ |
| Guardrail red-team / benign datasets | evaluation/datasets/guardrails/ |
tests/ |
| Long-lived clients, pools and limiters | Created in main.py's lifespan and exposed through dependencies.py |
Created per request |
| Document parsing and indexing logic | rag/extraction.py, rag/ingestion.py, rag/semantic/ |
providers/, workers |
| Query routing / classification | rag/router/ |
agents/ |
| Query rewriting, HyDE | rag/preprocessing/ |
rag/retrieval/ |
| A retrieval or ranking stage | rag/retrieval/ |
rag/generation/ |
| Answer context, citations, streaming | rag/generation/ |
rag/chat/ |
| Conversation API / history | rag/chat/ |
rag/ root |
| A new agent type | agents/<name>_agent.py implementing agent_interface.py |
cognition/ |
| A tool an agent can call | agents/tools/ + register in tool_registry.py |
common/ |
| A predefined multi-step workflow | agents/workflows/ |
execution/ |
| Planning, reasoning, decision logic | cognition/ |
agents/ |
| Agent memory | cognition/memory/ |
rag/chat/storage.py |
| Running actions, control flow, execution errors | execution/{executor,controller,action_resolver,error_handler}.py |
agents/ |
| Thread-offload helpers, per-dependency limiters | execution/threadpool.py |
Ad-hoc in routes |
| Enqueueing or scheduling a background job | execution/job_scheduler.py |
Routes calling the queue SDK directly |
| Queue consumer / job dispatch | execution/background_worker.py |
A new app or rag/worker.py |
| Lambda handler (thin adapter) 🟡 | Same codebase. The location needs approval. It calls the existing service functions. | A separate repo or app with copied logic |
| Chat/completion, embedding or rerank model access | llm/ |
rag/, agents/ |
| Prompt text | llm/prompts/{system,tasks,templates}/ |
Scattered prompts.py files |
| MCP, vector DB, SQS, Lambda or third-party API client | providers/<kind>/client.py |
rag/, agents/, llm/ |
| Logging, retry, timing, serialization | common/ |
Domain packages |
| Unit tests | tests/<mirrored module>/ |
Next to source |
| RAG/agent quality metrics, benchmarks | evaluation/ (layout in section 2) |
tests/ |
| Sample docs, KB files, agent state | data/ |
src/ |
| One-off CLIs (bulk upload, model training) | scripts/ |
src/ |
| Engineering standards | docs/standards/ |
README.md |
If none of these rows fit, ask the user before inventing a new top-level package.
4. RAG naming conventions 🟢
Use these names. The alternatives were rejected for the reasons given.
| Use | Instead of | Why |
|---|---|---|
rag/routes.py, rag/chat/routes.py |
rag/router.py |
rag/router.py would collide with the rag/router/ package. Keep one name inside rag/. |
rag/router/classifier.py |
rag/router/router.py |
Names the responsibility and avoids router/router.py |
rag/router/fusion.py (only one) |
Fusion routing in both preprocessing/ and router/ |
Routing fusion lives in exactly one place |
rag/retrieval/rrf.py |
retrieval/fusion.py |
Distinguishes rank fusion from routing fusion |
rag/retrieval/evidence_filter.py |
retrieval/filtering.py |
Explicit responsibility |
rag/retrieval/hybrid_retriever.py |
retrieval/hybrid.py |
Matches its role |
rag/retrieval/semantic_search.py |
Dense search hidden inside a vector-DB client | Dense search is its own stage, and the client lives in providers/vector_store/ |
rag/semantic/chunks.py |
rag/semantic/chunking.py |
Contextualized chunk building, kept distinct from any passage-formatting helper |
rag/generation/context_builder.py |
generation/context.py |
Avoids confusion with conversational context in chat/ |
rag/generation/citations.py |
Citation checks inside the context builder | Separate responsibilities |
rag/generation/service.py + llm/client.py |
rag/generation/llm.py |
Orchestration stays in RAG, and model access is shared |
llm/{client,embeddings,reranker}.py |
rag/<vendor>/… |
A vendor-neutral model layer, shared with agents |
llm/model_loader.py |
Client factories in rag/embeddings.py |
Model construction is shared |
providers/vector_store/client.py |
src/<vendor-db>/client.py |
A vendor-neutral provider boundary |
evaluation/deepeval/rag/, evaluation/ragas/rag/, evaluation/reports/ |
evals/ or artifacts inside src/ |
Evaluation suites and reports stay outside the app package; only contracts live in src/evaluation/ |
scripts/ for training and bulk jobs |
Training scripts inside src/rag/router/ |
CLIs stay outside src/ |
Trained model artifacts, datasets and domain-specific label lists are data, not structure.
Keep them under data/ or external storage 🟡, never hard-coded in src/.
5. Execution rules (summary of ASYNC_EXECUTION.md) 🟢
Async vs sync
- Use
async defonly when the whole call chain (route → service → client) awaits genuinely asynchronous libraries. - Use
defroutes for work dominated by sync blocking I/O. FastAPI runs them in AnyIO's thread pool, which has 40 tokens per process by default and is shared with sync dependencies and offloaded calls. - Never call blocking code inside
async def: sync HTTP/DB/vector clients,time.sleep, file parsing or heavy CPU. - Never "convert" sync code by adding
async. Never callasyncio.run()inside a request. - Offload unavoidable sync I/O with
anyio.to_thread.run_sync(..., limiter=...)orstarlette.concurrency.run_in_threadpool. Use AnyIO consistently rather thanasyncio.to_threadorrun_in_executor(None), which use a different pool.
Safeguards
- Give each heavy sync dependency its own
CapacityLimiter, created in the lifespan. - Put timeouts on the underlying client. An outer
asyncio.timeoutorfail_afterdoesn't stop a thread. - Apply backpressure when capacity runs out:
503+Retry-After, not unbounded queueing. - Monitor limiter utilization and waiters, plus event-loop lag.
- Don't share non-thread-safe clients across threads. Check each library's documentation.
- Cancelling the awaiting coroutine doesn't stop the thread, which keeps its token and resources.
- Don't add a custom thread pool or executor until measurements justify one.
execution/threadpool.pyis reserved for that.
CPU-intensive work
- Threads don't parallelize CPU-bound pure-Python code, because of the GIL, and they slow down the event loop too.
- Classify each library by measurement: pure Python, native code that releases the GIL, or native code with internal threads (cap
OMP_NUM_THREADSand similar to the vCPU count). - CPU-heavy work outside the interactive path goes to AWS Lambda (stateless, reliably under 15 minutes, ≤ 10 GB / 6 vCPU, no GPU) or the ECS worker service (longer, stateful, bigger). An in-process process pool needs approval 🟡.
Background processing (Path B)
- Ingestion, indexing, batch evaluation, bulk embedding and long agent runs go API → SQS → ECS worker or Lambda, never through FastAPI
BackgroundTasks. The API returns202with a job ID. - Messages carry IDs and S3 keys, never documents.
- Every job is idempotent, with deterministic IDs and upserts, because SQS delivers at least once.
- Every queue has a DLQ with a bounded
maxReceiveCountand an alarm on DLQ depth. - Visibility timeout > processing time, extended with heartbeats for long jobs. For SQS → Lambda: visibility ≥ 6× the function timeout, with partial batch failures.
- Workers handle SIGTERM gracefully (stop receiving, finish or abandon within ECS
stopTimeout≤ 120 s), checkpoint long jobs and use ECS task scale-in protection while working.
Scaling (ECS Fargate)
- ECS Service Auto Scaling must be explicitly configured. It isn't instantaneous (expect minutes), and adding tasks doesn't fix event-loop blocking.
- The API service and worker service scale independently. Workers scale on backlog per task = visible messages ÷ running tasks, with target = acceptable latency ÷ average processing time.
- Keep every stage and stream under the ALB idle timeout. Send heartbeats on SSE.
Open decisions 🟡. Don't choose any of these without the user:
- Celery vs native SQS workers
- Lambda adoption, packaging and handler location
- Process pools
- Limiter sizes
- Standard vs FIFO queues
- Job-status storage
- Scaling metrics and targets
- Task sizing and Uvicorn worker count
- Step Functions
- Observability tooling
- Python runtime
6. Pydantic and structured-output rules (summary of PYDANTIC_STANDARDS.md) 🟢
Models
- Use Pydantic v2 for API schemas, settings, LLM outputs and internal contracts that cross a trust boundary. Every non-settings model inherits
CustomModelfromsrc/common/schemas/base.py. CustomModelaccepts field names and aliases, setsextra="forbid"and providesto_json_dict()=model_dump(mode="json"). Provider payloads override withextra="ignore", and settings useextra="ignore".- Prefer declarative constraints (
Field(ge/le/min_length/max_length/pattern),Literal,StrEnum,EmailStr,HttpUrl) over custom validators. Usemodel_validator(mode="after")for cross-field rules. Validators do no I/O and raiseValueErrorwith safe messages, neverassert. - Every timestamp uses
UtcDatetime(aware, normalized to UTC, serialized as ISO 8601Z). Naive datetimes are rejected, never assumed to be UTC. Usedatetime.now(timezone.utc), neverutcnow(). - Use discriminated unions (a
Literaltag +Field(discriminator=...)) for multi-shape results. Build aTypeAdapteronce per module for non-model types. - Separate
<Entity>Create/Update/Response/ internal models. ORM models aren't Pydantic. Apply PATCH withexclude_unset=True. Don't create a base class per domain. - Avoid double work.
response_modelre-validates returned data,model_copy(update=)andmodel_construct()skip validation, and model→dict→model hops are waste. - Serialize with
model_dump(mode="json")/model_dump_json(). Usejsonable_encoderonly for untyped mixed structures.
Settings
- One
BaseSettingsclass per domain or component, each with a unique env prefix,SecretStrsecrets, required values without defaults, nested models viaenv_nested_delimiter="__"and an@lru_cachegetter. There is no giant global class. - Never instantiate settings at import time. Validate the enabled components' settings in the lifespan at startup. In tests, override with
dependency_overrides,_env_file=Noneandcache_clear().
Structured LLM outputs
- Output has three validation levels, and all three are required:
- (1) Generation constraints. The provider's JSON schema, tool calls or JSON mode, where supported.
- (2) Pydantic validation.
model_validate_jsonagainst the full model. - (3) Application validation. Citations exist in the retrieved evidence, tools exist and are permitted, plans are consistent.
- Output models are separate from API models, and every field has a
description. Adapt the exported schema to the provider's capabilities. Don't assume every provider supports the same JSON Schema features, and keep enforcing stripped constraints in Pydantic. - Retries are bounded (3 by default 🟡), with separate budgets for validation and transport. Never retry forever. Never fill required fields with fabricated defaults. The final failure is a typed
StructuredOutputError. - Correction feedback sent to the LLM contains only locations, messages and types (
errors(include_input=False, include_url=False, include_context=False)), never secrets, prompts, raw inputs or internal IDs. - RAG. The LLM cites short evidence labels (
E1…) that the application maps to real chunks. Every citation must exist in this request's evidence. Self-reported confidence is a coarse enum and not a calibrated probability. Pydantic can't prove the evidence supports the claim, so faithfulness is measured inevaluation/. - Agents. Validate plans, then tool arguments before execution, then tool results, then state updates (
validate_assignment=True). Pydantic validation isn't authorization. Permissions, allowlists, sandboxing and confirmations are enforced inexecution/and in the tools. - Validation is sync but fast, so do it inline. Profile before offloading, and move bulk validation to the background path.
- Schema tests use synthetic data, a fake LLM client, no live calls and no credentials. Assert on error
typeandloc.
7. API, dependency, database and tooling rules (summary of API_CONVENTIONS.md) 🟢
Dependencies
- Checks that need I/O (exists, owned, active, unique) go in dependencies (
valid_<entity>_id), never in Pydantic validators and never repeated in each route. - Dependencies return typed objects (internal models,
Principal), not dicts. Publish reusable ones asAnnotatedaliases (ValidPost). Raise domain exceptions, notHTTPException. - Chain small dependencies. FastAPI caches each result per request, so shared ones (token parsing) run once. Create dependency factories at module level. Use
use_cache=Falseonly with a comment explaining why. - Prefer
async defdependencies, but only when nothing inside them blocks.defdependencies take thread-pool tokens. - Resource dependencies use
yield. Never hand a request-scoped session toBackgroundTasksor a job. Pass IDs instead. - Authorization is enforced server-side, in dependencies and
execution/. Secrets come from settings (SecretStr). RAG retrieval receives an explicit access scope, applied as a store filter before ranking. Agents receive their permitted tools in the execution context. Jobs carry the principal's ID and re-check permissions in the worker.
API surface
- RESTful resources live under
/api/v1(set inapi/v1/router.py). Use the same path-parameter name for the same entity everywhere so dependencies chain. Long operations return202plus a job resource. - Declare the response shape once. FastAPI re-validates and filters output against
response_model. Returning a rawResponseskips both and needs a profiled reason. BackgroundTasksis only for tiny best-effort side effects. If you'd page someone when a task is lost, it goes to SQS (Path B). No Celery, Arq or RQ without approval.- Validator messages become client-visible 422s, so keep them safe. Use
PydanticCustomErrorfor stable error types, and never raiseHTTPExceptionin validators. LLM-output models are never request bodies. - OpenAPI docs are off by default (
openapi_url=None) and enabled for listed environments through settings. Every route setsresponse_model,status_code,summary,description, andresponsesusingErrorResponse.
Database (engine and ORM 🟡)
- One shared
MetaDatawith an explicit constraint naming convention. Tables arelower_snake, singular and module-prefixed (rag_chunk,agent_run)._atcolumns aretimestamptzand map toUtcDatetime;_datecolumns are dates. - SQL-first: joins, filters, pagination, access-scope filters and JSON aggregation happen in SQL, and Pydantic validates the result. Use bound parameters only.
- Migrations 🟡 are static and reversible, have descriptive
date_slugnames, and follow expand-then-contract. Review autogenerated output, and never run migrations from the app lifespan on ECS.
Tests and tooling
- Use the async test client (
httpx.AsyncClient+ASGITransport) from day 0. It doesn't run the lifespan, so override client accessors or use a lifespan manager 🟡. Pick one async pytest plugin 🟡. - Replace collaborators with
app.dependency_overrideson the exact callable, andpoponly what you added. No live LLM, vector store, queue or AWS calls. - Ruff is the only linter and formatter (
ruff check --fix,ruff format). Its configuration, pre-commit hooks and a type checker are 🟡.
8. Guardrail rules (summary of GUARDRAILS.md) 🟢
- Guardrails check content risk (harmful content, PII, prompt injection, denied topics). They don't replace Pydantic validation, citation checks, authorization or evaluation, and a system prompt is not a guardrail.
- Checkpoints: G0 documents at ingestion, G1 request input, G2 untrusted retrieved context, G3 answer output (after L2/L3 validation, on user-visible text), G4 tool input (after authorization), G5 tool output, G6 final agent response. Background jobs are guarded too.
- All checks go through the shared
src/guardrails/service, and vendor clients live inproviders/guardrails/. Every check returns a typedGuardrailVerdict:passed,denied,suppressed,redacted🟡 ornot_evaluated. - An input denial stops the pipeline before retrieval, the LLM or the tool runs. An output suppression withholds the entire reply. Clients get a stable code and a generic message. Categories and scores stay in logs.
- Fail closed.
not_evaluatedrefuses the request (503+Retry-Afteron the API). Fail-open needs approval per surface. - Denials are never retried or regenerated around. Only transport failures of the guardrail call get a bounded retry. Never feed finding details to the model.
- Layer the enforcement points: in-app checks (required for our API and in-process RAG/agents), gateway policy for AgentCore Gateway targets 🟡, and ingestion-time checks. Each tool in the registry declares its enforcement point.
- Gateway (Bedrock Guardrails in AgentCore Policy) 🟡: Cedar
forbid … when guardrailsfor input,suppressOutputfor output. Data paths (context.input.*,context.output.*) must exist in the target schema, so generate schemas from Pydantic models. Interpret results by body: an MCP denial is HTTP 200 with JSON-RPC-32002. Roll out in log-only mode, check Region availability, and alarm on "could not be evaluated". - Guardrails AI 🟡: pinned PyPI validator packages only (no hub URIs, hub tokens or remote inference), used behind a checker. No
Guard.for_pydanticor guard-driven LLM calls, no embedded Flask server, and no content-rewriting on-fail actions without approval. - Thresholds, categories, mode and
policy_versionare settings or policy, not literals. Pin a published managed-guardrail version. New policies start in log-only mode. - Sync guardrail SDKs and ML validators are offloaded with a dedicated limiter and client timeouts. Deterministic validators run inline and first.
- Never log triggering text. Alarm on
not_evaluatedspikes and block-rate shifts. Measure false positives and negatives with paired harmful and benign datasets inevaluation/datasets/guardrails/, including indirect injection.
9. AI evaluation rules (summary of AI_EVALUATION.md) 🟢
- Evaluation is offline: never in the interactive request path, never blocking the event loop. Large runs use the worker path.
src/evaluation/= contracts, registry and adapters only, with lazy DeepEval/Ragas imports in an optional dependency group 🟡. Importingsrc.mainnever importsdeepevalorragas. Everything else lives inevaluation/.- Every result is an
EvaluationResultwhose value is numeric, binary, categorical or unavailable. Missing fields, judge errors and timeouts areunavailablewith a reason, never0. - Deterministic checks never use an LLM judge: JSON validity, schema compliance, enums, citation IDs, document references, retrieval IDs (P@K, R@K, Hit@K, MRR, NDCG@K), tool names and tool-argument schemas.
- Starter suites: RAG = faithfulness, answer relevancy, contextual precision, contextual recall; agents = task completion, tool correctness, plus a deterministic end-state check. Add more as workflows grow.
- **
…(truncated)
