AI App Architecture

AI Application Architecture

Operating contract

Inputs

Outputs

• Produce: architecture decision record, component and trust boundaries, data flow, build-versus-buy decision, and evolution path.

Capability and permission boundaries

Default to read-only analysis. Read only scoped records; redact secrets and regulated data. Writes, execution, network calls, production configuration, customer communication, billing changes, and delegation require explicit authority and an identified owner. Never widen tenant, time-window, or system scope implicitly.

Degraded mode

When required telemetry, evidence, execution, network access, or write authority is unavailable, return a partial result with each unassessed item labelled, preserve the safest existing state, and state the evidence or approval needed to continue. Never convert missing evidence into a pass.

Decision rules

Anti-Patterns

• Treating absent evidence as success. Fix: mark the check unassessed and name the missing source.
• Expanding one tenant or workflow to all tenants. Fix: enforce supplied scope at every query and action.
• Performing a production write during analysis. Fix: emit a reviewed change plan until authority is explicit.
• Reporting a metric without population, window, or source. Fix: attach all three.
• Hiding a failed threshold inside an average. Fix: report failure slices and the remediation owner.

Acknowledgement: Shared by Peter Bamuhigire, techguypeter.com, +256 784 464178.

Use When

• Use when designing or building AI-powered application systems — choosing architecture style, selecting components, structuring the AI stack, making build-vs-buy decisions, and planning multi-tenant AI module gating

Evidence Produced

References

• Use the links and companion skills already referenced in this file when deeper context is needed.
• references/practical-ai-engineering.md - evaluation-first AI engineering, RAG quality checks, agentic workflow controls, guardrails, telemetry, and cost discipline.
• references/ai-transformation-operating-model.md - operating-model, adoption, governance, human-centred design, and sustainability gates distilled from supplied AI transformation source material.

Overview

AI-powered apps are built on top of foundation models via APIs. You are NOT training models — you are orchestrating them. Your value lies in the application layer: context construction, prompt engineering, retrieval, guardrails, and user experience.

Core principle: Start with the simplest architecture that works. Evolve deliberately.

Premier Agency Standard

Design AI systems for measurable economic value, production reliability, and long-term maintainability. Every serious AI application must connect model behavior to a business workflow, revenue lever, cost reduction, risk reduction, or service-quality improvement.

Before selecting a model or framework, define:

• Business outcome: the operational metric the AI feature should improve.
• User decision/action: what the user or system will do differently because AI exists.
• Data advantage: which proprietary, local, domain, or workflow data improves the result.
• Failure cost: what happens when the model is wrong, slow, unavailable, biased, or expensive.
• Evaluation target: the minimum acceptable quality, latency, cost, and safety threshold.
• Operating owner: who monitors, tunes, approves, and maintains the feature after launch.

Reject AI features that cannot state their economic value or decision impact.

Architecture Styles (Choose One to Start)

Default path: Wrap -> RAG -> deterministic workflow -> agents. Only fine-tune when prompts, retrieval, examples, routing, and workflow design cannot meet the target.

The AI Application Stack

┌──────────────────────────────────┐
│   User Interface (web/mobile)    │
├──────────────────────────────────┤
│   Input Guardrail                │  ← block PII, prompt injection, off-topic
│   Router / Intent Classifier     │  ← route to right model/solution
│   Context Builder (RAG / Tools)  │  ← feature engineering for AI
│   Model Gateway                  │  ← unified API wrapper, key mgmt, fallbacks
│   LLM API (OpenAI/Codex/Gemini) │
│   Output Guardrail               │  ← catch toxicity, format failures, PII
│   Cache Layer                    │  ← exact + semantic caching
│   Streaming Handler              │  ← MANDATORY — never block on full generation
├──────────────────────────────────┤
│   Token Ledger (MANDATORY)       │  ← log every call: tenant_id, user_id, tokens
│   AI Module Gate (OFF by default)│  ← per-tenant enable/disable
└──────────────────────────────────┘

Component Responsibilities

Input Guardrail

• Detect and mask PII before sending to external APIs
• Block prompt injection patterns
• Enforce topic restrictions (domain scope)
• Tools: Meta Purple Llama, NVIDIA NeMo Guardrails, OpenAI Moderation API

Router

• Classify intent → route to right model or solution
• Send simple queries to cheaper models (BERT, GPT-mini)
• Detect out-of-scope queries before wasting API calls
• Detect ambiguous queries → ask for clarification
• Pattern: routing → retrieval → generation → scoring

Context Builder

• Context construction = feature engineering for AI
• Retrieve relevant chunks (RAG), live data (APIs), user profile
• This is where most quality improvement happens — invest here
• Version prompts, retrieval settings, chunking rules, schemas, and tool definitions as production configuration
• Separate tenant/user context from global knowledge to prevent cross-client data leakage

Model Gateway

• Centralises all LLM provider calls (OpenAI, Anthropic, Google, self-hosted)
• Centralises: API key management, rate limiting, logging, fallback policies
• Enables swapping providers without touching application code
• Tools: Portkey AI Gateway, MLflow AI Gateway, Kong
• Expose common controls: request id, tenant id, model id, prompt version, timeout, retry policy, budget class, and safety profile

Output Guardrail

• Catch format failures (invalid JSON/schema) → retry automatically
• Catch hallucinations, toxic content, brand-risk responses
• Fall back to human operators for sensitive/tricky queries
• Note: streaming mode complicates output guardrails — plan for partial responses

Semantic Cache

• Exact cache: identical queries → return stored response
• Semantic cache: similar queries → return stored response (use embedding similarity)
• Cache at vector search level too (expensive embedding calls)
• Warning: improper cache can leak user-specific data — use tenant-scoped cache keys

Architecture Evolution Pattern

Step 1 (Baseline):   Query → Model API → Response
Step 2 (Context):    Query → Retriever → [Context + Query] → Model → Response
Step 3 (Guardrails): Input Guard → Context → Model → Output Guard → Response
Step 4 (Router):     Router → [Intent-specific path] → Model(s) → Response
Step 5 (Cache):      Router → Cache → [miss: full pipeline] → Cache Store
Step 6 (Agents):     Router → Agent Loop [Plan → Tools → Reflect] → Response

Add each layer only when its absence is causing a real problem.

For production AI features, load references/practical-ai-engineering.md before finalising the architecture. It adds evaluation, RAG, agent, safety, fallback, telemetry, and cost-control gates.

Production AI Platform Requirements

Workflow vs Agent Decision

• Use a deterministic workflow when the process steps are known, regulated, auditable, or cost-sensitive.
• Use an agent when the path genuinely depends on intermediate observations, tool results, or open-ended planning.
• Keep agents on a short leash: bounded tool set, max steps, scoped memory, explicit approvals, and rollback-safe actions.
• Do not use agents for simple summarization, extraction, classification, or transformation.

Data and ML System Design Checks

• Define online/offline data sources, freshness requirements, ownership, quality checks, and missing-data behavior.
• Choose metrics that reflect the business objective, not only model accuracy.
• Plan for distribution shift: seasonality, new user behavior, changing regulations, platform algorithm changes, and language mix.
• Store enough inputs, outputs, versions, and feedback to debug regressions without storing unnecessary sensitive data.
• Separate training/evaluation data from production data where supervised learning or fine-tuning is used.

Build vs Buy Decision

Rule: API wrap first. Justify self-hosting with actual cost/compliance numbers.

AI Module Gating (MANDATORY in SaaS)

Every AI feature MUST be gated. AI costs real money per token.

-- Schema: AI module per tenant
CREATE TABLE tenant_ai_config (
  tenant_id     INT PRIMARY KEY,
  ai_enabled    BOOLEAN DEFAULT FALSE,   -- OFF by default
  monthly_budget_usd DECIMAL(10,2),      -- null = unlimited
  budget_alert_pct   INT DEFAULT 80,     -- alert at 80% of budget
  plan_name     VARCHAR(50),             -- 'basic', 'pro', 'enterprise'
  enabled_at    TIMESTAMP,
  created_at    TIMESTAMP DEFAULT NOW()
);

Enforcement: Every AI endpoint checks tenant_ai_config.ai_enabled before processing. Return 402 Payment Required if disabled.

Token Ledger (MANDATORY)

Log every AI API call for billing, debugging, and cost visibility.

CREATE TABLE ai_token_usage (
  id            BIGINT AUTO_INCREMENT PRIMARY KEY,
  tenant_id     INT NOT NULL,
  user_id       INT NOT NULL,
  feature_name  VARCHAR(100),            -- 'invoice_analysis', 'report_summary'
  model         VARCHAR(50),             -- 'gpt-4o', 'Codex-3-sonnet'
  tokens_in     INT NOT NULL,
  tokens_out    INT NOT NULL,
  cost_usd      DECIMAL(10,6),           -- calculated at log time
  latency_ms    INT,
  created_at    TIMESTAMP DEFAULT NOW(),
  INDEX idx_tenant_date (tenant_id, created_at),
  INDEX idx_user_date (user_id, created_at)
);

-- Usage by tenant (for invoicing)
SELECT tenant_id,
       SUM(tokens_in + tokens_out) AS total_tokens,
       SUM(cost_usd) AS total_cost_usd,
       DATE_FORMAT(created_at, '%Y-%m') AS month
FROM ai_token_usage
GROUP BY tenant_id, month;

-- Usage by user (for analytics)
SELECT user_id, feature_name,
       SUM(tokens_in + tokens_out) AS tokens,
       COUNT(*) AS calls
FROM ai_token_usage
WHERE tenant_id = ? AND created_at > DATE_SUB(NOW(), INTERVAL 30 DAY)
GROUP BY user_id, feature_name;

Quota Enforcement

function checkAiQuota(int $tenantId): void {
    $config = TenantAiConfig::find($tenantId);

    if (!$config || !$config->ai_enabled) {
        throw new AiModuleDisabledException('AI module not enabled for this account.');
    }

    if ($config->monthly_budget_usd !== null) {
        $spent = AiTokenUsage::currentMonthCost($tenantId);
        if ($spent >= $config->monthly_budget_usd) {
            throw new AiBudgetExceededException('Monthly AI budget reached.');
        }
        if ($spent >= $config->monthly_budget_usd * ($config->budget_alert_pct / 100)) {
            notifyTenantBudgetAlert($tenantId, $spent, $config->monthly_budget_usd);
        }
    }
}

Infrastructure Options

Anti-Patterns

• No module gating — every user can trigger AI calls, destroying your margins
• No token logging — you cannot invoice clients or debug runaway costs
• Orchestrator too early — LangChain/LlamaIndex before you understand your pipeline adds complexity
• Fine-tuning first — always try prompt engineering and RAG before fine-tuning
• Blocking on full generation — always stream tokens to the user immediately
• Hard-coded system prompts — make prompts configurable, not hardcoded in code

Sources

Chip Huyen — AI Engineering (2025); David Spuler — Generative AI Applications (2024); Andrea De Mauro — AI Applications Made Easy (2024)

Consolidated Child References

• Load references/routing.md to map retired AI child skill slugs to their reference modules.