Software AI Integration

AI-Augmented Product Engineering

Integrate LLMs and AI capabilities into production applications with clean architecture, cost discipline, and reliable user experience.

Quick Reference

When to Use This Skill

• Adding AI-powered features to an existing product (chat, generation, suggestions)
• Building streaming UI for LLM responses in web or mobile applications
• Implementing structured output with schema validation from LLM calls
• Designing cost control and caching strategies for AI features
• Building multi-provider fallback and model routing logic
• Implementing guardrails, content moderation, and safety layers
• Choosing AI UX patterns (loading states, regenerate, feedback, attribution)

When NOT to Use This Skill

• LLM lifecycle management (fine-tuning, deployment, monitoring) → ai-llm
• Agent system architecture and orchestration → ai-agents
• Prompt engineering techniques and patterns → ai-prompt-engineering
• RAG system architecture (indexing, retrieval, chunking) → ai-rag
• ML model training and data science → ai-ml-data-science
• MLOps and model serving infrastructure → ai-mlops
• Building MCP servers and tool protocols → agents-mcp

When NOT to Add AI At All

Not every "AI feature" request should become one. Apply this filter before the decision tree below:

• Deterministic logic solves it. If the mapping from input to output is a rule, a regex, a lookup table, or a small classifier that can be trained offline, an LLM call adds latency, cost, and non-determinism for no accuracy gain. Reach for an LLM when the input space is open-ended natural language or the task requires judgment a rule set cannot encode.
• The eval bar cannot be met. If nobody can articulate what "correct" looks like well enough to write 20-50 test cases with expected outputs, the team cannot tell if the feature works, regresses, or is safe to ship. Build the eval set before writing the first prompt (see ai-evals) — "we'll know it when we see it" is not a launch gate.
• The failure mode is unacceptable and unrecoverable. High-stakes one-shot actions (irreversible financial transfers, medical dosing, legal filings) need a human-in-the-loop confirmation step even when the model is highly accurate; if the product cannot afford any rate of wrong output and cannot insert a checkpoint, do not automate that step with an LLM.
• A cheaper, boring solution already ships the value. Autocomplete from historical data, templated responses, or a search index often satisfy the underlying user need without a model call. Prototype the non-AI version first if it is cheap to build — it is also the fallback path required by the guardrail rules below.

Build vs. Buy

The recurring judgment call: build the thin layer that encodes your product's specific logic (prompts, schemas, eval criteria, business rules); buy the generic infrastructure (streaming plumbing, moderation classifiers, tracing storage) that every AI product needs and that a vendor has already hardened against edge cases you have not seen yet.

Workflow

1. Classify the feature shape: chat, generation, extraction, search, or agent-adjacent UX.
2. Confirm the product boundary and route architecture-heavy or retrieval-heavy work to the adjacent skill when needed.
3. Pick the primary pattern from the decision tree, then define the request contract, latency target, and safety checks.
4. Apply the relevant implementation guidance for streaming, structure, cost, and guardrails.
5. Verify current provider capabilities with the navigation references and fact-checking rules before final recommendations.

ASCII Flow

AI feature request
  -> Classify: chat, generation, extraction, search, or agent-adjacent UX
  -> Route architecture, RAG, or agent-heavy work to companion skills
  -> Define typed request and response contract
  -> Choose provider, streaming, safety, and cost controls
  -> Implement product UX states and observability
  -> Verify provider behavior and eval evidence

Decision Tree

What kind of AI feature?
├─ Chat / conversational UI
│  ├─ Web app → Vercel AI SDK (useChat + streamText)
│  │  ├─ Conversation history → Store in DB, not just client state
│  │  ├─ Streaming markdown → Progressive render with remark/rehype
│  │  └─ Multi-turn with tools → Tool results in message history
│  └─ Mobile / native → Direct SSE consumption + custom UI
├─ Content generation ("write for me")
│  ├─ Short-form (titles, descriptions) → Single generation + schema
│  └─ Long-form (articles, reports)
│     └─ Draft → Review → Edit → Apply pattern with undo support
├─ Inline suggestions (autocomplete)
│  ├─ Latency-critical → Use fastest model (Haiku-class)
│  ├─ UI pattern → Ghost text, accept with Tab, dismiss with Esc
│  └─ Trigger → Debounce input (300-500ms), cancel in-flight requests
├─ Data extraction / classification
│  ├─ Structured output with Zod schema validation
│  ├─ Batch processing → Queue + worker pattern
│  └─ Confidence scores → Include in schema, filter by threshold
├─ Search / Q&A over content
│  └─ RAG pattern (see ai-rag) + this skill for product integration
└─ Agent features (multi-step autonomous)
   └─ ai-agents for architecture, this skill for product UX integration

Framework And Gateway Choice

The repo source list highlighted a practical split that belongs in this skill:

Rules:

• Do not reach for LangGraph or CrewAI just because a feature uses a model. Most product AI flows are still request/response plus tools.
• Use a gateway only when multiple products, models, or policy layers need a shared control point.
• If the problem is agent architecture first, route to ai-agents. If the implementation is specifically bots or LangGraph, use ai-bot-builder. If it is product integration first, stay here.

Streaming Architecture

Never buffer a full LLM response and then send it. Always stream to the user.

• Server-side: use SDK streaming functions (streamText, streamObject in AI SDK, or native SDK streaming). Pipe the stream directly to the HTTP response as Server-Sent Events or a ReadableStream.
• Client-side: consume the ReadableStream, render text progressively as tokens arrive. For structured output, handle partial JSON gracefully — do not parse until a complete object boundary.
• Error handling in streams: the stream can error mid-response. Handle connection drops, timeouts, and model errors. Surface errors to the user inline (not as a separate error page). Provide a "retry" action that preserves conversation context.
• Cancellation: support "stop generating" — abort the fetch on the client, which should propagate to cancel the upstream API call. Do not charge for tokens you did not use.
• Backpressure: if the client cannot consume tokens fast enough (slow rendering, network congestion), the server should respect backpressure rather than buffering unboundedly.

Structured Output Patterns

When you need the LLM to return data, not prose.

• Structured output (strict/schema-constrained mode) vs. tool_use / function calling: both achieve near-perfect schema adherence on current frontier-tier models, but they answer different questions. Use structured output when there is no decision to make — the model's terminal answer must simply match a shape (e.g., "extract these fields"). Use tool/function calling when the model must decide whether and which action to take, and the schema is the arguments to that action. Prefer either over legacy unconstrained JSON mode or regex-parsed prose — schema-constrained modes are the current baseline, not an upgrade path.
• Zod schemas: define your expected output shape with Zod. Use with AI SDK generateObject / streamObject for automatic validation. Zod gives you TypeScript types and runtime validation from one definition.
• Fallback parsing: for models that occasionally break schema, implement graceful fallback — attempt parse, if it fails try to extract partial data, log the failure for monitoring, and retry once with a more explicit prompt.
• Discriminated unions: when the AI can return different types of responses (e.g., "answer" vs. "clarification_needed" vs. "refused"), use discriminated union schemas. The type field tells your code which branch to handle.
• Streaming structured output: streamObject delivers partial objects as they generate. Use for progressive UI updates (show fields as they arrive), but validate the complete object before persisting.

Conversation & Context Management

• Message history storage: store in a database, not just client state. Users expect conversations to persist across sessions and devices. Schema: { id, conversationId, role, content, toolCalls, toolResults, createdAt }.
• Context window management: LLMs have finite context. Choose a strategy: sliding window (drop oldest messages), summarization (compress old messages into a summary), or hard truncation with a warning. Track token usage per conversation.
• System prompts: version them in code, do not hardcode strings. System prompts are product logic — they should go through code review, have tests, and be deployable independently when possible.
• Multi-turn with tools: when the model calls a tool, execute it and include the result in the message history. The model needs to see previous tool results to maintain coherent multi-step reasoning.
• Conversation branching: when a user "regenerates" a response, you are branching the conversation. Decide: replace the last message (simpler) or maintain a tree of branches (more flexible, more complex).

Cost Control

See references/rollout-and-observability.md for cost dashboards, eval loops, and feature-flag rollout patterns.

Worked example: is caching worth it?

Exact per-token prices change often — pull current numbers from the provider pricing page before using this in a real budget. The method below is what matters and stays stable: prompt caching only pays for itself once a cached prefix is reused enough times to amortize the cache-write premium.

Providers commonly price cache writes at a premium over a normal input token (e.g., roughly 1.25x for a short-TTL cache, ~2x for a longer-TTL cache) and cache reads at a steep discount off the normal input price (commonly ~0.1x, i.e. a ~90% discount). Given:

• P = normal input token price
• w = cache-write multiplier (e.g., 1.25)
• r = cache-read multiplier (e.g., 0.1)
• N = number of times the cached prefix is reused before it expires or changes

Break-even reuse count N* = (w - r) / (1 - r). With w = 1.25, r = 0.1: N* = 1.15 / 0.9 ≈ 1.28 — so caching a stable system prompt or long tool-definition block pays for itself after roughly the second reuse, not after dozens of reuses as intuition might suggest. This is why caching large, stable prefixes (system prompts, tool schemas, few-shot examples, retrieved document sets reused across a session) is close to a free win in most chat and agent architectures — the failure mode is forgetting to structure prompts so the stable part is a shared, byte-identical prefix, which breaks cache hits.

Apply the same break-even logic before adopting batch-API discounts (commonly ~50% off both directions) versus real-time calls: batch trades latency for cost, so it is only a substitute for interactive features, not a default.

AI UX Patterns

Guardrails & Safety

Multi-Provider Strategy

Abstract provider calls behind a single AIProvider interface (AI SDK's provider pattern achieves this). Never swap models for all users at once — use feature flags and circuit breakers.

See references/rollout-and-observability.md for full rollout and eval loop patterns.

Do / Avoid

Known Traps

• Treating graceful degradation as a diagram-only requirement instead of proving the fallback path under real provider failure.
• Keeping conversation history only in browser or mobile client state, then discovering regeneration, resume, and support workflows have no canonical record.
• Assuming schema-shaped output is safe because the model usually behaves, without validating every response before persistence or side effects.
• Swapping models or providers behind the same endpoint without re-running prompt, latency, and evaluation gates.
• Capturing user feedback but never routing it into prompt, model, or retrieval evaluation loops.

Common Anti-Patterns

Scenarios

Recipes keyed to symptoms or integration moments. Each lists the shortest path to a working, production-safe implementation.

S1 — Streaming chat with citations

1. Define the server endpoint using streamText (AI SDK) with the chosen provider.
2. Include a citations tool or instruct the model to embed [source:N] markers in prose.
3. Pipe the ReadableStream to the HTTP response as SSE; never buffer the full response.
4. On the client, use useChat to render tokens progressively; parse [source:N] markers into inline links.
5. Handle mid-stream errors with a visible inline retry control that preserves conversation context.
6. Log each completed turn with { conversationId, model, inputTokens, outputTokens, latencyMs } for cost tracking.

S2 — Structured extraction with Zod + retry-on-schema-fail

1. Define the target shape as a Zod schema; use generateObject (AI SDK) to bind schema to the model call.
2. On ZodError, log the raw response and retry once with a more explicit prompt that names the failing field.
3. Use discriminated unions ({ type: "success" | "parse_error" | "refused" }) to handle all branches.
4. Validate the complete object before any persistence or side-effect call.
5. Track schema-fail rate per endpoint; alert if it exceeds 1% — signals prompt drift or model regression.

S3 — Tool-calling agent with indirect-prompt-injection guard

1. Define tools with minimal, read-only side effects; server owns writes, model only requests them.
2. Add an input-sanitization step: strip \nAssistant:, \nHuman:, and instruction-override patterns before injecting user input into prompts.
3. After each tool call, verify the returned result matches the declared tool schema before passing back to the model.
4. Audit-log every tool call with { tool, args, result, conversationId } for post-incident tracing.
5. Add an output filter that refuses responses containing suspicious lateral instructions or role-change attempts.

S4 — Multi-provider routing on rate-limit

1. Abstract provider calls behind a single AIProvider interface; swap implementations without changing call sites.
2. Implement a circuit breaker: after N 429 responses in M seconds, route to the fallback provider.
3. Maintain per-model prompt variants; run quality gates before switching traffic to an alternate model.
4. Alert on circuit-breaker activation; log which provider is active per request for cost attribution.
5. Add a --dry-run mode in staging that exercises the fallback path without real traffic.

S5 — RAG with stale-cache invalidation

1. Cache retrieval results with a content_hash and retrieved_at timestamp in the cache key.
2. On each query, compare the source document's updated_at against cached retrieved_at; evict on mismatch.
3. Build a background job to re-index documents when source content changes; see ai-rag for chunking strategy.
4. Return { answer, sources[{ id, title, url, retrieved_at }] } to the UI for attribution.
5. Track cache-hit rate and stale-eviction rate; tune TTL to balance freshness against provider cost.

Navigation

References

• references/product-integration-patterns.md — streaming UX, structured output, persistence, and degraded-mode patterns
• references/framework-and-gateway-patterns.md — app SDK vs copilot framework vs agent framework vs gateway choice
• references/rollout-and-observability.md — cost controls, feature flags, evaluation loops, and operations
• references/prompt-injection-and-ai-act.md — direct/indirect prompt injection taxonomy, multi-hop agent attacks, defenses, model version pin strategy, and the full EU AI Act compliance treatment (obligation timeline, prohibited practices, provider/operator boundary, Article 50 disclosure, high-risk Annex III, Article 12 logging, enforcement and fines)
• data/sources.json — official SDK, provider, and safety/eval sources

Related Skills

• ai-llm — LLM lifecycle, fine-tuning, and deployment
• ai-agents — Agent system architecture and orchestration
• ai-prompt-engineering — Prompt design techniques and patterns
• ai-rag — RAG system architecture and retrieval patterns
• software-backend — Backend service patterns
• software-frontend — Frontend application development
• software-realtime — Real-time communication and streaming
• software-security-appsec — Application security and threat modeling

Freshness Protocol

AI integration tooling changes rapidly. Freshness-check before answering questions about SDKs, model capabilities, or provider-specific patterns.

Triggers: SDK version questions, "is X still recommended?", streaming API changes, provider capability changes, new model releases.

Process: start from data/sources.json, run a targeted web search, check SDK changelogs (Vercel AI SDK, Anthropic SDK, and OpenAI SDK all release frequently with breaking changes).

Regulatory Traps

Verify current obligation timelines at official EU AI Act sources at use-time — the Digital Omnibus on AI (political agreement reached 7 May 2026, EU Parliament endorsed 16 June 2026, Council final sign-off 29 June 2026) pushed the Annex III high-risk deadline from 2 August 2026 to 2 December 2027; confirm formal Official Journal publication and effective date at eur-lex.europa.eu before relying on either date as of 2026-07-11.

Indirect prompt injection is the primary exploit path: attacker-controlled data in retrieved documents, tool outputs, or web results overrides system instructions. Mitigations: isolate retrieved content with structural tags, enforce least-privilege tool scopes, validate model output before write/send operations, and test with adversarial documents in CI. Defense must be architectural — model-side mitigations reduce but do not eliminate risk.

See references/prompt-injection-and-ai-act.md for full obligation timelines and injection defence patterns.

Fact-Checking

• Known bugs, regressions, framework/compiler/runtime footguns, and version-specific crash or workaround guidance must be verified against current primary web sources before being treated as current fact.
• Use web search/web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
• Prefer primary sources; report source links and dates for volatile information.
• If web access is unavailable, state the limitation and mark guidance as unverified.

Learnings Loop

Before applying this skill on a non-trivial task, read learnings.consolidated.md in this directory (and learnings.md if present).

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.