LLM Integration
Patterns for integrating LLMs into production applications: tool use, streaming, local inference, and fine-tuning. Each category has individual rule files in rules/ loaded on-demand.
Quick Reference
| Category |
Rules |
Impact |
When to Use |
| Function Calling |
3 |
CRITICAL |
Tool definitions, parallel execution, input validation |
| Streaming |
3 |
HIGH |
SSE endpoints, structured streaming, backpressure handling |
| Local Inference |
3 |
HIGH |
Ollama setup, model selection, GPU optimization |
| Fine-Tuning |
3 |
HIGH |
LoRA/QLoRA training, dataset preparation, evaluation |
| Context Optimization |
2 |
HIGH |
Window management, compression, caching, budget scaling |
| Evaluation |
2 |
HIGH |
LLM-as-judge, RAGAS metrics, quality gates, benchmarks |
| Prompt Engineering |
4 |
HIGH |
CoT, few-shot, versioning, DSPy optimization, ReAct, cost optimization |
Total: 20 rules across 7 categories
Quick Start
# Function calling: strict mode tool definition
tools = [{
"type": "function",
"function": {
"name": "search_documents",
"description": "Search knowledge base",
"strict": True,
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search query"},
"limit": {"type": "integer", "description": "Max results"}
},
"required": ["query", "limit"],
"additionalProperties": False
}
}
}]
# Streaming: SSE endpoint with FastAPI
@app.get("/chat/stream")
async def stream_chat(prompt: str):
async def generate():
async for token in async_stream(prompt):
yield {"event": "token", "data": token}
yield {"event": "done", "data": ""}
return EventSourceResponse(generate())
# Local inference: Ollama with LangChain
llm = ChatOllama(
model="deepseek-r1:70b",
base_url="http://localhost:11434",
temperature=0.0,
num_ctx=32768,
)
# Fine-tuning: QLoRA with Unsloth
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/Meta-Llama-3.1-8B",
max_seq_length=2048, load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(model, r=16, lora_alpha=32)
Function Calling
Enable LLMs to use external tools and return structured data. Use strict mode schemas (2026 best practice) for reliability. On Claude keep the tools array stable across turns so the prompt cache holds; add tools mid-session with an inline tool_addition (beta inline-tools-2026-09-15, Claude API only, available on Sonnet 5.5 but not on Sonnet 5) instead of swapping subsets (https://platform.claude.com/docs/en/build-with-claude/mid-conversation-system-messages#mid-conversation-tool-changes). For large tool sets, use a non-deferred tool search tool with defer_loading: true on the rest, never defer_loading alone (a request cannot have every tool deferred). Forced tool_choice (any or a named tool) returns a 400 on Sonnet 5.5, Opus 5.5, Fable 5.1 and Mythos 5.1; use auto with strict: true tools. Validate all inputs with Pydantic/Zod, and return errors as tool results.
calling-tool-definition.md -- Strict mode schemas, OpenAI/Anthropic formats, LangChain binding
calling-parallel.md -- Parallel tool execution, asyncio.gather, strict mode constraints
calling-validation.md -- Input validation, error handling, tool execution loops
Streaming
Deliver LLM responses in real-time for better UX. Use SSE for web, WebSocket for bidirectional. Handle backpressure with bounded queues.
streaming-sse.md -- FastAPI SSE endpoints, frontend consumers, async iterators
streaming-structured.md -- Streaming with tool calls, partial JSON parsing, chunk accumulation
streaming-backpressure.md -- Backpressure handling, bounded buffers, cancellation
Local Inference
Run LLMs locally with Ollama for cost savings (93% vs cloud), privacy, and offline development. Pre-warm models, use provider factory for cloud/local switching.
local-ollama-setup.md -- Installation, model pulling, environment configuration
local-model-selection.md -- Model comparison by task, hardware profiles, quantization
local-gpu-optimization.md -- Apple Silicon tuning, keep-alive, CI integration
Fine-Tuning
Customize LLMs with parameter-efficient techniques. Fine-tune ONLY after exhausting prompt engineering and RAG. Requires 1000+ quality examples.
tuning-lora.md -- LoRA/QLoRA configuration, Unsloth training, adapter merging
tuning-dataset-prep.md -- Synthetic data generation, quality validation, deduplication
tuning-evaluation.md -- DPO alignment, evaluation metrics, anti-patterns
Context Optimization
Manage context windows, compression, and attention-aware positioning. Optimize for tokens-per-task.
context-window-management.md -- Five-layer architecture, anchored summarization, compression triggers
context-caching.md -- Just-in-time loading, budget scaling, probe evaluation, CC 2.1.32+
Evaluation
Evaluate LLM outputs with multi-dimension scoring, quality gates, and benchmarks.
evaluation-metrics.md -- LLM-as-judge, RAGAS metrics, hallucination detection
evaluation-benchmarks.md -- Quality gates, batch evaluation, pairwise comparison
Prompt Engineering
Design, version, and optimize prompts for production LLM applications.
prompt-design.md -- Chain-of-Thought, few-shot learning, pattern selection guide
prompt-testing.md -- Langfuse versioning, DSPy optimization, A/B testing, self-consistency
prompt-react-pattern.md -- ReAct loop for tool-using agents, thought-action-observation format
prompt-optimization.md -- Token reduction, cost optimization, model tiering, prompt spec format
Upstream coverage (do not restate)
These topics are covered by their vendors' own documentation. This skill points
at them instead of teaching them; the rules above keep only our floors, ceilings
and scars. Our delta on all of it is in references/ork-delta.md.
| Topic |
First-party source |
| Strict-mode tool schemas, structured outputs |
https://platform.openai.com/docs/guides/function-calling |
Anthropic input_schema / tool_use |
https://docs.claude.com/en/docs/agents-and-tools/tool-use/overview |
| SSE client mechanics, reconnection, cancellation |
https://developer.mozilla.org/en-US/docs/Web/API/EventSource |
| LoRA / QLoRA config, target modules, adapter merging |
https://huggingface.co/docs/peft/developer_guides/lora |
| Unsloth training loop, 4-bit loading |
https://docs.unsloth.ai/get-started/fine-tuning-llms-guide |
| DPO, preference pairs, beta tuning, RLHF comparison |
https://huggingface.co/docs/trl/dpo_trainer |
| SFT dataset formats (Alpaca, ChatML) |
https://huggingface.co/docs/trl/sft_trainer |
| Embedding similarity for dataset dedup |
https://sbert.net/ |
| Fine-tune vs prompt vs RAG decision framework |
https://platform.openai.com/docs/guides/optimizing-llm-accuracy |
| Vendor token pricing (never hardcode it here) |
https://platform.openai.com/docs/pricing |
Supporting Files
references/ork-delta.md -- our delta: scars, house ceilings, retired-file provenance
references/model-selection.md -- local model comparison by task and hardware
scripts/create-lora-config.md -- LoRA config scaffold with auto-detected model type
Key Decisions
| Decision |
Recommendation |
| Tool schema mode |
strict: true (2026 best practice) |
| Tool count |
Load roughly 5-15 up front; defer the rest behind a non-deferred tool search tool |
| Streaming protocol |
SSE for web, WebSocket for bidirectional |
| Buffer size |
50-200 tokens |
| Local model (reasoning) |
deepseek-r1:70b |
| Local model (coding) |
qwen2.5-coder:32b |
| Fine-tuning approach |
LoRA/QLoRA (try prompting first) |
| LoRA rank |
16-64 typical |
| Training epochs |
1-3 (more risks overfitting) |
| Context compression |
Anchored iterative (60-80%) |
| Compress trigger |
70% utilization, target 50% |
| Judge model |
claude-haiku-5-5 (cost tier), gpt-5.5, or gemini-3.8-flash (Google's GA general-purpose model and a third vendor for the different-model rule; price lives in models.vocab.json, not here) |
| Quality threshold |
0.7 production, 0.6 drafts |
| Few-shot examples |
3-5 diverse, representative |
| Prompt versioning |
Langfuse with labels |
| Auto-optimization |
DSPy MIPROv2 |
Related Skills
ork-extras:rag-retrieval -- Embedding patterns, when RAG is better than fine-tuning
agent-loops -- Multi-step tool use with reasoning
llm-evaluation -- Evaluate fine-tuned and local models
langfuse-observability -- Track training experiments
Capability Details
function-calling
Keywords: tool, function, define tool, tool schema, function schema, strict mode, parallel tools
Solves:
- Define tools with clear descriptions and strict schemas
- Execute tool calls in parallel with asyncio.gather
- Validate inputs and handle errors in tool execution loops
streaming
Keywords: streaming, SSE, Server-Sent Events, real-time, backpressure, token stream
Solves:
- Stream LLM tokens via SSE endpoints
- Handle tool calls within streams
- Manage backpressure with bounded queues
local-inference
Keywords: Ollama, local, self-hosted, model selection, GPU, Apple Silicon
Solves:
- Set up Ollama for local LLM inference
- Select models based on task and hardware
- Optimize GPU usage and CI integration
fine-tuning
Keywords: LoRA, QLoRA, fine-tune, DPO, synthetic data, PEFT, alignment
Solves:
- Configure LoRA/QLoRA for parameter-efficient training
- Generate and validate synthetic training data
- Align models with DPO and evaluate results
1---2name: llm-integration3license: MIT4compatibility: "Claude Code 2.1.277+."5description: LLM integration patterns for function calling, streaming responses, local inference with Ollama, and fine-tuning customization. Use when implementing tool use, SSE streaming, local model deployment, LoRA/QLoRA fine-tuning, or multi-provider LLM APIs.6user-invocable: false7metadata:8 source: https://skillmd.com/skills/yonatangross/llm-integration9 owner-agent: "llm-integrator"10 category: mcp-enhancement11 version: "2.0.0"12 author: "OrchestKit"13 complexity: "medium"14 tags: "llm, function-calling, streaming, ollama, fine-tuning, lora, tool-use, local-inference"15allowed-tools: "Read Glob Grep WebFetch WebSearch"16---1718# LLM Integration1920Patterns for integrating LLMs into production applications: tool use, streaming, local inference, and fine-tuning. Each category has individual rule files in `rules/` loaded on-demand.2122## Quick Reference2324| Category | Rules | Impact | When to Use |25|----------|-------|--------|-------------|26| [Function Calling](#function-calling) | 3 | CRITICAL | Tool definitions, parallel execution, input validation |27| [Streaming](#streaming) | 3 | HIGH | SSE endpoints, structured streaming, backpressure handling |28| [Local Inference](#local-inference) | 3 | HIGH | Ollama setup, model selection, GPU optimization |29| [Fine-Tuning](#fine-tuning) | 3 | HIGH | LoRA/QLoRA training, dataset preparation, evaluation |30| [Context Optimization](#context-optimization) | 2 | HIGH | Window management, compression, caching, budget scaling |31| [Evaluation](#evaluation) | 2 | HIGH | LLM-as-judge, RAGAS metrics, quality gates, benchmarks |32| [Prompt Engineering](#prompt-engineering) | 4 | HIGH | CoT, few-shot, versioning, DSPy optimization, ReAct, cost optimization |3334**Total: 20 rules across 7 categories**3536## Quick Start3738```python39# Function calling: strict mode tool definition40tools = [{41 "type": "function",42 "function": {43 "name": "search_documents",44 "description": "Search knowledge base",45 "strict": True,46 "parameters": {47 "type": "object",48 "properties": {49 "query": {"type": "string", "description": "Search query"},50 "limit": {"type": "integer", "description": "Max results"}51 },52 "required": ["query", "limit"],53 "additionalProperties": False54 }55 }56}]57```5859```python60# Streaming: SSE endpoint with FastAPI61@app.get("/chat/stream")62async def stream_chat(prompt: str):63 async def generate():64 async for token in async_stream(prompt):65 yield {"event": "token", "data": token}66 yield {"event": "done", "data": ""}67 return EventSourceResponse(generate())68```6970```python71# Local inference: Ollama with LangChain72llm = ChatOllama(73 model="deepseek-r1:70b",74 base_url="http://localhost:11434",75 temperature=0.0,76 num_ctx=32768,77)78```7980```python81# Fine-tuning: QLoRA with Unsloth82model, tokenizer = FastLanguageModel.from_pretrained(83 model_name="unsloth/Meta-Llama-3.1-8B",84 max_seq_length=2048, load_in_4bit=True,85)86model = FastLanguageModel.get_peft_model(model, r=16, lora_alpha=32)87```8889## Function Calling9091Enable LLMs to use external tools and return structured data. Use strict mode schemas (2026 best practice) for reliability. On Claude keep the `tools` array stable across turns so the prompt cache holds; add tools mid-session with an inline `tool_addition` (beta `inline-tools-2026-09-15`, Claude API only, available on Sonnet 5.5 but not on Sonnet 5) instead of swapping subsets (https://platform.claude.com/docs/en/build-with-claude/mid-conversation-system-messages#mid-conversation-tool-changes). For large tool sets, use a non-deferred tool search tool with `defer_loading: true` on the rest, never `defer_loading` alone (a request cannot have every tool deferred). Forced `tool_choice` (`any` or a named tool) returns a 400 on Sonnet 5.5, Opus 5.5, Fable 5.1 and Mythos 5.1; use `auto` with `strict: true` tools. Validate all inputs with Pydantic/Zod, and return errors as tool results. <!-- model-recency-ok: per-model contrast, Sonnet 5 lacks what Sonnet 5.5 has (platform docs 2026-09-28) -->9293- `calling-tool-definition.md` -- Strict mode schemas, OpenAI/Anthropic formats, LangChain binding94- `calling-parallel.md` -- Parallel tool execution, asyncio.gather, strict mode constraints95- `calling-validation.md` -- Input validation, error handling, tool execution loops9697## Streaming9899Deliver LLM responses in real-time for better UX. Use SSE for web, WebSocket for bidirectional. Handle backpressure with bounded queues.100101- `streaming-sse.md` -- FastAPI SSE endpoints, frontend consumers, async iterators102- `streaming-structured.md` -- Streaming with tool calls, partial JSON parsing, chunk accumulation103- `streaming-backpressure.md` -- Backpressure handling, bounded buffers, cancellation104105## Local Inference106107Run LLMs locally with Ollama for cost savings (93% vs cloud), privacy, and offline development. Pre-warm models, use provider factory for cloud/local switching.108109- `local-ollama-setup.md` -- Installation, model pulling, environment configuration110- `local-model-selection.md` -- Model comparison by task, hardware profiles, quantization111- `local-gpu-optimization.md` -- Apple Silicon tuning, keep-alive, CI integration112113## Fine-Tuning114115Customize LLMs with parameter-efficient techniques. Fine-tune ONLY after exhausting prompt engineering and RAG. Requires 1000+ quality examples.116117- `tuning-lora.md` -- LoRA/QLoRA configuration, Unsloth training, adapter merging118- `tuning-dataset-prep.md` -- Synthetic data generation, quality validation, deduplication119- `tuning-evaluation.md` -- DPO alignment, evaluation metrics, anti-patterns120121## Context Optimization122123Manage context windows, compression, and attention-aware positioning. Optimize for tokens-per-task.124125- `context-window-management.md` -- Five-layer architecture, anchored summarization, compression triggers126- `context-caching.md` -- Just-in-time loading, budget scaling, probe evaluation, CC 2.1.32+127128## Evaluation129130Evaluate LLM outputs with multi-dimension scoring, quality gates, and benchmarks.131132- `evaluation-metrics.md` -- LLM-as-judge, RAGAS metrics, hallucination detection133- `evaluation-benchmarks.md` -- Quality gates, batch evaluation, pairwise comparison134135## Prompt Engineering136137Design, version, and optimize prompts for production LLM applications.138139- `prompt-design.md` -- Chain-of-Thought, few-shot learning, pattern selection guide140- `prompt-testing.md` -- Langfuse versioning, DSPy optimization, A/B testing, self-consistency141- `prompt-react-pattern.md` -- ReAct loop for tool-using agents, thought-action-observation format142- `prompt-optimization.md` -- Token reduction, cost optimization, model tiering, prompt spec format143144## Upstream coverage (do not restate)145146These topics are covered by their vendors' own documentation. This skill points147at them instead of teaching them; the rules above keep only our floors, ceilings148and scars. Our delta on all of it is in `references/ork-delta.md`.149150| Topic | First-party source |151|-------|--------------------|152| Strict-mode tool schemas, structured outputs | https://platform.openai.com/docs/guides/function-calling |153| Anthropic `input_schema` / `tool_use` | https://docs.claude.com/en/docs/agents-and-tools/tool-use/overview |154| SSE client mechanics, reconnection, cancellation | https://developer.mozilla.org/en-US/docs/Web/API/EventSource |155| LoRA / QLoRA config, target modules, adapter merging | https://huggingface.co/docs/peft/developer_guides/lora |156| Unsloth training loop, 4-bit loading | https://docs.unsloth.ai/get-started/fine-tuning-llms-guide |157| DPO, preference pairs, beta tuning, RLHF comparison | https://huggingface.co/docs/trl/dpo_trainer |158| SFT dataset formats (Alpaca, ChatML) | https://huggingface.co/docs/trl/sft_trainer |159| Embedding similarity for dataset dedup | https://sbert.net/ |160| Fine-tune vs prompt vs RAG decision framework | https://platform.openai.com/docs/guides/optimizing-llm-accuracy |161| Vendor token pricing (never hardcode it here) | https://platform.openai.com/docs/pricing |162163## Supporting Files164165- `references/ork-delta.md` -- our delta: scars, house ceilings, retired-file provenance166- `references/model-selection.md` -- local model comparison by task and hardware167- `scripts/create-lora-config.md` -- LoRA config scaffold with auto-detected model type168169## Key Decisions170171| Decision | Recommendation |172|----------|----------------|173| Tool schema mode | `strict: true` (2026 best practice) |174| Tool count | Load roughly 5-15 up front; defer the rest behind a non-deferred tool search tool |175| Streaming protocol | SSE for web, WebSocket for bidirectional |176| Buffer size | 50-200 tokens |177| Local model (reasoning) | `deepseek-r1:70b` |178| Local model (coding) | `qwen2.5-coder:32b` |179| Fine-tuning approach | LoRA/QLoRA (try prompting first) |180| LoRA rank | 16-64 typical |181| Training epochs | 1-3 (more risks overfitting) |182| Context compression | Anchored iterative (60-80%) |183| Compress trigger | 70% utilization, target 50% |184| Judge model | `claude-haiku-5-5` (cost tier), `gpt-5.5`, or `gemini-3.8-flash` (Google's GA general-purpose model and a third vendor for the different-model rule; price lives in `models.vocab.json`, not here) |185| Quality threshold | 0.7 production, 0.6 drafts |186| Few-shot examples | 3-5 diverse, representative |187| Prompt versioning | Langfuse with labels |188| Auto-optimization | DSPy MIPROv2 |189190## Related Skills191192- `ork-extras:rag-retrieval` -- Embedding patterns, when RAG is better than fine-tuning193- `agent-loops` -- Multi-step tool use with reasoning194- `llm-evaluation` -- Evaluate fine-tuned and local models195- `langfuse-observability` -- Track training experiments196197## Capability Details198199### function-calling200**Keywords:** tool, function, define tool, tool schema, function schema, strict mode, parallel tools201**Solves:**202- Define tools with clear descriptions and strict schemas203- Execute tool calls in parallel with asyncio.gather204- Validate inputs and handle errors in tool execution loops205206### streaming207**Keywords:** streaming, SSE, Server-Sent Events, real-time, backpressure, token stream208**Solves:**209- Stream LLM tokens via SSE endpoints210- Handle tool calls within streams211- Manage backpressure with bounded queues212213### local-inference214**Keywords:** Ollama, local, self-hosted, model selection, GPU, Apple Silicon215**Solves:**216- Set up Ollama for local LLM inference217- Select models based on task and hardware218- Optimize GPU usage and CI integration219220### fine-tuning221**Keywords:** LoRA, QLoRA, fine-tune, DPO, synthetic data, PEFT, alignment222**Solves:**223- Configure LoRA/QLoRA for parameter-efficient training224- Generate and validate synthetic training data225- Align models with DPO and evaluate results