CATLX — AI Provider Architecture
This skill owns the AI provider abstraction and routing: a uniform ProviderClient interface, a router
that picks the best provider per request, a failover protocol, cost-based routing, and local LLM hosting.
Canonical detail: ../knowledge/references/ai-provider.md. Policy template:
../templates/providers.yaml. Load on demand.
Purpose
Make every AI inference go through a single abstraction (complete() / streamComplete() / embed() /
transcribe() / synthesize()), so callers never know which provider serves a request, and so requests are
routed to the best provider for the situation (latency, cost, quality, health, hardware tier).
When to activate
- User asks which provider is used, or how CATLX fails over between providers.
- Configuring a routing policy, local model, or quant level.
- Debugging provider latency/timeout/cost.
What this skill handles
- Provider Abstraction Layer (PAL) — normalizes requests/responses across all providers; single
ProviderClient interface; callers unaware of the provider.
- Current providers — Google AI Studio (Gemini Pro/Flash, embedding, multimodal, 1M-token context),
Groq (fast completion, Whisper STT, sub-100 ms first-token), Hugging Face (specialized models,
fine-tuned, 40k+ models).
- Future providers — OpenAI, Anthropic Claude, llama.cpp, vLLM (T2/T3), Ollama (T1+), custom GGUF/
safetensors, Azure OpenAI, AWS Bedrock (planned phases).
- Provider routing & failover — choose best provider by request type, latency, token budget, health,
and tier. Declarative routing policy in YAML (
providers.yaml).
- Failover protocol — timeout default 10 s (3 s fast-path); mark degraded (30 s cooldown); next in
priority; all remote degraded → best local model (T1+); no local (T0) → queue + notify + retry; log with
correlation IDs.
- Cost optimization router — for non-latency-critical tasks, pick the cheapest tier meeting quality;
fast models for classification/extraction; large models for complex reasoning/code/multi-step planning.
- Local LLM integration — Local Model Manager: download/verify GGUF from HF, quantization (Q4_K_M for
T1, Q8_0 for T2), start llama.cpp server (or vLLM T2+), health monitor; OpenAI-compatible API on
localhost = drop-in replacement for any remote provider.
Local model recommendations (by tier)
| Tier |
Models |
| T1 (16 GB, no GPU) |
Llama 3.2 3B Q8_0, Phi-3 mini Q8_0, Gemma 2 2B Q8_0 |
| T2 (64 GB, RTX) |
Llama 3.1 70B Q4_K_M, Qwen 2.5 72B Q4, Mistral Large Q4 |
| T3 (enterprise) |
Full precision Llama 3.1 405B, custom fine-tuned |
Requirements / constraints
- R6 (single access point): no application code calls a provider directly; always through PAL.
- R10 (local-first): on T2+ prefer local providers; offline mode disables remote providers.
- R13 (telemetry): failover events logged with correlation IDs.
Canonical knowledge it reads
../knowledge/references/ai-provider.md · ../knowledge/references/memory-architecture.md ·
../knowledge/references/hardware-adaptation.md · ../knowledge/rules/architectural-rules.md.
Delegation
- RAG context before a completion → delegate to
catlx-memory
(skill({ name: "catlx-memory" })).
- Which tier / whether local is available → delegate to
catlx-hardware-adaptation
(skill({ name: "catlx-hardware-adaptation" })).
- Where the local model server runs (container) → delegate to
catlx-docker
(skill({ name: "catlx-docker" })).
- Offline/capability fallback → delegate to
catlx-capability-routing
(skill({ name: "catlx-capability-routing" })).
- Credential access for provider API keys → delegate to
catlx-security
(skill({ name: "catlx-security" })).
Edge cases & warnings
- Timeout/degraded provider — follow the failover chain; do not fail the request if a fallback exists.
- All remote degraded on T0 — queue with user notification and retry on recovery; be honest about the
limitation.
- Local server unavailable — route back to remote; never silently return a fabricated completion.
- Cost vs quality — for latency-critical tasks use fast path; for batch/background use cost router.
- Credential health — never log API keys; PAL keeps them in the vault.
Component lifecycle policy (reuse → install → adapt → create)
NEVER create a new component as the default. Before building/creating anything (a sub-skill, dependency,
reference, workflow, helper, adapter, or template), check, in order:
- Reuse an existing local component (resolve aliases/equivalent capabilities first) — reuse, don't rebuild.
- Use an already-registered component from the registry.
- Install a suitable existing, trusted, supported component → validate → register → connect to the graph → use.
- Adapt an existing compatible component via a small persistent adapter/wrapper instead of re-creating it.
- Create only as last resort — then make it permanent immediately: stable id, canonical location, register,
add to the capability index + dependency graph, add provenance, use, and allow future reuse.
- Never reorganise/recreate already-generated components (no
Skill X 2 / new / temp variants); extend the
existing one. Never create a second competing knowledge source; connect back to the canonical knowledge/ layer.
Promote any reusable artifact out of /tmp/scratch into the permanent ecosystem.
Full policy: ../knowledge/rules/component-lifecycle.md.
Source / provenance
- Source: PART VIII §8.1–8.5 (PAL, current/future providers, routing failover, cost router, local LLM).
- Inferred: none beyond Windows mapping of local server/OpenAI-compatible host.
1---2> Mirror copy of `skills\catlx-core\catlx-ai-provider\SKILL.md` (canonical content lives in `catlx-core`). Kept at the repository root only for registry-path compatibility. Edit the canonical copy, not this file.3name: catlx-ai-provider4description: "Handles the CATLX AI provider architecture: the Provider Abstraction Layer (PAL), current and future providers, provider routing and failover protocol, the cost optimization router, and local LLM integration (llama.cpp/vLLM with OpenAI-compatible API). Use when the user asks about which AI provider CATLX uses, provider failover, routing policies, the cost router, local models, embeddings, or how AI inference is abstracted."5metadata:6 catlx: subsystem7 category: architecture8 subsystem: AI_Provider_Layer9 capability: ai-provider-routing10 version: "1.0.0"11 source: https://skillmd.com/skills/samraddhashrivastavatech/catlx-ai-provider12 upstream: "PART VIII §8.1-8.5"13 aliases: "ai provider, llm, model, inference, embeddings, provider routing, failover, local model, pal"14 depends-on: "catlx-capability-routing"15---1617# CATLX — AI Provider Architecture1819This skill owns the **AI provider abstraction and routing**: a uniform `ProviderClient` interface, a router20that picks the best provider per request, a failover protocol, cost-based routing, and local LLM hosting.2122> Canonical detail: `../knowledge/references/ai-provider.md`. Policy template:23> `../templates/providers.yaml`. Load on demand.2425---2627## Purpose2829Make every AI inference go through a single abstraction (`complete()` / `streamComplete()` / `embed()` /30`transcribe()` / `synthesize()`), so callers never know which provider serves a request, and so requests are31routed to the best provider for the situation (latency, cost, quality, health, hardware tier).3233## When to activate3435- User asks which provider is used, or how CATLX fails over between providers.36- Configuring a routing policy, local model, or quant level.37- Debugging provider latency/timeout/cost.3839## What this skill handles40411. **Provider Abstraction Layer (PAL)** — normalizes requests/responses across all providers; single42 `ProviderClient` interface; callers unaware of the provider.432. **Current providers** — Google AI Studio (Gemini Pro/Flash, embedding, multimodal, 1M-token context),44 Groq (fast completion, Whisper STT, sub-100 ms first-token), Hugging Face (specialized models,45 fine-tuned, 40k+ models).463. **Future providers** — OpenAI, Anthropic Claude, llama.cpp, vLLM (T2/T3), Ollama (T1+), custom GGUF/47 safetensors, Azure OpenAI, AWS Bedrock (planned phases).484. **Provider routing & failover** — choose best provider by request type, latency, token budget, health,49 and tier. Declarative routing policy in YAML (`providers.yaml`).505. **Failover protocol** — timeout default 10 s (3 s fast-path); mark degraded (30 s cooldown); next in51 priority; all remote degraded → best local model (T1+); no local (T0) → queue + notify + retry; log with52 correlation IDs.536. **Cost optimization router** — for non-latency-critical tasks, pick the cheapest tier meeting quality;54 fast models for classification/extraction; large models for complex reasoning/code/multi-step planning.557. **Local LLM integration** — Local Model Manager: download/verify GGUF from HF, quantization (Q4_K_M for56 T1, Q8_0 for T2), start llama.cpp server (or vLLM T2+), health monitor; **OpenAI-compatible API on57 localhost** = drop-in replacement for any remote provider.5859## Local model recommendations (by tier)6061| Tier | Models |62|---|---|63| T1 (16 GB, no GPU) | Llama 3.2 3B Q8_0, Phi-3 mini Q8_0, Gemma 2 2B Q8_0 |64| T2 (64 GB, RTX) | Llama 3.1 70B Q4_K_M, Qwen 2.5 72B Q4, Mistral Large Q4 |65| T3 (enterprise) | Full precision Llama 3.1 405B, custom fine-tuned |6667## Requirements / constraints6869- **R6 (single access point):** no application code calls a provider directly; always through PAL.70- **R10 (local-first):** on T2+ prefer local providers; offline mode disables remote providers.71- **R13 (telemetry):** failover events logged with correlation IDs.7273## Canonical knowledge it reads7475`../knowledge/references/ai-provider.md` · `../knowledge/references/memory-architecture.md` ·76`../knowledge/references/hardware-adaptation.md` · `../knowledge/rules/architectural-rules.md`.7778## Delegation7980- **RAG context before a completion** → delegate to `catlx-memory`81 (`skill({ name: "catlx-memory" })`).82- **Which tier / whether local is available** → delegate to `catlx-hardware-adaptation`83 (`skill({ name: "catlx-hardware-adaptation" })`).84- **Where the local model server runs (container)** → delegate to `catlx-docker`85 (`skill({ name: "catlx-docker" })`).86- **Offline/capability fallback** → delegate to `catlx-capability-routing`87 (`skill({ name: "catlx-capability-routing" })`).88- **Credential access for provider API keys** → delegate to `catlx-security`89 (`skill({ name: "catlx-security" })`).9091## Edge cases & warnings9293- **Timeout/degraded provider** — follow the failover chain; do not fail the request if a fallback exists.94- **All remote degraded on T0** — queue with user notification and retry on recovery; be honest about the95 limitation.96- **Local server unavailable** — route back to remote; never silently return a fabricated completion.97- **Cost vs quality** — for latency-critical tasks use fast path; for batch/background use cost router.98- **Credential health** — never log API keys; PAL keeps them in the vault.99100## Component lifecycle policy (reuse → install → adapt → create)101102**NEVER create a new component as the default.** Before building/creating anything (a sub-skill, dependency,103reference, workflow, helper, adapter, or template), check, in order:1041. **Reuse** an existing local component (resolve aliases/equivalent capabilities first) — reuse, don't rebuild.1052. **Use** an already-registered component from the registry.1063. **Install** a suitable existing, trusted, supported component → validate → register → connect to the graph → use.1074. **Adapt** an existing compatible component via a small persistent adapter/wrapper instead of re-creating it.1085. **Create only as last resort** — then make it permanent immediately: stable id, canonical location, register,109 add to the capability index + dependency graph, add provenance, use, and allow future reuse.1106. Never reorganise/recreate already-generated components (no `Skill X 2` / `new` / `temp` variants); extend the111 existing one. Never create a second competing knowledge source; connect back to the canonical `knowledge/` layer.112 Promote any reusable artifact out of `/tmp`/scratch into the permanent ecosystem.113114> Full policy: `../knowledge/rules/component-lifecycle.md`.115116## Source / provenance117118- **Source:** PART VIII §8.1–8.5 (PAL, current/future providers, routing failover, cost router, local LLM).119- **Inferred:** none beyond Windows mapping of local server/OpenAI-compatible host.