CATLX — AI Provider Architecture
This skill owns the AI provider abstraction and routing: a uniform ProviderClient interface, a router
that picks the best provider per request, a failover protocol, cost-based routing, and local LLM hosting.
Canonical detail: ../../knowledge/references/ai-provider.md. Policy template:
../../templates/providers.yaml. Load on demand.
Purpose
Make every AI inference go through a single abstraction (complete() / streamComplete() / embed() /
transcribe() / synthesize()), so callers never know which provider serves a request, and so requests are
routed to the best provider for the situation (latency, cost, quality, health, hardware tier).
When to activate
- User asks which provider is used, or how CATLX fails over between providers.
- Configuring a routing policy, local model, or quant level.
- Debugging provider latency/timeout/cost.
What this skill handles
- Provider Abstraction Layer (PAL) — normalizes requests/responses across all providers; single
ProviderClient interface; callers unaware of the provider.
- Current providers — Google AI Studio (Gemini Pro/Flash, embedding, multimodal, 1M-token context),
Groq (fast completion, Whisper STT, sub-100 ms first-token), Hugging Face (specialized models,
fine-tuned, 40k+ models).
- Future providers — OpenAI, Anthropic Claude, llama.cpp, vLLM (T2/T3), Ollama (T1+), custom GGUF/
safetensors, Azure OpenAI, AWS Bedrock (planned phases).
- Provider routing & failover — choose best provider by request type, latency, token budget, health,
and tier. Declarative routing policy in YAML (
providers.yaml).
- Failover protocol — timeout default 10 s (3 s fast-path); mark degraded (30 s cooldown); next in
priority; all remote degraded → best local model (T1+); no local (T0) → queue + notify + retry; log with
correlation IDs.
- Cost optimization router — for non-latency-critical tasks, pick the cheapest tier meeting quality;
fast models for classification/extraction; large models for complex reasoning/code/multi-step planning.
- Local LLM integration — Local Model Manager: download/verify GGUF from HF, quantization (Q4_K_M for
T1, Q8_0 for T2), start llama.cpp server (or vLLM T2+), health monitor; OpenAI-compatible API on
localhost = drop-in replacement for any remote provider.
Local model recommendations (by tier)
| Tier |
Models |
| T1 (16 GB, no GPU) |
Llama 3.2 3B Q8_0, Phi-3 mini Q8_0, Gemma 2 2B Q8_0 |
| T2 (64 GB, RTX) |
Llama 3.1 70B Q4_K_M, Qwen 2.5 72B Q4, Mistral Large Q4 |
| T3 (enterprise) |
Full precision Llama 3.1 405B, custom fine-tuned |
Requirements / constraints
- R6 (single access point): no application code calls a provider directly; always through PAL.
- R10 (local-first): on T2+ prefer local providers; offline mode disables remote providers.
- R13 (telemetry): failover events logged with correlation IDs.
Canonical knowledge it reads
../../knowledge/references/ai-provider.md · ../../knowledge/references/memory-architecture.md ·
../../knowledge/references/hardware-adaptation.md · ../../knowledge/rules/architectural-rules.md.
Delegation
- RAG context before a completion → delegate to
catlx-memory
(skill({ name: "catlx-memory" })).
- Which tier / whether local is available → delegate to
catlx-hardware-adaptation
(skill({ name: "catlx-hardware-adaptation" })).
- Where the local model server runs (container) → delegate to
catlx-docker
(skill({ name: "catlx-docker" })).
- Offline/capability fallback → delegate to
catlx-capability-routing
(skill({ name: "catlx-capability-routing" })).
- Credential access for provider API keys → delegate to
catlx-security
(skill({ name: "catlx-security" })).
Edge cases & warnings
- Timeout/degraded provider — follow the failover chain; do not fail the request if a fallback exists.
- All remote degraded on T0 — queue with user notification and retry on recovery; be honest about the
limitation.
- Local server unavailable — route back to remote; never silently return a fabricated completion.
- Cost vs quality — for latency-critical tasks use fast path; for batch/background use cost router.
- Credential health — never log API keys; PAL keeps them in the vault.
Component lifecycle policy (reuse → install → adapt → create)
NEVER create a new component as the default. Before building/creating anything (a sub-skill, dependency,
reference, workflow, helper, adapter, or template), check, in order:
- Reuse an existing local component (resolve aliases/equivalent capabilities first) — reuse, don't rebuild.
- Use an already-registered component from the registry.
- Install a suitable existing, trusted, supported component → validate → register → connect to the graph → use.
- Adapt an existing compatible component via a small persistent adapter/wrapper instead of re-creating it.
- Create only as last resort — then make it permanent immediately: stable id, canonical location, register,
add to the capability index + dependency graph, add provenance, use, and allow future reuse.
- Never reorganise/recreate already-generated components (no
Skill X 2 / new / temp variants); extend the
existing one. Never create a second competing knowledge source; connect back to the canonical knowledge/ layer.
Promote any reusable artifact out of /tmp/scratch into the permanent ecosystem.
Full policy: ../../knowledge/rules/component-lifecycle.md.
Source / provenance
- Source: PART VIII §8.1–8.5 (PAL, current/future providers, routing failover, cost router, local LLM).
- Inferred: none beyond Windows mapping of local server/OpenAI-compatible host.
1---2name: catlx-ai-provider-23description: "Handles the CATLX AI provider architecture: the Provider Abstraction Layer (PAL), current and future providers, provider routing and failover protocol, the cost optimization router, and local LLM integration (llama.cpp/vLLM with OpenAI-compatible API). Use when the user asks about which AI provider CATLX uses, provider failover, routing policies, the cost router, local models, embeddings, or how AI inference is abstracted."4metadata:5 catlx: subsystem6 category: architecture7 subsystem: AI_Provider_Layer8 capability: ai-provider-routing9 version: "1.0.0"10 source: https://skillmd.com/skills/samraddhashrivastavatech/catlx-ai-provider-211 upstream: "PART VIII §8.1-8.5"12 aliases: "ai provider, llm, model, inference, embeddings, provider routing, failover, local model, pal"13 depends-on: "catlx-capability-routing"14---1516# CATLX — AI Provider Architecture1718This skill owns the **AI provider abstraction and routing**: a uniform `ProviderClient` interface, a router19that picks the best provider per request, a failover protocol, cost-based routing, and local LLM hosting.2021> Canonical detail: `../../knowledge/references/ai-provider.md`. Policy template:22> `../../templates/providers.yaml`. Load on demand.2324---2526## Purpose2728Make every AI inference go through a single abstraction (`complete()` / `streamComplete()` / `embed()` /29`transcribe()` / `synthesize()`), so callers never know which provider serves a request, and so requests are30routed to the best provider for the situation (latency, cost, quality, health, hardware tier).3132## When to activate3334- User asks which provider is used, or how CATLX fails over between providers.35- Configuring a routing policy, local model, or quant level.36- Debugging provider latency/timeout/cost.3738## What this skill handles39401. **Provider Abstraction Layer (PAL)** — normalizes requests/responses across all providers; single41 `ProviderClient` interface; callers unaware of the provider.422. **Current providers** — Google AI Studio (Gemini Pro/Flash, embedding, multimodal, 1M-token context),43 Groq (fast completion, Whisper STT, sub-100 ms first-token), Hugging Face (specialized models,44 fine-tuned, 40k+ models).453. **Future providers** — OpenAI, Anthropic Claude, llama.cpp, vLLM (T2/T3), Ollama (T1+), custom GGUF/46 safetensors, Azure OpenAI, AWS Bedrock (planned phases).474. **Provider routing & failover** — choose best provider by request type, latency, token budget, health,48 and tier. Declarative routing policy in YAML (`providers.yaml`).495. **Failover protocol** — timeout default 10 s (3 s fast-path); mark degraded (30 s cooldown); next in50 priority; all remote degraded → best local model (T1+); no local (T0) → queue + notify + retry; log with51 correlation IDs.526. **Cost optimization router** — for non-latency-critical tasks, pick the cheapest tier meeting quality;53 fast models for classification/extraction; large models for complex reasoning/code/multi-step planning.547. **Local LLM integration** — Local Model Manager: download/verify GGUF from HF, quantization (Q4_K_M for55 T1, Q8_0 for T2), start llama.cpp server (or vLLM T2+), health monitor; **OpenAI-compatible API on56 localhost** = drop-in replacement for any remote provider.5758## Local model recommendations (by tier)5960| Tier | Models |61|---|---|62| T1 (16 GB, no GPU) | Llama 3.2 3B Q8_0, Phi-3 mini Q8_0, Gemma 2 2B Q8_0 |63| T2 (64 GB, RTX) | Llama 3.1 70B Q4_K_M, Qwen 2.5 72B Q4, Mistral Large Q4 |64| T3 (enterprise) | Full precision Llama 3.1 405B, custom fine-tuned |6566## Requirements / constraints6768- **R6 (single access point):** no application code calls a provider directly; always through PAL.69- **R10 (local-first):** on T2+ prefer local providers; offline mode disables remote providers.70- **R13 (telemetry):** failover events logged with correlation IDs.7172## Canonical knowledge it reads7374`../../knowledge/references/ai-provider.md` · `../../knowledge/references/memory-architecture.md` ·75`../../knowledge/references/hardware-adaptation.md` · `../../knowledge/rules/architectural-rules.md`.7677## Delegation7879- **RAG context before a completion** → delegate to `catlx-memory`80 (`skill({ name: "catlx-memory" })`).81- **Which tier / whether local is available** → delegate to `catlx-hardware-adaptation`82 (`skill({ name: "catlx-hardware-adaptation" })`).83- **Where the local model server runs (container)** → delegate to `catlx-docker`84 (`skill({ name: "catlx-docker" })`).85- **Offline/capability fallback** → delegate to `catlx-capability-routing`86 (`skill({ name: "catlx-capability-routing" })`).87- **Credential access for provider API keys** → delegate to `catlx-security`88 (`skill({ name: "catlx-security" })`).8990## Edge cases & warnings9192- **Timeout/degraded provider** — follow the failover chain; do not fail the request if a fallback exists.93- **All remote degraded on T0** — queue with user notification and retry on recovery; be honest about the94 limitation.95- **Local server unavailable** — route back to remote; never silently return a fabricated completion.96- **Cost vs quality** — for latency-critical tasks use fast path; for batch/background use cost router.97- **Credential health** — never log API keys; PAL keeps them in the vault.9899## Component lifecycle policy (reuse → install → adapt → create)100101**NEVER create a new component as the default.** Before building/creating anything (a sub-skill, dependency,102reference, workflow, helper, adapter, or template), check, in order:1031. **Reuse** an existing local component (resolve aliases/equivalent capabilities first) — reuse, don't rebuild.1042. **Use** an already-registered component from the registry.1053. **Install** a suitable existing, trusted, supported component → validate → register → connect to the graph → use.1064. **Adapt** an existing compatible component via a small persistent adapter/wrapper instead of re-creating it.1075. **Create only as last resort** — then make it permanent immediately: stable id, canonical location, register,108 add to the capability index + dependency graph, add provenance, use, and allow future reuse.1096. Never reorganise/recreate already-generated components (no `Skill X 2` / `new` / `temp` variants); extend the110 existing one. Never create a second competing knowledge source; connect back to the canonical `knowledge/` layer.111 Promote any reusable artifact out of `/tmp`/scratch into the permanent ecosystem.112113> Full policy: `../../knowledge/rules/component-lifecycle.md`.114115## Source / provenance116117- **Source:** PART VIII §8.1–8.5 (PAL, current/future providers, routing failover, cost router, local LLM).118- **Inferred:** none beyond Windows mapping of local server/OpenAI-compatible host.