Agf Wiring Multi LLM Sdk

Wiring Multi-LLM SDK (DeepSeek / Doubao / Qwen / MiniMax)

Use this skill when:

• You add a new LLM provider to backend/app/agents/ or any backend module
• You switch the default provider (e.g. DeepSeek → Doubao for cost reasons)
• You wire fallback / retry logic between providers
• You suspect a provider mismatch is the cause of a bug

Decision: which SDK style?

All four providers expose OpenAI-compatible endpoints. Default to the openai Python SDK with a custom base_url rather than each vendor's bespoke SDK — fewer dependencies, easier to swap, less drift.

Bespoke SDK exceptions:

• Doubao multimodal (image/video gen): use volcengine-python-sdk for Ark image API
• MiniMax video / TTS: use minimax official SDK
• Streaming nuance: confirm OAI-compat client handles vendor's stream chunk format

Before wiring any SDK, pull its current docs via Context7 (resolve-library-id → get-library-docs) — all four vendors iterate fast and training-data memory of their APIs is likely stale. Context7 coverage of domestic SDKs varies; if a library isn't indexed, fall back to WebFetch on official docs.

Env var contract (locked)

All providers follow the same pattern. Never hardcode keys. Each is read from environment at module init; a missing key raises early.

Doubao gotcha: the "model name" in OAI-compat call is actually the Ark endpoint id (ep-2024xxxx), not a public model id like doubao-pro-32k. Get the endpoint id from Volc Ark console.

Adapter skeleton (Python / FastAPI)

# backend/app/agents/llm_clients.py
import os
from openai import OpenAI

def get_client(provider: str) -> tuple[OpenAI, str]:
    if provider == "deepseek":
        return OpenAI(
            api_key=os.environ["DEEPSEEK_API_KEY"],
            base_url=os.getenv("DEEPSEEK_BASE_URL", "https://api.deepseek.com/v1"),
        ), os.getenv("DEEPSEEK_MODEL", "deepseek-chat")
    if provider == "doubao":
        return OpenAI(
            api_key=os.environ["ARK_API_KEY"],
            base_url=os.getenv("ARK_BASE_URL", "https://ark.cn-beijing.volces.com/api/v3"),
        ), os.environ["ARK_MODEL_ENDPOINT_ID"]
    if provider == "qwen":
        return OpenAI(
            api_key=os.environ["DASHSCOPE_API_KEY"],
            base_url=os.getenv("DASHSCOPE_BASE_URL", "https://dashscope.aliyuncs.com/compatible-mode/v1"),
        ), os.getenv("QWEN_MODEL", "qwen-plus")
    if provider == "minimax":
        return OpenAI(
            api_key=os.environ["MINIMAX_API_KEY"],
            base_url=os.getenv("MINIMAX_BASE_URL", "https://api.minimaxi.com/v1"),
        ), os.getenv("MINIMAX_MODEL", "abab6.5s-chat")
    raise ValueError(f"unknown provider: {provider}")

Fallback strategy

Default order (tunable in CLAUDE.md per project): DeepSeek → Doubao → Qwen → MiniMax.

• 5xx / network → next provider
• 4xx (auth / quota) → DO NOT failover; raise (signals config bug)
• Latency P95 breach (> 5s for non-streaming) → tier down silently, log warning

Implement with tenacity retry + a thin orchestrator that walks the list. Never silently swap models without telemetry — every failover emits a structured log line per observability.md.

Token + cost telemetry

Every call must record the LLM fields mandated by observability.md:17 plus provider. DeepSeek + Doubao support prompt caching — read cache_hit_ratio off the response usage object.

Verifications before "done"

Before claiming the integration works, run this checklist explicitly. Verify outputs match expectations — do not assume.

• Smoke: each enabled provider answers "你好" with non-empty text and >0 token usage
• Streaming: chunk delivery is real (not buffered all-at-once); first-token latency logged
• Failover: simulate one provider down (point base_url at localhost:1) and confirm auto-fallback + log line
• Cost: token usage written to log + DB on every call
• Secrets: no key appears in logs / git diff (run git diff | grep -iE 'api[_-]?key|secret|token' before commit)
• Unit test mocks at the OpenAI SDK boundary (not at HTTP level) — survives base_url changes
• SDK surface: API calls cross-checked against current docs (Context7 or official site) — not from training memory

Anti-patterns

• ❌ Hardcoding base_url strings — always read from env
• ❌ Passing public model id to Doubao — use endpoint id
• ❌ Catching all exceptions and returning empty string — masks auth failures and quota issues
• ❌ Assuming all providers return the same usage shape — validate and normalize
• ❌ Storing keys in CLAUDE.md / settings.json — only .env (gitignored) or secret manager

References

• DeepSeek: https://platform.deepseek.com/api-docs
• Doubao / Volc Ark: https://www.volcengine.com/docs/82379
• Qwen / DashScope: https://help.aliyun.com/zh/dashscope/developer-reference/compatibility-of-openai-with-dashscope
• MiniMax: https://platform.minimaxi.com/document/ChatCompletion