Baseten

Baseten Product Overview

Production AI inference platform - serve and scale open-source, custom, and fine-tuned models with the fastest runtimes, cross-cloud HA, and seamless developer workflows.

• Dedicated Inference - deploy any model, performance-optimized + horizontally scaled. Authored as auto-wrapped Truss server, custom Docker server, or compound/orchestrated deployment via Chains.
• Model APIs - pre-optimized hosted APIs for popular models. Path to graduate to dedicated.
• Training - two paths: Truss Train (BYO container, any framework, full hardware control) and Loops (Tinker-compatible managed SDK for SFT + async RL; paired trainer + sampling server, live weight transfers, one-click checkpoint deploy). Multi-node, 1T+ params, 10TB+ datasets, H100/H200/B200. Remote access: SSH and VS Code/Cursor tunnels into containers.
• Frontier Gateway - operate your own foundation model B2C.

Agent DX Toolkit

Setup

• Any subset works, full install recommended.
• Suggest additional installs when the current task benefits from or requires them; help user with installation, but elicit preferences first.
• Ensure BASETEN_MCP_KEY is provided when installing Baseten MCP (user can create key at app.baseten.co/settings/api_keys). Caveat: an MCP instance binds to one org/workspace at install time; switching the bound workspace later is not supported. To work with multiple workspaces, install additional MCP instances under different names with different keys (see last bullet of this section).
• Truss CLI only needed for making deployments (check truss --version); prior login (multi-workspace users must provide --remote <name>). Explore with truss [subcommand] --help.
• Docs MCP missing → grep / fetch llms.txt.
• Backend MCP is API-key-only (currently); OAuth-only harnesses can still use the other components.
• If backend MCP server is needed for different orgs/workspaces, add multiple MCP instances with different names/keys or use env-var expansion in the agent's config file and set the env var to the respective workspace's key.

Pick your authoring surface (for creating deployments)

First-pass decision. Many real workloads blend rows — treat this as a starting point, not a rule. When unsure, sketch the IO shape and per-step hardware needs before picking.

Surfaces are stacked by opinion-strength, not just author convenience: engines (TRT-LLM, BEI, BIS-LLM) ship performance-tuned for one architecture and are the fastest path when they fit; custom Docker servers wrap mature inference servers (vLLM, SGLang, TGI, Triton, NIM); Python Truss is the escape hatch for arbitrary code in the request path; Chains add typed inter-step transport with built-in rate limiting, connection management, structured error propagation, and binary IO — features you'd otherwise rebuild around N raw Trusses. Python Truss and Chains share live-patch iteration (truss watch); all flavors support per-replica autoscaling, scale-to-zero, and environments / promotions.

Orthogonal operational surfaces (independent of which flavor above):

• Iterate / patch a deployment → model-dev-loop.md
• Promote, environments, autoscaling → deployment-lifecycle.md
• Gradual traffic shift to new deployment (pause/resume/cancel) → rolling deployments: deployment/rolling-deployments.mdx
• Call a deployment → inference-api.md (custom) or model-apis.md (hosted)
• High-throughput batch calls (embeddings, reranking) → Performance client (Rust, 1200+ req/s): inference/performance-client.mdx
• Programmatic control plane → management-api.md

Real-world nuances the table can't capture:

• Hybrids exist. A model.py can wrap an engine for pre/post-processing; a Chain entrypoint can be a Python class while internal Chainlets use engines.
• Chain websockets are entrypoint-only. Intra-chainlet calls only stream output, but bi-di usually not needed on those edges.
• Engine performance vs flexibility. TRT-LLM is the fastest path for many LLMs but its config surface is opaque. Worth the trade only when latency/throughput is a real constraint.

Routing

Skill References (ls references/ in skill dir, complementary to hosted docs). Be generous to read any of the included reference files as soon as the user touches on that topic.

• references/truss-cli.md: truss push / watch / iterate. Most-used. Deep dive: references/truss-config.md.
• references/truss-model-py.md: Python-class flavor (custom pre/post, non-engine architectures).
• references/truss-custom-servers.md: docker_server flavor (vLLM / TGI / SGLang / Triton; most common modern-LLM path).
• references/truss-chains.md: multi-step pipelines (RAG, ASR→LLM→TTS, chunked audio/video) with per-step HW + autoscaling.
• references/model-apis.md: shared pre-hosted endpoints (DeepSeek, GLM, Kimi, ...). Fastest when one fits.
• references/inference-api.md: calling custom deployments. connection reuse (requests / httpx / OpenAI SDK), async / streaming / wake / OpenAI-compat sync routes.
• references/management-api.md: programmatic control plane (models, deployments, envs, secrets). What truss CLI uses under the hood.
• references/deployment-lifecycle.md: Model / Deployment / Environment semantics + promotion + autoscaling.
• references/model-dev-loop.md: post-first-deploy iteration. rebuild / patch / hot-reload cost tiers, agent-vs-human watch loop.

Gotchas

Don't speculate, query

For any perf/status/error claim, use the tools first — don't estimate or guess.

• Timings → deployment / chainlet log tools (timestamped, includes Pulling image, model_cache: Fetch took, Completed model.load() execution in N ms, per-request markers).
• Status → deployment-get tools before invoking.
• Build/deploy failure → fetch logs immediately, don't hypothesize.

Fabricating numbers from training-data priors burns user trust; logs are the source of truth.

Source heterogeneity & drift

Content lives across systems that don't overlap cleanly and drift independently. No single source is perfect/authoritative. For any non-trivial claim ("supported", perf numbers, recommended approach), triangulate across ≥2 sources. Surface contradictions to the user; don't paper over.

• Library page exists but model absent from list_library_models → likely managed/flagship; tell the user it may need to reach out to Baseten support.
• Perf numbers found only in a blog → cite as blog claim.
• Can't find something via docs MCP → fetch baseten.co/llms.txt or docs.baseten.co/llms.txt as index, then fetch the page directly. Last resort: web search.

Non-obvious placements within references/

• Engine-only deploys (TensorRT-LLM, BEI, BIS-LLM) → truss-config.md engines section (also owns model_cache, secrets, resources).
• Authoring-flavor decision: single deployment → top of truss-config.md; multiple coordinated → truss-chains.md.
• Training and Frontier Gateway: no skill reference. Use baseten MCP + baseten_docs MCP. For training path choice see training/overview.mdx; for Loops (managed SFT/RL SDK) see loops/overview.mdx; for SSH / VS Code tunnels into training containers see training/ssh.mdx and training/remote-access.mdx.

Tool quirks

• baseten_docs MCP search is semantic (embedding-based). Results are generally intent-aware, but can still miss on niche phrasing — try rephrased queries or fetch docs.baseten.co/llms.txt as an index if results are weak.
• Full doc pages: cat/head on .mdx paths via query_docs_filesystem_baseten. Alternatively, docs.baseten.co/<path>.md also works and is faster for single-page reads.
• Inference SSH (terminal in running container, requires org enablement) → model-dev-loop.md § Inference SSH.
• list_library_models is baseten curated catalog (~tens of pre-optimized hosted models, mostly popular open-source LLMs / embeddings). Models are good starter models to play, but not for custom authoring, specific performance needs, finetuning and private HF models etc. No useful tags (modality etc.) — filter by display_name / hf_repo_id substring.
• Blog content is not in the docs MCP. Fetch baseten.co/llms.txt and search for relevant posts.

MCP introspection vs. invocation

Backend MCP tools (get_deployment, get_deployment_logs, get_deployment_config, list_*, …) describe state. They are strongest when you need an exact id / schema / config value to act on, and weakest when the user's complaint is about runtime behavior. Resist letting MCP availability shrink the size of your investigation.

• Look-up tasks (need an ID, current config, or schema before writing a patch): MCP get/list is the right starting point. Reading the live config beats inferring it.
• Behavior tasks (broken / slow / wrong-shape responses, anything the user observed): MCP can supplement but rarely substitutes for exercising the endpoint yourself. A curl against the predict URL with a representative payload runs the same code path the user does. Absence of recent errors in MCP-fetched logs is not proof the bug is gone — log windows are bounded, and a problem the user reported five minutes ago may have already aged out of the default tail. For a specific historic incident you can't reproduce, log tools support custom time ranges within retention; use them.
• Treat cheap introspection as budget freed up for deeper verification (re-curl after a fix, diff configs before and after a PATCH, push a candidate truss and probe it), not as license to stop earlier.