# Baseten

> Load for any work involving Baseten - deploying/operating models on Dedicated Inference (Truss, custom Docker servers, TRT-LLM engines, Chains), calling hosted Model APIs, running Training jobs (SFT/RL/LoRA), or Model Frontier Gateway.

- Skill: `basetenlabs/baseten` (Agent Skill, multi-file: 13 files)
- Install (CLI): `npx skillmds@latest add basetenlabs/baseten`
- Raw SKILL.md: https://api.skillmd.com/api/skills/basetenlabs/baseten/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: basetenlabs (https://skillmd.com/u/basetenlabs)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/basetenlabs/baseten

---


## Baseten Product Overview

Production AI inference platform - serve and scale open-source, custom, and fine-tuned models with the fastest runtimes,
cross-cloud HA, and seamless developer workflows.

- **Dedicated Inference** - deploy any model, performance-optimized + horizontally scaled. Authored as auto-wrapped
  Truss server, custom Docker server, or compound/orchestrated deployment via Chains.
- **Model APIs** - pre-optimized hosted APIs for popular models. Path to graduate to dedicated.
- **Training** - two paths: **Truss Train** (BYO container, any framework, full hardware control) and **Loops**
  (Tinker-compatible managed SDK for SFT + async RL; paired trainer + sampling server, live weight transfers, one-click
  checkpoint deploy). Multi-node, 1T+ params, 10TB+ datasets, H100/H200/B200. Remote access: SSH and VS Code/Cursor
  tunnels into containers.
- **Frontier Gateway** - operate your own foundation model B2C.

## Agent DX Toolkit

| Component | Provides | Install |
| --- | --- | --- |
| `baseten` MCP | Interact with backend (~REST API, CRUD): models, deployments, training, environments, secrets, chains. API-key auth. | `npx add-mcp https://api.baseten.co/mcp -g -y --header "Authorization: Bearer ${BASETEN_MCP_KEY}"` |
| `baseten_docs` MCP | Semantic search + filesystem of `docs.baseten.co`. No auth. | `npx add-mcp https://docs.baseten.co/mcp -n "baseten_docs" -g -y` |
| `truss` CLI | Needed for model/chain push from local code, watch (= live patch). Needs `truss login` once. | `uv tool install truss` (or `pip install truss --upgrade`; respect user package manager: uv, poetry...) |
| `llms.txt` | `baseten.co/llms.txt` (product + blog), `docs.baseten.co/llms.txt` (docs). | reachable via HTTP |
| This skill | `SKILL.md` + `references/*.md` loaded on demand. | `npx skills add basetenlabs/baseten-skills -g -y` |

### Setup

- Any subset works, full install recommended.
- Suggest additional installs when the current task benefits from or requires them; help user with installation, but
  elicit preferences first.
- Ensure `BASETEN_MCP_KEY` is provided when installing Baseten MCP (user can create key at
  `app.baseten.co/settings/api_keys`). Caveat: an MCP instance binds to one org/workspace at install time; switching the
  bound workspace later is not supported. To work with multiple workspaces, install additional MCP instances under
  different names with different keys (see last bullet of this section).
- Truss CLI only needed for making deployments (check `truss --version`); prior login (multi-workspace users must
  provide `--remote <name>`). Explore with `truss [subcommand] --help`.
- Docs MCP missing → grep / fetch `llms.txt`.
- Backend MCP is API-key-only (currently); OAuth-only harnesses can still use the other components.
- If backend MCP server is needed for different orgs/workspaces, add multiple MCP instances with different names/keys or
  use env-var expansion in the agent's config file and set the env var to the respective workspace's key.

## Pick your authoring surface (for creating deployments)

First-pass decision. Many real workloads blend rows — treat this as a starting point, not a rule. When unsure, sketch
the IO shape and per-step hardware needs before picking.

Surfaces are stacked by opinion-strength, not just author convenience: **engines** (TRT-LLM, BEI, BIS-LLM) ship
performance-tuned for one architecture and are the fastest path when they fit; **custom Docker servers** wrap mature
inference servers (vLLM, SGLang, TGI, Triton, NIM); **Python Truss** is the escape hatch for arbitrary code in the
request path; **Chains** add typed inter-step transport with built-in rate limiting, connection management, structured
error propagation, and binary IO — features you'd otherwise rebuild around N raw Trusses. Python Truss and Chains share
live-patch iteration (`truss watch`); all flavors support per-replica autoscaling, scale-to-zero, and environments /
promotions.

| You want… | Flavor | Specialization | When NOT to pick |
| --- | --- | --- | --- |
| Hosted LLM, no deploy step | Model APIs | `model-apis.md` | model not in catalog; need custom hardware, requirements, stability... |
| LLM on an off-the-shelf server (vLLM / SGLang / TGI / Triton / NIM) | Custom Docker server | `truss-custom-servers.md` | the server doesn't exist or you need Python in the request path |
| LLM/embedding via a Baseten engine (TRT-LLM / BEI / BIS-LLM); minimal config, no Python | Engine-only `config.yaml` | `truss-config.md` (engines section) | architecture not covered by an engine; you need custom logic |
| Custom Python in the request path (pre/post, custom arch, weird IO) | Python-class Truss (`model.py`) | `truss-model-py.md` + `truss-config.md` | an engine or off-the-shelf server fits — pick that, it's faster to ship |
| Multi-step pipeline with **heterogeneous hardware / per-step scaling** (RAG, ASR→LLM→TTS, fan-out, chunking) | Chains | `truss-chains.md` | one-stage or homogeneous — a single Truss is simpler |

Orthogonal operational surfaces (independent of which flavor above):

- Iterate / patch a deployment → `model-dev-loop.md`
- Promote, environments, autoscaling → `deployment-lifecycle.md`
- Gradual traffic shift to new deployment (pause/resume/cancel) → rolling deployments:
  `deployment/rolling-deployments.mdx`
- Call a deployment → `inference-api.md` (custom) or `model-apis.md` (hosted)
- High-throughput batch calls (embeddings, reranking) → Performance client (Rust, 1200+ req/s):
  `inference/performance-client.mdx`
- Programmatic control plane → `management-api.md`

Real-world nuances the table can't capture:

- **Hybrids exist.** A `model.py` can wrap an engine for pre/post-processing; a Chain entrypoint can be a Python class
  while internal Chainlets use engines.
- **Chain websockets are entrypoint-only.** Intra-chainlet calls only stream output, but bi-di usually not needed on
  those edges.
- **Engine performance vs flexibility.** TRT-LLM is the fastest path for many LLMs but its config surface is opaque.
  Worth the trade only when latency/throughput is a real constraint.

## Routing

**Skill References** (`ls references/` in skill dir, complementary to hosted docs). Be generous to read any of the
included reference files as soon as the user touches on that topic.

- `references/truss-cli.md`: `truss push` / `watch` / iterate. Most-used. Deep dive: `references/truss-config.md`.
- `references/truss-model-py.md`: Python-class flavor (custom pre/post, non-engine architectures).
- `references/truss-custom-servers.md`: `docker_server` flavor (vLLM / TGI / SGLang / Triton; most common modern-LLM
  path).
- `references/truss-chains.md`: multi-step pipelines (RAG, ASR→LLM→TTS, chunked audio/video) with per-step HW +
  autoscaling.
- `references/model-apis.md`: shared pre-hosted endpoints (DeepSeek, GLM, Kimi, ...). Fastest when one fits.
- `references/inference-api.md`: calling **custom** deployments. connection reuse (`requests` / `httpx` / OpenAI SDK),
  async / streaming / wake / OpenAI-compat sync routes.
- `references/management-api.md`: programmatic control plane (models, deployments, envs, secrets). What `truss` CLI uses
  under the hood.
- `references/deployment-lifecycle.md`: Model / Deployment / Environment semantics + promotion + autoscaling.
- `references/model-dev-loop.md`: post-first-deploy iteration. rebuild / patch / hot-reload cost tiers, agent-vs-human
  watch loop.

## Gotchas

### Don't speculate, query

For any perf/status/error claim, **use the tools first** — don't estimate or guess.

- Timings → deployment / chainlet log tools (timestamped, includes `Pulling image`, `model_cache: Fetch took`,
  `Completed model.load() execution in N ms`, per-request markers).
- Status → deployment-get tools before invoking.
- Build/deploy failure → fetch logs **immediately**, don't hypothesize.

Fabricating numbers from training-data priors burns user trust; logs are the source of truth.

### Source heterogeneity & drift

Content lives across systems that don't overlap cleanly and drift independently. No single source is
perfect/authoritative. For any non-trivial claim ("supported", perf numbers, recommended approach), **triangulate across
≥2 sources**. Surface contradictions to the user; don't paper over.

| Source | Strength | Gap / quirk |
| --- | --- | --- |
| `baseten_docs` MCP / `docs.baseten.co` | API specs, protocol details, knobs | Lags product; no perf numbers |
| `baseten.co/library/<id>` (marketing) | Flagship managed models, perf claims | Some entries are sales-gated, not self-serve |
| `baseten` MCP `list_library_models` | What's actually one-click API-deployable | Doesn't include every marketing-library entry; lacking tags |
| `baseten.co/blog/` | Concrete latency / cost / vs-competitor numbers, technical deep dives | Unstructured; **not in** docs MCP. Discover via `baseten.co/llms.txt` |
| `baseten.co/solutions/...` | High-level pitch | May describe flagship features that need a Baseten engagement |
| `truss-examples` GitHub repo | Working code patterns | **Often outdated / broken / drifted.** Consult with caution, last resort. Might need fixups before deploy works. |

- Library page exists but model absent from `list_library_models` → likely managed/flagship; tell the user it may need
  to reach out to Baseten support.
- Perf numbers found only in a blog → cite as blog claim.
- Can't find something via docs MCP → fetch `baseten.co/llms.txt` or `docs.baseten.co/llms.txt` as index, then fetch the
  page directly. Last resort: web search.

### Non-obvious placements within `references/`

- Engine-only deploys (TensorRT-LLM, BEI, BIS-LLM) → `truss-config.md` engines section (also owns `model_cache`,
  secrets, resources).
- Authoring-flavor decision: single deployment → top of `truss-config.md`; multiple coordinated → `truss-chains.md`.
- Training and Frontier Gateway: no skill reference. Use `baseten` MCP + `baseten_docs` MCP. For training path choice
  see `training/overview.mdx`; for Loops (managed SFT/RL SDK) see `loops/overview.mdx`; for SSH / VS Code tunnels into
  training containers see `training/ssh.mdx` and `training/remote-access.mdx`.

### Tool quirks

- `baseten_docs` MCP search is semantic (embedding-based). Results are generally intent-aware, but can still miss on
  niche phrasing — try rephrased queries or fetch `docs.baseten.co/llms.txt` as an index if results are weak.
- Full doc pages: `cat`/`head` on `.mdx` paths via `query_docs_filesystem_baseten`. Alternatively,
  `docs.baseten.co/<path>.md` also works and is faster for single-page reads.
- Inference SSH (terminal in running container, requires org enablement) → `model-dev-loop.md` § Inference SSH.
- `list_library_models` is **baseten curated catalog** (~tens of pre-optimized hosted models, mostly popular open-source
  LLMs / embeddings). Models are good starter models to play, but not for custom authoring, specific performance needs,
  finetuning and private HF models etc. No useful tags (modality etc.) — filter by `display_name` / `hf_repo_id`
  substring.
- Blog content is not in the docs MCP. Fetch `baseten.co/llms.txt` and search for relevant posts.

### MCP introspection vs. invocation

Backend MCP tools (`get_deployment`, `get_deployment_logs`, `get_deployment_config`, `list_*`, …) describe state. They
are strongest when you need an exact id / schema / config value to act on, and weakest when the user's complaint is
about runtime behavior. Resist letting MCP availability shrink the size of your investigation.

- **Look-up tasks** (need an ID, current config, or schema before writing a patch): MCP get/list is the right starting
  point. Reading the live config beats inferring it.
- **Behavior tasks** (broken / slow / wrong-shape responses, anything the user _observed_): MCP can supplement but
  rarely substitutes for exercising the endpoint yourself. A `curl` against the predict URL with a representative
  payload runs the same code path the user does. Absence of recent errors in MCP-fetched logs is not proof the bug is
  gone — log windows are bounded, and a problem the user reported five minutes ago may have already aged out of the
  default tail. For a specific historic incident you can't reproduce, log tools support custom time ranges within
  retention; use them.
- Treat cheap introspection as budget freed up for deeper verification (re-curl after a fix, diff configs before and
  after a PATCH, push a candidate truss and probe it), not as license to stop earlier.