Managed Model Endpoints

Managed model endpoints

A managed model endpoint is a model service the daemon owns: you register it once, then every compute_provider cell against it just works — the daemon swaps the resident model off the device (one model at a time, via the resident's own approved stop), runs your approved start script, waits for the readiness route, then runs your cell, streaming its lifecycle progress into the cell as it goes. You never run the container runtime yourself, never poll readiness in cells, and never see the credential value. Two verbs: register() (asks the user once) and ordinary inference cells.

Container specifics — image, registry login, internal port, cache mount target, readiness route — come from the model's own runbook skill; this skill is the translation contract.

Calling a registered endpoint — inference cells

Calling a registered endpoint — use the using-model-endpoint skill (this skill is the REGISTRATION contract; that one documents the call side in full).

The ONLY dispatch form is the compute_provider tool with the endpoint's registered name (list_compute shows them):

compute_provider(provider="boltz2-service", code="""
import requests
r = requests.post(BASE_URL + "/v1/infer", json=payload)
""")

The daemon brings the model up on demand (a first cold start downloads image + weights — minutes; let it run) and preloads BASE_URL into the cell — both as a Python variable (use it directly, as above) and as os.environ["BASE_URL"] (plus INFER_API_KEY for remote endpoints). Endpoints are not kernel environments: environment="boltz2-service" on a plain python cell fails — plain cells get no BASE_URL.

Enablement — once per machine

The user connects the family under Customize → Compute → Model endpoints (the setup flow saves the family credential first — connect-without-key is not a state) and picks ONE mode: Local (container registrations) or URL/remote (https against the configured host). Until connected, free_port()/register() raise a precise error — relay it; in the wrong mode they refuse with a teaching error naming the setting (existing endpoints of the unarmed leg keep dispatching — only NEW registrations refuse). Disconnecting is a full teardown: every active local service is stopped via its approved stop script and every registration (local AND hosted) is removed; caches stay on disk; a failing stop keeps that one row, FAILED. The "Local machine GPU" toggle never gates registration — it governs cell GPU access only; the approval card is the per-registration gate.

Credential contract (platform rule): every registration passes credential="NVIDIA_API_KEY" — the daemon rejects any other name. Locally the value feeds the start script's registry login and never enters your kernel env; for remote endpoints it authenticates the upstream and is delivered only into the inference cell's env (as INFER_API_KEY), never the repl kernel.

Register (repl kernel)

port = host.model_endpoints.free_port()        # local only; random 20000-29999

host.model_endpoints.register(
    name="boltz2-service",            # <model>-service -- descriptive, never the
                                      # bare model name (collides/ambiguous)
    url=f"http://127.0.0.1:{port}",   # LITERAL 127.0.0.1 -- `localhost` rejected
    credential="NVIDIA_API_KEY",      # the family credential NAME, never a value
    skill="<model-runbook-skill>",
    start=START_SCRIPT,               # composed below
    stop="docker stop boltz2-service",# exit 0 ONLY once actually stopped
    live="/v1/health/ready",          # readiness ROUTE (200 = model answers;
                                      # "up but loading" must read not-ready)
)

Name endpoints <model>-service (e.g. diffdock-service) — unambiguous in provider lists; never just the bare model name. Name the CONTAINER after the endpoint too (the template above does): the UI then follows the service's own logs live while it starts.

register() always cards the user (scripts verbatim, port, service dir, credential name). One exception: a byte-identical re-registration is silent — same bytes are approved forever; any byte change re-cards. The registration stays inspectable under Customize → Compute. Re-registering to fix scripts: reuse the existing url — never call free_port() again (the port is the endpoint's stable mutex).

Remote endpoints — upstream APIs (no lifecycle)

Pass url="https://<upstream>" and omit start/stop/live — no port, no scripts, no readiness. Requires URL/remote mode (the setup radio; in Local mode https registrations refuse). The url's HOST must equal the configured upstream host exactly — you pick the path leaf, never the authority. After approval, cells are plain HTTP clients of BASE_URL authenticating with $INFER_API_KEY. list_compute labels every row location: "local" | "remote".

Composing the start script

The daemon hands scripts three things in their process environment (never argv, never sudo): HOST_PORT (the registered port), SERVICE_DIR (this endpoint's persistent directory — put the model cache here), and the credential value under its own name. Nothing else is inherited — ambient tokens are not visible; the ONLY secret a script sees is its registered credential.

The start script must be idempotent (cold create / warm start / crash re-entry), with the port-mismatch guard — the runtime freezes port mappings at container creation, so a container created under an OLD port must be recreated or readiness can never pass:

mkdir -p "$SERVICE_DIR/cache"
# docker login persists auth in $DOCKER_CONFIG/config.json; scope it to the
# service dir so the credential dies with the service (never ~/.docker).
export DOCKER_CONFIG="$SERVICE_DIR/.docker"
create_service() {
  docker run -d --name boltz2-service \
    --restart unless-stopped \
    -p 127.0.0.1:${HOST_PORT}:8000 --gpus all \
    -e NVIDIA_API_KEY \
    -v "$SERVICE_DIR/cache:<cache target from the runbook>" \
    <image from the runbook>
}
if docker inspect boltz2-service >/dev/null 2>&1 && \
   [ "$(docker inspect -f '{{(index (index .HostConfig.PortBindings "8000/tcp") 0).HostPort}}' boltz2-service)" != "$HOST_PORT" ]; then
  docker rm -f boltz2-service          # stale port mapping -- recreate below
fi
if docker inspect boltz2-service >/dev/null 2>&1; then
  docker start boltz2-service          # warm wake -- no credential, no chown needed
else
  echo "$NVIDIA_API_KEY" | docker login <registry> --username '<user>' --password-stdin
  docker pull <image from the runbook>
  # Cache must be writable by the CONTAINER's user, whose uid the image
  # defines (container uid != host uid). chown needs root the script doesn't
  # have; a throwaway root container does it -- and the chmod, which the
  # host user can no longer do once the dir is chowned away -- without sudo.
  CUID="$(docker inspect --format '{{.Config.User}}' <image from the runbook> 2>/dev/null | cut -d: -f1)"
  case "$CUID" in ''|root) CUID=0;; *[!0-9]*) CUID=1000;; esac   # named user -> default 1000; runbook may override
  if [ "$CUID" != "0" ]; then
    docker run --rm -v "$SERVICE_DIR/cache:/c" alpine sh -c "chown -R $CUID:$CUID /c && chmod 700 /c"
  else
    chmod 700 "$SERVICE_DIR/cache" 2>/dev/null || true
  fi
  create_service
fi
# RUNTIME-binding guard (one retry): after a port-conflict crash the engine
# can start the container yet silently skip port programming -- the CONFIG
# still matches $HOST_PORT (so the guard above cannot catch it) but
# `docker port` prints nothing and the model serves to nobody. Recreate.
if [ -z "$(docker port boltz2-service 2>/dev/null)" ]; then
  docker rm -f boltz2-service
  create_service
fi

Translation rules:

• Keep scripts ASCII — non-ASCII (em dashes, arrows, curly quotes) triggers the approval card's spoofing warning; use -- and -> in comments.
• export DOCKER_CONFIG="$SERVICE_DIR/.docker" before any docker login — login persists the credential in config.json, and scoping it to the service dir means Remove honestly reclaims it (never ~/.docker, which outlives stop/Remove/Disable).
• -p 127.0.0.1:${HOST_PORT}:<internal> — loopback-only publish; the internal port comes from the runbook.
• If the image reads a different env name, bridge env→env at the top: export OTHER_NAME="$NVIDIA_API_KEY" (never argv, never a file).
• -e NAME bare (argv is world-readable); the key rides the login stdin pipe only.
• -d, no --rm — managed containers are stopped, never removed: stop parks them with weights loaded; --rm throws the cache away.
• Cache under $SERVICE_DIR, owned by the container's uid: the runbook states it when it matters; otherwise derive it post-pull with docker inspect --format '{{.Config.User}}' <image> (empty or root ⇒ runs as root, no chown needed; a NAMED user can't be resolved without running the image — default 1000). Getting it wrong is the cache-empty symptom: the container can't write the mount, weights leak into the writable layer and die on recreate (or the image crash-loops on Permission denied). Never 777 — world-writable cache on a multi-user host. The mount TARGET comes from the runbook.
• Cells need no auth header against local endpoints — the credential is a pull key that never enters your kernel.

Failures

A failed start/stop flips the endpoint FAILED (transcript on the endpoint panel — never echoed into cell errors; ask the user to read it there) and your cell errors with the daemon's one-line cause. FAILED is sticky: further cells fail fast until the user presses Stop or you re-register (byte-identical re-register also clears it). If a stop is stuck (exit 0 but the port never frees), removal is refused while the port is bound — recover out-of-band; the daemon absorbs the freed port on its next probe. A first-ever cold start downloads image + weights — minutes, once; the cell streams the phase lines live and the endpoint detail view streams the full script output, so let it run.