Replicate

Replicate platform operations

This skill is about how a model runs in production on Replicate — clients, async, deployments, Cog packaging, webhooks, scaling, and spend. It is the platform-engineering counterpart to image prompt craft. If the question is what prompt, aspect ratio, or model family produces a good image, that is replicate-images, not this skill. Here the mental model is: a prediction is a job. You either wait for it, poll it, or get pinged about it — and where it runs (shared cold pool vs a private warm deployment) is a cost-and-latency dial you set deliberately.

Pinned facts (verified 2026-06-02): Python client replicate 1.x (latest 1.0.7), Python 3.8+; JS client replicate on npm; auth via REPLICATE_API_TOKEN. A 2.0.0aN alpha exists on PyPI but is NOT the default — pin replicate>=1,<2 so a fresh install never silently pulls it.

Decision: how should this model run?

Pick the row by latency tolerance and whether your process can block. Do not default to run() for everything — a 10-minute job inside a web request will time out and burn a worker.

Rule: if a human or HTTP request is waiting longer than a few seconds, do not block on run() — switch to predictions + webhook. Why: synchronous timeouts kill the request but the GPU job keeps running and billing.

Auth & install

export REPLICATE_API_TOKEN=r8_...           # clients read this env var automatically
pip install 'replicate>=1,<2'               # pin: 2.0.0aN is alpha; unpinned can pull it
npm install replicate                        # Node client

Python 3.8+ is required. Never pip install replicate unpinned in a Dockerfile or requirements file — a rebuild months later can resolve to the 2.0 alpha and break your imports. Why: the alpha is a full Stainless/httpx rewrite with a different surface.

Run a model

import replicate

output = replicate.run(
    "black-forest-labs/flux-schnell",
    input={"prompt": "a red bicycle", "num_outputs": 1},
)

Since client 1.0.0, file outputs come back as FileOutput objects, not URL strings. Treating one as a string is the single most common bug.

# Bad — output[0] is a FileOutput, not a str; this writes the repr, not the bytes
open("out.png", "w").write(output[0])

# Good — read the bytes, or take the URL explicitly
with open("out.png", "wb") as f:
    f.write(output[0].read())          # bytes
print(output[0].url)                    # hosted URL if you'd rather link

Rule: call .read() for bytes or .url for the link. Why: silently coercing a FileOutput to a string corrupts the file and the error surfaces far from the cause.

Long-running & async

For jobs over a few seconds, create a prediction instead of blocking:

client = replicate.Client()
prediction = client.predictions.create(
    model="owner/model",
    input={"prompt": "..."},
)
prediction.reload()                      # refresh status from the API
while prediction.status not in ("succeeded", "failed", "canceled"):
    time.sleep(2)
    prediction.reload()

Set a deadline so a stuck job auto-cancels instead of billing forever, and cancel() on cleanup paths. Why: a hung prediction with no deadline is silent, open-ended GPU spend. Poll with a small backoff, not a tight loop — you are charged for the prediction, not the polling, but a tight loop wastes your own process and rate budget. Full polling loop with backoff and 5xx handling is in references/webhooks-and-async.md.

Webhooks

For fire-and-forget, hand Replicate a URL and filter to the events you care about:

client.predictions.create(
    model="owner/model",
    input={"prompt": "..."},
    webhook="https://your.app/hooks/replicate",
    webhook_events_filter=["completed"],   # not every intermediate "logs" event
)

Rule: ALWAYS verify the signature before trusting a webhook body. Why: the URL is public — anyone can POST forged completions to it. Replicate signs each delivery; the secret has a whsec_ prefix. Reconstruct signed content as {webhook-id}.{webhook-timestamp}.{body}, HMAC-SHA256 with the base64-decoded secret, base64-encode, and constant-time compare against the webhook-signature header (a space-separated v1,<sig> list). The clients expose a verification helper; the full Python and Node recipe (including idempotency via webhook-id) is in references/webhooks-and-async.md.

Deployments

A deployment is a private, dedicated API endpoint for one model version that autoscales from zero to hundreds of instances. Reach for it when you need warm instances, a private endpoint, or a hard spend cap — not for one-off runs.

• min_instances — the warm floor. Set >0 to kill cold starts for latency-sensitive traffic; every warm instance bills whether or not it serves a request.
• max_instances — the spend cap. The ceiling on concurrent instances; protects you from a traffic spike turning into a surprise bill.
• Hardware (NVIDIA T4, A100, H100, ...) is switchable without touching code — change the deployment config, not predict.py.
• Rolling updates ship a new version with no downtime; built-in monitoring covers latency, throughput, error rate, and GPU memory.

You still call a deployment with run() / predictions.create() — it just routes to your private instances. Create/update via HTTP API, the clients, or CLI; fields and the rolling-update flow are in references/deployments-api.md.

Package a custom model with Cog

Cog packages a model into a production container. You need two files; Replicate builds the API server for you on push.

# cog.yaml
build:
  gpu: true
  python_version: "3.11"
  python_packages:
    - "torch==2.4.0"
predict: "predict.py:Predictor"

# predict.py
from cog import BasePredictor, Input, Path

class Predictor(BasePredictor):
    def setup(self):
        # load weights ONCE here, not per request
        self.model = load_model("weights.pth")

    def predict(self, prompt: str = Input(description="text prompt")) -> Path:
        result = self.model(prompt)
        return Path(result)            # Cog uploads the file

cog predict -i prompt="hello"          # run locally (needs Docker)
cog push r8.im/owner/model             # build + push; Replicate hosts it

Rule: load weights in setup(), never in predict(). Why: setup() runs once per instance; predict() runs every request — loading weights per request makes every call pay the model-load cost. Full cog.yaml (system packages, run steps), typed Input(...), GPU config, version pinning, and common build failures are in references/cog-packaging.md. Building requires Docker.

Cost & scaling levers

• Scale to zero is the default — idle deployments stop billing. Only raise min_instances above zero when cold-start latency actually hurts users, and treat the warm floor as a line item.
• Deadlines on predictions cap the worst case so a stuck job cannot bill open-ended.
• Right-size the GPU: do not run a 1B model on an H100. Hardware is a deployment-config change.
• Cache weights in setup() so per-request work is just inference.

This skill covers Replicate-specific levers only. Tracking total AI spend across many providers as a discipline is cost-tracking.

Anti-patterns

References

• references/cog-packaging.md — full cog.yaml + predict.py, GPU config, build/push, version pinning, build failures.
• references/webhooks-and-async.md — signature verification (Python + Node), event filters, idempotency, polling with backoff, 5xx retries.
• references/deployments-api.md — create/update/get deployment via HTTP + clients, autoscaling fields, rolling updates, monitoring metrics, CLI.

scripts/verify.sh statically checks an emitted artifact dir: a cog.yaml declaring build: and predict:, a predict.py Predictor with setup + predict, and warns on an unpinned replicate dependency or a FileOutput written as a string. Presence + key checks only — it does not run Docker.