Bentoml Ec2 Deploy

Deploy a BentoML service to plain EC2 instances

For production / CI-CD, generate a standalone script bundle (no agent at deploy time) with the bentoml-deploy-scriptgen skill.

In: a pushed image ref from bentoml-containerize (e.g. 123456789012.dkr.ecr.us-east-1.amazonaws.com/summarization:v1). Out: it running under Docker on one or more EC2 instances, verified. Docker is the instance's only dependency — no Kubernetes, no agents.

Image facts (bentoml containerize):

• Serves HTTP on port 3000. /livez liveness, /readyz readiness, /metrics Prometheus, / Swagger UI.
• The entrypoint starts the server — never pass a command to docker run.
• Models are usually baked in, so images run multi-GB; size volumes to match. Runtime model downloads need env vars such as HF_TOKEN via -e.

Safety rules, no exceptions:

• Before EVERY mutating AWS CLI command (run-instances, create-security-group, authorize-security-group-ingress, create-key-pair, iam create-*/put-*/add-*, associate-iam-instance-profile, terminate-instances, delete-*): show the exact command + a cost note (hourly/monthly, or free), then wait for explicit confirmation. Read-only (describe-*, get-*, sts get-caller-identity, ssm get-parameters, wait) runs freely.
• Mutating SSH commands echo their host first: echo ">>> on $HOST:" ....
• Never terminate, stop, reboot or modify instances, security groups, key pairs or IAM resources this skill did not create this session. Attaching an instance profile to an existing instance counts as modifying it.
• Track every ID you create (instances, SG, key pair, IAM). Teardown touches only that list.
• Secrets reach the container as -e NAME="$NAME" from the local shell env — never a file, user-data or a saved command.
• Always sudo docker ... over SSH: usermod -aG docker misses open sessions and races fresh instances.

Step 0 — Preflight

ssh -V must succeed. From the bentoml-containerize handoff, or ask:

1. Image ref — full, with registry and tag.
2. Registry access — ECR / other private / public. Picks Step 2.
3. Runtime env var names; values come from the user's shell env.
4. Image arch (amd64/arm64) — must match the instance (t3.*/m5.*/c5.* amd64, t4g.*/m7g.* arm64), else exec format error.
5. Container/service name — bento name, underscores → hyphens (my_service → my-service).
6. Mode — A: user's existing instance(s) over SSH. B: provision new via AWS CLI.
7. ECR auth path (ECR only, decide now, see references/registry-auth.md):
   • Instance profile (preferred for Mode B) must exist before launch, to attach at run-instances. Needs IAM writes, which PowerUserAccess-style SSO roles usually lack despite full EC2/ECR. Probe: aws iam get-role --role-name nonexistent-probe 2>&1 — NoSuchEntity = IAM read works; AccessDenied = pick token-over-SSH now. On AccessDenied from the first IAM create (Way 1), fall back; nothing was created.
   • Token over SSH: nothing to create or install, and the only path that leaves a Mode A instance unmodified.

AWS CLI checks — Mode B and any ECR image. Skip for Mode A + non-ECR image: that path runs no aws command, so don't block on them.

aws --version                       # AWS CLI v2 installed?
aws sts get-caller-identity         # credentials valid? note the Account
aws configure get region            # suggest as default, then ASK — never assume the region

--region "$AWS_REGION" goes on every later aws command. For ECR, the region in the ref (<account>.dkr.ecr.<region>.amazonaws.com/...) must match the login commands.

Step 1A — Existing instance(s) (SSH mode)

Collect HOST (public IP/DNS), SSH_USER (ec2-user Amazon Linux, ubuntu Ubuntu), KEY (private key path), then set the array used everywhere below:

SSH=(ssh -i "$KEY" -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 "$SSH_USER@$HOST")
"${SSH[@]}" 'echo connected; uname -m;
  if ! command -v docker >/dev/null 2>&1; then echo NO_DOCKER;
  elif ! sudo docker info >/dev/null 2>&1;  then echo DAEMON_DOWN;
  else sudo docker info --format {{.ServerVersion}}; fi'

• uname -m: x86_64 = amd64, aarch64 = arm64 — must match the image arch (Step 0.4).
• NO_DOCKER: offer to install (a mutation — echo host, confirm). AL2023: sudo dnf install -y docker && sudo systemctl enable --now docker. Ubuntu: sudo apt-get update && sudo apt-get install -y docker.io && sudo systemctl enable --now docker.
• DAEMON_DOWN: offer sudo systemctl enable --now docker (also a mutation). enable, not just start: --restart unless-stopped needs the daemon back after a reboot.
• Check the security group allows your verification path: 3000 from the user's IP, or an SSH tunnel (Step 4). Never edit an existing instance's SG — name the missing rule, let the user add it, or tunnel.

Multiple hosts: collect all now; Steps 2–4 loop per host, with a ready-made loop in references/multi-node.md.

Step 1B — Provision new instance(s) (AWS CLI mode)

Follow references/provisioning.md fully; every mutating call gets command + cost note + confirmation.

1. AMI from SSM public parameters (reference §4) — never hardcode or guess AMI IDs. AL2023 → ec2-user, Ubuntu 24.04 → ubuntu. Show the resolved ami-....
2. Key pair: reuse the user's, or create one (free).
3. Security group (free): 22 from the user's IP (detect and confirm it — VPNs skew it) and port 3000 at a scope the user picks: their IP (default), tunnel-only (no 3000 rule), or 0.0.0.0/0 — that last only after warning that it exposes an unauthenticated inference API to the whole internet, plus explicit confirmation.
4. ECR instance profile (if Step 0 chose it): create role + profile per references/registry-auth.md now; it must exist to be attached at launch, and there is no later attach step.
5. Instance: default t3.medium (~$0.04/hr on-demand us-east-1, region-dependent — always state the estimate; the public IPv4 address adds $0.005/hr), t4g.medium for arm64. Size RAM to the model, and the root volume to the image: the 8 GiB AMI default is too small for most bentos — 30–50 GiB gp3, ~$0.08/GB-month. User-data installs and enables Docker. Tag everything managed-by=bentoml-ec2-deploy.
6. Wait and connect: aws ec2 wait instance-running, public IP as HOST, SSH_USER=ec2-user (AL2023; ubuntu for Ubuntu AMIs), the Step 1A SSH=(...) array, then retry SSH until cloud-init finishes installing Docker.

Record INSTANCE_ID, SG_ID, key pair name, IAM names — the complete teardown list.

Step 2 — Registry auth on the instance

Execute the Step 0 path per references/registry-auth.md:

• ECR + instance profile: the instance logs in with its own role (IAM done in Step 1B).
• ECR + token over SSH: pipe aws ecr get-login-password into docker login on the host. Tokens expire after 12 hours — matters for redeploys.
• Other private registry (Docker Hub, GHCR): pipe the token from the local env into docker login over SSH.
• Public image: skip.

Step 3 — Pull and run

Per host, echoed first because this mutates the instance:

echo ">>> on $HOST: pulling $IMAGE"
"${SSH[@]}" "sudo docker pull '$IMAGE'"

echo ">>> on $HOST: starting container $SERVICE_NAME"
"${SSH[@]}" "sudo docker run -d --name '$SERVICE_NAME' --restart unless-stopped -p 3000:3000 '$IMAGE'"

• A container already named $SERVICE_NAME: do not remove it unless this skill created it this session or the user confirms — show sudo docker ps -a --filter name=$SERVICE_NAME, ask.
• Env vars: -e HF_TOKEN='$HF_TOKEN' style flags before the image ref. Double quotes around the remote command expand values from the local env; never a file or user-data.
• --restart unless-stopped survives crashes and reboots if the Docker service is enabled (user-data does that).
• -p <host_port>:3000 if 3000 is taken on the host; the container side is always 3000.
• GPU: add --gpus all (needs NVIDIA driver + nvidia-container-toolkit on the host, not installed here).

Confirm it stayed up; crash-loops show Restarting. sleep 3 is load-bearing: a container that dies 2 seconds in still reports Up less than a second right after docker run -d.

sleep 3
"${SSH[@]}" "sudo docker ps --filter name='$SERVICE_NAME' --format '{{.Status}}'"   # must show "Up ..."

Step 4 — Verify: /readyz + one real inference request

Match the port-3000 scope from Step 1. Direct (3000 open to your IP): probe http://$HOST:3000. SSH tunnel (no 3000 rule, or direct probing fails): forward an uncommon local port (3200) — local 3000 often hosts a dev server, and a tunnel that fails to bind sends curls to the squatter, faking success. Keep it to one shell invocation: the background PID dies with the shell.

TUN_ERR=$(mktemp)
# ExitOnForwardFailure=yes is load-bearing: without it ssh prints "Could not request
# local forwarding." and KEEPS RUNNING when local 3200 is taken — the kill -0 guard
# would pass while curls hit the squatter.
ssh -i "$KEY" -o StrictHostKeyChecking=accept-new -o BatchMode=yes \
  -o ExitOnForwardFailure=yes \
  -N -L 3200:127.0.0.1:3000 "$SSH_USER@$HOST" >"$TUN_ERR" 2>&1 &
TUNNEL_PID=$!
BASE=http://127.0.0.1:3200
OK=""
for i in $(seq 1 60); do            # model loading can take minutes
  kill -0 $TUNNEL_PID 2>/dev/null || { echo "tunnel exited:"; cat "$TUN_ERR"; rm -f "$TUN_ERR"; exit 1; }
  curl -sfo /dev/null "$BASE/readyz" && { OK=1; break; }
  sleep 5
done
[ -n "$OK" ] || { echo "not ready after 5 min; tunnel log:"; cat "$TUN_ERR"; rm -f "$TUN_ERR"; kill $TUNNEL_PID; exit 1; }
echo READY

# Real inference request — route/payload from service.py (@bentoml.api methods are
# POST /<method_name> with a JSON body of the parameters). No source? Fetch
# $BASE/docs.json for the schema, then re-run with the call filled in:
curl -s -X POST "$BASE/summarize" \
  -H 'Content-Type: application/json' \
  -d '{"text": "EC2 is a web service that provides resizable compute capacity."}'

kill $TUNNEL_PID
rm -f "$TUN_ERR"

Direct path: same loop and call with BASE=http://$HOST:3000, no tunnel, no PID checks — but re-check sudo docker ps over SSH first, so you know your container is the responder.

Judge the response body, not the status code. Only a plausible, correct result proves the deployment. HTTP 200 proves nothing; an error body means the payload or the service needs fixing; output unrelated to the API means a port squatter (tunnel) or a stale host process. Verify every host individually — a fleet with one dead node still answers most probes.

Step 5 — Tell the user how to reach the service

• http://<public-ip-or-dns>:3000, only from IPs the SG allows. / Swagger UI, /metrics, /livez, /readyz.
• Tunnel-only: ssh -i <key> -N -L 3000:127.0.0.1:3000 <user>@<host> → http://127.0.0.1:3000.
• The public IP changes on stop/start; for a stable address use an Elastic IP (aws ec2 allocate-address) or DNS. Since Feb 2024 AWS bills every public IPv4 address $0.005/hr (~$3.65/mo), attached or not — an EIP costs the same as the auto-assigned IP (stability is free), but an unattached EIP bills for nothing.
• Raw HTTP is fine for testing; for production suggest TLS in front (nginx/caddy, or an ALB). Say plainly: the service has no authentication unless they built it in.

Teardown

Offer at the end of every session, for session-created resources only, each command shown + confirmed. Container only, any mode (echo the host): "${SSH[@]}" "sudo docker rm -f '$SERVICE_NAME'". Mode B infrastructure — order matters, IAM cleanup in references/provisioning.md:

aws ec2 terminate-instances --region "$AWS_REGION" --instance-ids "$INSTANCE_ID"   # billing stops at termination
aws ec2 wait instance-terminated --region "$AWS_REGION" --instance-ids "$INSTANCE_ID"
aws ec2 delete-security-group --region "$AWS_REGION" --group-id "$SG_ID"           # fails while the instance lives — hence the wait
aws ec2 delete-key-pair --region "$AWS_REGION" --key-name "$KEY_NAME"              # only if this session created it; also rm the local .pem

Registry: if the deployment is fully retired AND the ECR repo was created just for it (by bentoml-containerize here, never a pre-existing repo), offer to delete it. ECR bills storage (~$0.10/GB-month) until the repo is gone:

aws ecr delete-repository --region "$ECR_REGION" --repository-name "$REPO_NAME" --force
# --force deletes the images inside — confirm the user wants them gone, not just the instance

If they keep the instance: it bills every hour until terminated, and a stopped instance still bills its EBS volume and its public IPv4 address.

Troubleshooting

Handle EC2 symptoms here; the Kubernetes runbook (bentoml-k8s-deploy/references/troubleshooting.md) does not apply. Docker commands run over SSH with sudo.

Out of scope

Autoscaling groups, HTTPS termination, spot instances, multi-region. Load balancing too: the AWS answer is an ALB (aws elbv2 create-load-balancer + a target group on port 3000, health check /readyz) — hand over that pointer plus the ALB note in references/multi-node.md, then stop. For fleet-scale EC2 work (launch templates, Auto Scaling groups, Spot, SSM Session Manager): /plugin install aws-core@claude-plugins-official; its aws-compute skill picks up where this ends.

References, all linked above where they are needed: references/provisioning.md (Mode B provisioning, cost table, teardown including IAM), references/registry-auth.md (ECR by instance profile or token-over-SSH, other registries), references/multi-node.md (N hosts, ALB pointer).