Managing Bentoml

BentoML Management Skill

Manage and monitor BentoML model packaging, serving, and deployments.

MANDATORY: Discovery-First Pattern

Always list existing models, bentos, and deployments before creating or modifying anything.

Phase 1: Discovery

#!/bin/bash

echo "=== BentoML Version ==="
bentoml --version 2>/dev/null

echo ""
echo "=== Local Models ==="
bentoml models list 2>/dev/null | head -15

echo ""
echo "=== Local Bentos ==="
bentoml list 2>/dev/null | head -15

echo ""
echo "=== Running Services ==="
bentoml serve --help >/dev/null 2>&1
# Check for running bentoml processes
ps aux | grep -E 'bentoml|bento' | grep -v grep | awk '{print $2"\t"$11" "$12" "$13}' | head -10

echo ""
echo "=== BentoCloud Deployments (if available) ==="
bentoml deployment list 2>/dev/null | head -10 || echo "BentoCloud not configured"

Core Helper Functions

#!/bin/bash

# BentoML CLI wrapper
bml() {
    bentoml "$@" 2>/dev/null
}

# BentoCloud API helper (if using BentoCloud)
bentocloud_api() {
    local method="${1:-GET}"
    local endpoint="$2"
    local data="${3:-}"
    if [ -n "$data" ]; then
        curl -s -X "$method" -H "Authorization: Bearer $BENTOCLOUD_API_TOKEN" \
            -H "Content-Type: application/json" \
            "${BENTOCLOUD_ENDPOINT}/api/v1/${endpoint}" -d "$data"
    else
        curl -s -X "$method" -H "Authorization: Bearer $BENTOCLOUD_API_TOKEN" \
            "${BENTOCLOUD_ENDPOINT}/api/v1/${endpoint}"
    fi
}

# Test a running BentoML service endpoint
bento_test() {
    local url="${1:-http://localhost:3000}"
    local endpoint="${2:-/healthz}"
    curl -s -w "\nHTTP_STATUS:%{http_code}\nTIME:%{time_total}s\n" "${url}${endpoint}"
}

Output Rules

• TOKEN EFFICIENCY: Target ≤50 lines per output
• Use CLI commands with structured output where available
• Never dump full Bento build files -- extract key configurations
• Test endpoints with concise request/response pairs

Common Operations

Model Management

#!/bin/bash
echo "=== All Models ==="
bentoml models list 2>/dev/null

MODEL_TAG="${1:-}"
if [ -n "$MODEL_TAG" ]; then
    echo ""
    echo "=== Model Details: $MODEL_TAG ==="
    bentoml models get "$MODEL_TAG" 2>/dev/null

    echo ""
    echo "=== Model Metadata ==="
    bentoml models get "$MODEL_TAG" -o json 2>/dev/null | jq '{
        tag: .tag,
        module: .module,
        creation_time: .creation_time,
        labels: .labels,
        metadata: .metadata,
        context: .context
    }' | head -30
fi

Bento Build and Packaging

#!/bin/bash
echo "=== All Bentos ==="
bentoml list 2>/dev/null

BENTO_TAG="${1:-}"
if [ -n "$BENTO_TAG" ]; then
    echo ""
    echo "=== Bento Details: $BENTO_TAG ==="
    bentoml get "$BENTO_TAG" 2>/dev/null

    echo ""
    echo "=== Bento Info (JSON) ==="
    bentoml get "$BENTO_TAG" -o json 2>/dev/null | jq '{
        tag: .tag,
        service: .entry_service,
        models: [.models[]? | .tag],
        apis: .apis,
        size: .size,
        creation_time: .creation_time
    }' | head -30
fi

Deployment Status

#!/bin/bash
echo "=== BentoCloud Deployments ==="
bentoml deployment list 2>/dev/null | head -15

DEPLOYMENT="${1:-}"
if [ -n "$DEPLOYMENT" ]; then
    echo ""
    echo "=== Deployment Detail: $DEPLOYMENT ==="
    bentoml deployment get "$DEPLOYMENT" 2>/dev/null | head -30

    echo ""
    echo "=== Deployment Logs ==="
    bentoml deployment logs "$DEPLOYMENT" --tail 20 2>/dev/null
fi

API Testing

#!/bin/bash
SERVICE_URL="${1:-http://localhost:3000}"

echo "=== Health Check ==="
curl -s "${SERVICE_URL}/healthz" | head -5

echo ""
echo "=== Service Metadata ==="
curl -s "${SERVICE_URL}/docs.json" 2>/dev/null | jq '{
    title: .info.title,
    version: .info.version,
    endpoints: [.paths | keys[]]
}' 2>/dev/null || echo "OpenAPI docs not available"

echo ""
echo "=== Readiness Check ==="
curl -s -o /dev/null -w "Status: %{http_code}\nResponse Time: %{time_total}s\n" "${SERVICE_URL}/readyz"

echo ""
echo "=== Metrics (if available) ==="
curl -s "${SERVICE_URL}/metrics" 2>/dev/null | grep -E '^(bentoml_|HELP|TYPE)' | head -20

Runner Configuration Inspection

#!/bin/bash
BENTO_TAG="${1:?Bento tag required}"

echo "=== Service Configuration ==="
bentoml get "$BENTO_TAG" -o json 2>/dev/null | jq '{
    runners: [.runners[]? | {
        name: .name,
        runnable_type: .runnable_type,
        models: .models,
        resource_config: .resource_config
    }],
    apis: [.apis[]? | {
        name: .name,
        input_type: .input_type,
        output_type: .output_type,
        route: .route
    }]
}' | head -40

echo ""
echo "=== Docker Configuration ==="
bentoml get "$BENTO_TAG" -o json 2>/dev/null | jq '.docker // "not configured"'

Safety Rules

• NEVER delete models or bentos that are referenced by active deployments -- causes immediate serving failures
• NEVER update production deployments without testing the new Bento version first
• Always health-check endpoints after deployment changes -- verify /healthz and /readyz respond
• Test with sample input before routing production traffic -- schema mismatches cause 500 errors
• Back up model artifacts before deleting old versions -- BentoML model store deletion is permanent

Output Format

Present results as a structured report:

Managing Bentoml Report
═══════════════════════
Resources discovered: [count]

Resource       Status    Key Metric    Issues
──────────────────────────────────────────────
[name]         [ok/warn] [value]       [findings]

Summary: [total] resources | [ok] healthy | [warn] warnings | [crit] critical
Action Items: [list of prioritized findings]

Target ≤50 lines of output. Use tables for multi-resource comparisons.

Anti-Hallucination Rules

1. NEVER assume resource names — always discover via CLI/API in Phase 1 before referencing in Phase 2.
2. NEVER fabricate metric names or dimensions — verify against the service documentation or --help output.
3. NEVER mix CLI commands between service versions — confirm which version/API you are targeting.
4. ALWAYS use the discovery → verify → analyze chain — every resource referenced must have been discovered first.
5. ALWAYS handle empty results gracefully — an empty response is valid data, not an error to retry.

Counter-Rationalizations

Common Pitfalls

• Model tag versioning: Tags use name:version format -- omitting version uses "latest" which may not be what you expect
• Runner resource allocation: Runners run in separate processes -- GPU allocation must be explicit in runner config
• Build context size: Large files in the build directory slow down bentoml build -- use .bentoignore to exclude unnecessary files
• Python version mismatch: Bento Python version must match the serving environment -- mismatches cause import errors
• Containerization: bentoml containerize requires Docker -- ensure Docker daemon is running
• Adaptive batching: Batch size tuning affects latency vs throughput tradeoff -- monitor p99 latency after changes
• Service dependencies: Multi-service bentos require all dependent services to be healthy -- partial failures cascade
• Port conflicts: Default port 3000 may conflict with other services -- use --port flag to specify alternate ports