LLMOps Platform Engineering
Design and operate an internal LLM platform that supports rapid experimentation without compromising reliability, cost, or compliance.
When to Use This Skill
- Building an internal platform for teams to deploy and manage LLM-powered features
- Designing CI/CD pipelines that include model evaluation gates
- Setting up A/B testing infrastructure for model versions
- Creating Kubernetes-based model serving infrastructure
- Establishing governance workflows for model promotion
Prerequisites
- Kubernetes cluster with GPU node pools (or cloud inference API access)
- Container registry (Harbor, ECR, GCR, or ACR)
- CI/CD system (GitHub Actions, GitLab CI, or Argo Workflows)
- Observability stack (Prometheus + Grafana + OpenTelemetry)
- Model registry (MLflow or custom metadata store)
Outcomes
- Standardized path from experiment to production
- Safe model rollout with quality and safety gates
- Repeatable infra modules for inference, vector DB, and observability
- Clear ownership model across platform, app, and security teams
Reference Architecture
- Control Plane: model registry, prompt/version catalog, policy checks, eval pipeline.
- Data Plane: inference gateway, vector database, cache, feature store.
- Ops Plane: telemetry, alerting, SLO dashboards, cost analytics.
- Security Plane: IAM boundaries, secret rotation, content filters, audit logs.
Model Promotion Pipeline
# .github/workflows/model-promotion.yaml
name: Model Promotion Pipeline
on:
workflow_dispatch:
inputs:
model_name:
description: "Model identifier"
required: true
model_version:
description: "Model version to promote"
required: true
target_env:
description: "Target environment"
required: true
type: choice
options: [staging, production]
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run quality evaluation suite
run: |
python -m evals.run \
--model "${{ inputs.model_name }}:${{ inputs.model_version }}" \
--suite quality \
--output results/quality.json
- name: Run safety evaluation suite
run: |
python -m evals.run \
--model "${{ inputs.model_name }}:${{ inputs.model_version }}" \
--suite safety \
--output results/safety.json
- name: Run latency benchmark
run: |
python -m evals.benchmark \
--model "${{ inputs.model_name }}:${{ inputs.model_version }}" \
--concurrent-users 50 \
--duration 300 \
--output results/latency.json
- name: Gate check - quality
run: |
python -m evals.gate_check \
--results results/quality.json \
--threshold-file thresholds/quality.yaml
- name: Gate check - safety
run: |
python -m evals.gate_check \
--results results/safety.json \
--threshold-file thresholds/safety.yaml
- name: Gate check - latency
run: |
python -m evals.gate_check \
--results results/latency.json \
--threshold-file thresholds/latency.yaml
- name: Upload eval evidence
uses: actions/upload-artifact@v4
with:
name: eval-results-${{ inputs.model_version }}
path: results/
approve:
needs: evaluate
runs-on: ubuntu-latest
environment: ${{ inputs.target_env }}
steps:
- name: Record approval
run: |
echo "Approved by: ${{ github.actor }}"
echo "Model: ${{ inputs.model_name }}:${{ inputs.model_version }}"
echo "Target: ${{ inputs.target_env }}"
echo "Time: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
deploy:
needs: approve
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Deploy canary
run: |
kubectl set image deployment/${{ inputs.model_name }}-canary \
model=${{ inputs.model_name }}:${{ inputs.model_version }} \
-n ai-${{ inputs.target_env }}
- name: Wait for canary validation (15 min)
run: |
python -m canary.validate \
--deployment ${{ inputs.model_name }}-canary \
--namespace ai-${{ inputs.target_env }} \
--duration 900 \
--quality-threshold 0.85 \
--error-rate-threshold 0.02
- name: Promote to full rollout
run: |
kubectl set image deployment/${{ inputs.model_name }} \
model=${{ inputs.model_name }}:${{ inputs.model_version }} \
-n ai-${{ inputs.target_env }}
kubectl rollout status deployment/${{ inputs.model_name }} \
-n ai-${{ inputs.target_env }} --timeout=300s
Evaluation Gate Thresholds
# thresholds/quality.yaml
gates:
groundedness:
metric: groundedness_score
min: 0.85
comparison: gte
task_success:
metric: task_success_rate
min: 0.90
comparison: gte
hallucination:
metric: hallucination_rate
max: 0.08
comparison: lte
regression:
metric: quality_delta_vs_baseline
min: -0.02
comparison: gte
description: "Must not regress more than 2% vs current production"
# thresholds/latency.yaml
gates:
p50_latency:
metric: latency_p50_ms
max: 800
comparison: lte
p95_latency:
metric: latency_p95_ms
max: 2000
comparison: lte
p99_latency:
metric: latency_p99_ms
max: 5000
comparison: lte
throughput:
metric: requests_per_second
min: 50
comparison: gte
A/B Testing Configuration
# ab-test-config.yaml
apiVersion: gateway.ai/v1
kind: ABTest
metadata:
name: model-comparison-q1
namespace: ai-production
spec:
duration: 7d
traffic_split:
control:
model: gpt-4o-2024-08-06
weight: 70
treatment:
model: gpt-4o-2025-01-15
weight: 30
metrics:
primary:
- task_success_rate
- user_satisfaction_score
secondary:
- latency_p95
- cost_per_request
- hallucination_rate
guardrails:
auto_rollback_if:
- metric: task_success_rate
threshold: 0.80
window: 1h
- metric: hallucination_rate
threshold: 0.15
window: 30m
assignment:
strategy: sticky_user
hash_key: user_id
Kubernetes Model Serving Deployment
# model-serving-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: llm-inference
namespace: ai-production
labels:
app: llm-inference
model: gpt-4o
version: "2025-01"
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector:
matchLabels:
app: llm-inference
template:
metadata:
labels:
app: llm-inference
model: gpt-4o
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
prometheus.io/path: "/metrics"
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: llm-inference
containers:
- name: model
image: registry.internal/vllm-server:0.4.1
args:
- "--model=/models/current"
- "--tensor-parallel-size=1"
- "--max-model-len=8192"
- "--gpu-memory-utilization=0.90"
ports:
- containerPort: 8000
name: inference
- containerPort: 8080
name: metrics
resources:
requests:
cpu: "4"
memory: "16Gi"
nvidia.com/gpu: "1"
limits:
cpu: "8"
memory: "32Gi"
nvidia.com/gpu: "1"
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 60
periodSeconds: 10
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 120
periodSeconds: 30
volumeMounts:
- name: model-weights
mountPath: /models
readOnly: true
- name: config
mountPath: /etc/vllm
volumes:
- name: model-weights
persistentVolumeClaim:
claimName: model-weights-pvc
- name: config
configMap:
name: vllm-config
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
nodeSelector:
gpu-type: a100
---
apiVersion: v1
kind: Service
metadata:
name: llm-inference
namespace: ai-production
spec:
selector:
app: llm-inference
ports:
- name: inference
port: 8000
targetPort: 8000
- name: metrics
port: 8080
targetPort: 8080
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: llm-inference-hpa
namespace: ai-production
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: llm-inference
minReplicas: 2
maxReplicas: 10
metrics:
- type: Pods
pods:
metric:
name: llm_queue_depth
target:
type: AverageValue
averageValue: "5"
- type: Pods
pods:
metric:
name: gpu_utilization_percent
target:
type: AverageValue
averageValue: "75"
behavior:
scaleUp:
stabilizationWindowSeconds: 60
policies:
- type: Pods
value: 2
periodSeconds: 120
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Pods
value: 1
periodSeconds: 300
CI/CD Design for AI Services
- Build immutable containers with pinned dependencies and model hashes.
- Use environment promotion:
dev -> stage -> prod.
- Fail deployment if:
- regression evals drop below baseline,
- safety tests exceed risk threshold,
- p95 latency exceeds SLO budget.
- Store deployment evidence for audits (commit SHA, eval report, approver).
Operational SLOs
| Signal |
Target |
Measurement Window |
| Availability |
99.9% |
30-day rolling |
| p95 Latency |
< 1200ms |
5-min buckets |
| Cost per request |
< $0.05 |
1-hour average |
| Task success rate |
> 90% |
24-hour rolling |
| Groundedness |
> 85% |
24-hour rolling |
Platform Guardrails
- Enforce tenant quotas and model allow-lists.
- Require structured output contracts for automation paths.
- Default to low-risk model settings for critical workflows.
- Disable unconstrained tool execution in production.
Tooling Stack (Example)
| Layer |
Tools |
| Orchestration |
Argo Workflows, GitHub Actions, Airflow |
| Model Registry |
MLflow, custom metadata DB |
| Gateway |
LiteLLM, Bifrost, Envoy-based API gateway |
| Observability |
OpenTelemetry + Prometheus + Grafana + Langfuse |
| Policy |
OPA/Rego for deployment and runtime checks |
| Evaluation |
RAGAS, custom eval harness, Promptfoo |
| Serving |
vLLM, TGI, Triton Inference Server |
Troubleshooting
| Issue |
Diagnosis |
Resolution |
| Canary fails quality gate |
Compare eval results with baseline |
Adjust model config or revert version |
| Deployment stuck in rollout |
Check pod events and resource quotas |
Fix resource limits or node availability |
| A/B test shows no significant difference |
Verify traffic split and sample size |
Extend test duration or increase treatment weight |
| Model cold start too slow |
Large model weight download |
Use pre-cached PVCs or init containers |
| Eval pipeline flaky |
Non-deterministic model outputs |
Set temperature=0 for evals, increase sample size |
Related Skills
- ai-pipeline-orchestration (
ai-pipeline-orchestration) - Orchestrate ingestion and inference workflows
- agent-evals (
agent-evals) - Build evaluation gates for releases
- llm-gateway (
llm-gateway) - Route and control LLM traffic
- model-registry-governance (
model-registry-governance) - Model lifecycle and approval workflows
- ai-sre-incident-response (
ai-sre-incident-response) - AI-specific incident response
Limitations
- Guidance executes against real environments: confirm target, blast radius, and rollback plan before applying anything.
- Never deploy to production without explicit approval. Docs-only import: upstream scripts and templates not bundled.
Example
git status && git diff --stat
kubectl diff -f manifest.yaml
Adapted from BagelHole/DevOps-Security-Agent-Skills (MIT); frontmatter, When to Use/Limitations, and safety boundaries added for upstream compliance. Docs-only import: helper scripts and templates not bundled.
1---2name: llmops-platform-engineering3description: Build production LLMOps platforms with CI/CD, model promotion workflows,4 evaluation gates, rollback, and governance across cloud and self-hosted inference.5category: devops6risk: critical7source: https://github.com/BagelHole/DevOps-Security-Agent-Skills8source_repo: BagelHole/DevOps-Security-Agent-Skills9source_type: community10date_added: '2026-09-20'11license: MIT12license_source: https://github.com/BagelHole/DevOps-Security-Agent-Skills/blob/main/LICENSE13compatibility: Requires the relevant platform CLIs (kubectl, helm, terraform, git,14 CI runners) and authorized access to the target environment. Docs-only; helper scripts15 and templates not bundled.16metadata:17 source: https://skillmd.com/skills/iradoweck/llmops-platform-engineering18 author: devops-skills19 version: '1.0'20---2122# LLMOps Platform Engineering2324Design and operate an internal LLM platform that supports rapid experimentation without compromising reliability, cost, or compliance.2526## When to Use This Skill2728- Building an internal platform for teams to deploy and manage LLM-powered features29- Designing CI/CD pipelines that include model evaluation gates30- Setting up A/B testing infrastructure for model versions31- Creating Kubernetes-based model serving infrastructure32- Establishing governance workflows for model promotion3334## Prerequisites3536- Kubernetes cluster with GPU node pools (or cloud inference API access)37- Container registry (Harbor, ECR, GCR, or ACR)38- CI/CD system (GitHub Actions, GitLab CI, or Argo Workflows)39- Observability stack (Prometheus + Grafana + OpenTelemetry)40- Model registry (MLflow or custom metadata store)4142## Outcomes4344- Standardized path from experiment to production45- Safe model rollout with quality and safety gates46- Repeatable infra modules for inference, vector DB, and observability47- Clear ownership model across platform, app, and security teams4849## Reference Architecture50511. **Control Plane**: model registry, prompt/version catalog, policy checks, eval pipeline.522. **Data Plane**: inference gateway, vector database, cache, feature store.533. **Ops Plane**: telemetry, alerting, SLO dashboards, cost analytics.544. **Security Plane**: IAM boundaries, secret rotation, content filters, audit logs.5556## Model Promotion Pipeline5758```yaml59# .github/workflows/model-promotion.yaml60name: Model Promotion Pipeline61on:62 workflow_dispatch:63 inputs:64 model_name:65 description: "Model identifier"66 required: true67 model_version:68 description: "Model version to promote"69 required: true70 target_env:71 description: "Target environment"72 required: true73 type: choice74 options: [staging, production]7576jobs:77 evaluate:78 runs-on: ubuntu-latest79 steps:80 - uses: actions/checkout@v48182 - name: Run quality evaluation suite83 run: |84 python -m evals.run \85 --model "${{ inputs.model_name }}:${{ inputs.model_version }}" \86 --suite quality \87 --output results/quality.json8889 - name: Run safety evaluation suite90 run: |91 python -m evals.run \92 --model "${{ inputs.model_name }}:${{ inputs.model_version }}" \93 --suite safety \94 --output results/safety.json9596 - name: Run latency benchmark97 run: |98 python -m evals.benchmark \99 --model "${{ inputs.model_name }}:${{ inputs.model_version }}" \100 --concurrent-users 50 \101 --duration 300 \102 --output results/latency.json103104 - name: Gate check - quality105 run: |106 python -m evals.gate_check \107 --results results/quality.json \108 --threshold-file thresholds/quality.yaml109110 - name: Gate check - safety111 run: |112 python -m evals.gate_check \113 --results results/safety.json \114 --threshold-file thresholds/safety.yaml115116 - name: Gate check - latency117 run: |118 python -m evals.gate_check \119 --results results/latency.json \120 --threshold-file thresholds/latency.yaml121122 - name: Upload eval evidence123 uses: actions/upload-artifact@v4124 with:125 name: eval-results-${{ inputs.model_version }}126 path: results/127128 approve:129 needs: evaluate130 runs-on: ubuntu-latest131 environment: ${{ inputs.target_env }}132 steps:133 - name: Record approval134 run: |135 echo "Approved by: ${{ github.actor }}"136 echo "Model: ${{ inputs.model_name }}:${{ inputs.model_version }}"137 echo "Target: ${{ inputs.target_env }}"138 echo "Time: $(date -u +%Y-%m-%dT%H:%M:%SZ)"139140 deploy:141 needs: approve142 runs-on: ubuntu-latest143 steps:144 - uses: actions/checkout@v4145146 - name: Deploy canary147 run: |148 kubectl set image deployment/${{ inputs.model_name }}-canary \149 model=${{ inputs.model_name }}:${{ inputs.model_version }} \150 -n ai-${{ inputs.target_env }}151152 - name: Wait for canary validation (15 min)153 run: |154 python -m canary.validate \155 --deployment ${{ inputs.model_name }}-canary \156 --namespace ai-${{ inputs.target_env }} \157 --duration 900 \158 --quality-threshold 0.85 \159 --error-rate-threshold 0.02160161 - name: Promote to full rollout162 run: |163 kubectl set image deployment/${{ inputs.model_name }} \164 model=${{ inputs.model_name }}:${{ inputs.model_version }} \165 -n ai-${{ inputs.target_env }}166 kubectl rollout status deployment/${{ inputs.model_name }} \167 -n ai-${{ inputs.target_env }} --timeout=300s168```169170## Evaluation Gate Thresholds171172```yaml173# thresholds/quality.yaml174gates:175 groundedness:176 metric: groundedness_score177 min: 0.85178 comparison: gte179 task_success:180 metric: task_success_rate181 min: 0.90182 comparison: gte183 hallucination:184 metric: hallucination_rate185 max: 0.08186 comparison: lte187 regression:188 metric: quality_delta_vs_baseline189 min: -0.02190 comparison: gte191 description: "Must not regress more than 2% vs current production"192193# thresholds/latency.yaml194gates:195 p50_latency:196 metric: latency_p50_ms197 max: 800198 comparison: lte199 p95_latency:200 metric: latency_p95_ms201 max: 2000202 comparison: lte203 p99_latency:204 metric: latency_p99_ms205 max: 5000206 comparison: lte207 throughput:208 metric: requests_per_second209 min: 50210 comparison: gte211```212213## A/B Testing Configuration214215```yaml216# ab-test-config.yaml217apiVersion: gateway.ai/v1218kind: ABTest219metadata:220 name: model-comparison-q1221 namespace: ai-production222spec:223 duration: 7d224 traffic_split:225 control:226 model: gpt-4o-2024-08-06227 weight: 70228 treatment:229 model: gpt-4o-2025-01-15230 weight: 30231 metrics:232 primary:233 - task_success_rate234 - user_satisfaction_score235 secondary:236 - latency_p95237 - cost_per_request238 - hallucination_rate239 guardrails:240 auto_rollback_if:241 - metric: task_success_rate242 threshold: 0.80243 window: 1h244 - metric: hallucination_rate245 threshold: 0.15246 window: 30m247 assignment:248 strategy: sticky_user249 hash_key: user_id250```251252## Kubernetes Model Serving Deployment253254```yaml255# model-serving-deployment.yaml256apiVersion: apps/v1257kind: Deployment258metadata:259 name: llm-inference260 namespace: ai-production261 labels:262 app: llm-inference263 model: gpt-4o264 version: "2025-01"265spec:266 replicas: 3267 strategy:268 type: RollingUpdate269 rollingUpdate:270 maxSurge: 1271 maxUnavailable: 0272 selector:273 matchLabels:274 app: llm-inference275 template:276 metadata:277 labels:278 app: llm-inference279 model: gpt-4o280 annotations:281 prometheus.io/scrape: "true"282 prometheus.io/port: "8080"283 prometheus.io/path: "/metrics"284 spec:285 topologySpreadConstraints:286 - maxSkew: 1287 topologyKey: topology.kubernetes.io/zone288 whenUnsatisfiable: DoNotSchedule289 labelSelector:290 matchLabels:291 app: llm-inference292 containers:293 - name: model294 image: registry.internal/vllm-server:0.4.1295 args:296 - "--model=/models/current"297 - "--tensor-parallel-size=1"298 - "--max-model-len=8192"299 - "--gpu-memory-utilization=0.90"300 ports:301 - containerPort: 8000302 name: inference303 - containerPort: 8080304 name: metrics305 resources:306 requests:307 cpu: "4"308 memory: "16Gi"309 nvidia.com/gpu: "1"310 limits:311 cpu: "8"312 memory: "32Gi"313 nvidia.com/gpu: "1"314 readinessProbe:315 httpGet:316 path: /health317 port: 8000318 initialDelaySeconds: 60319 periodSeconds: 10320 livenessProbe:321 httpGet:322 path: /health323 port: 8000324 initialDelaySeconds: 120325 periodSeconds: 30326 volumeMounts:327 - name: model-weights328 mountPath: /models329 readOnly: true330 - name: config331 mountPath: /etc/vllm332 volumes:333 - name: model-weights334 persistentVolumeClaim:335 claimName: model-weights-pvc336 - name: config337 configMap:338 name: vllm-config339 tolerations:340 - key: nvidia.com/gpu341 operator: Exists342 effect: NoSchedule343 nodeSelector:344 gpu-type: a100345---346apiVersion: v1347kind: Service348metadata:349 name: llm-inference350 namespace: ai-production351spec:352 selector:353 app: llm-inference354 ports:355 - name: inference356 port: 8000357 targetPort: 8000358 - name: metrics359 port: 8080360 targetPort: 8080361---362apiVersion: autoscaling/v2363kind: HorizontalPodAutoscaler364metadata:365 name: llm-inference-hpa366 namespace: ai-production367spec:368 scaleTargetRef:369 apiVersion: apps/v1370 kind: Deployment371 name: llm-inference372 minReplicas: 2373 maxReplicas: 10374 metrics:375 - type: Pods376 pods:377 metric:378 name: llm_queue_depth379 target:380 type: AverageValue381 averageValue: "5"382 - type: Pods383 pods:384 metric:385 name: gpu_utilization_percent386 target:387 type: AverageValue388 averageValue: "75"389 behavior:390 scaleUp:391 stabilizationWindowSeconds: 60392 policies:393 - type: Pods394 value: 2395 periodSeconds: 120396 scaleDown:397 stabilizationWindowSeconds: 300398 policies:399 - type: Pods400 value: 1401 periodSeconds: 300402```403404## CI/CD Design for AI Services405406- Build immutable containers with pinned dependencies and model hashes.407- Use environment promotion: `dev -> stage -> prod`.408- Fail deployment if:409 - regression evals drop below baseline,410 - safety tests exceed risk threshold,411 - p95 latency exceeds SLO budget.412- Store deployment evidence for audits (commit SHA, eval report, approver).413414## Operational SLOs415416| Signal | Target | Measurement Window |417|--------|--------|--------------------|418| Availability | 99.9% | 30-day rolling |419| p95 Latency | < 1200ms | 5-min buckets |420| Cost per request | < $0.05 | 1-hour average |421| Task success rate | > 90% | 24-hour rolling |422| Groundedness | > 85% | 24-hour rolling |423424## Platform Guardrails425426- Enforce tenant quotas and model allow-lists.427- Require structured output contracts for automation paths.428- Default to low-risk model settings for critical workflows.429- Disable unconstrained tool execution in production.430431## Tooling Stack (Example)432433| Layer | Tools |434|-------|-------|435| Orchestration | Argo Workflows, GitHub Actions, Airflow |436| Model Registry | MLflow, custom metadata DB |437| Gateway | LiteLLM, Bifrost, Envoy-based API gateway |438| Observability | OpenTelemetry + Prometheus + Grafana + Langfuse |439| Policy | OPA/Rego for deployment and runtime checks |440| Evaluation | RAGAS, custom eval harness, Promptfoo |441| Serving | vLLM, TGI, Triton Inference Server |442443## Troubleshooting444445| Issue | Diagnosis | Resolution |446|-------|-----------|------------|447| Canary fails quality gate | Compare eval results with baseline | Adjust model config or revert version |448| Deployment stuck in rollout | Check pod events and resource quotas | Fix resource limits or node availability |449| A/B test shows no significant difference | Verify traffic split and sample size | Extend test duration or increase treatment weight |450| Model cold start too slow | Large model weight download | Use pre-cached PVCs or init containers |451| Eval pipeline flaky | Non-deterministic model outputs | Set temperature=0 for evals, increase sample size |452453## Related Skills454455- ai-pipeline-orchestration (`ai-pipeline-orchestration`) - Orchestrate ingestion and inference workflows456- agent-evals (`agent-evals`) - Build evaluation gates for releases457- llm-gateway (`llm-gateway`) - Route and control LLM traffic458- model-registry-governance (`model-registry-governance`) - Model lifecycle and approval workflows459- ai-sre-incident-response (`ai-sre-incident-response`) - AI-specific incident response460461## Limitations462463- Guidance executes against real environments: confirm target, blast radius, and rollback plan before applying anything.464- Never deploy to production without explicit approval. Docs-only import: upstream scripts and templates not bundled.465466### Example467468```bash469git status && git diff --stat470kubectl diff -f manifest.yaml471```472473> Adapted from [BagelHole/DevOps-Security-Agent-Skills](https://github.com/BagelHole/DevOps-Security-Agent-Skills) (MIT); frontmatter, When to Use/Limitations, and safety boundaries added for upstream compliance. Docs-only import: helper scripts and templates not bundled.