Enterprise Agent Ops

Enterprise Agent Ops

Use this skill for cloud-hosted or continuously running agent systems that need operational controls beyond single CLI sessions.

When to Activate

• Deploying agents in production (cloud, on-prem, or hybrid)
• Operating multi-agent systems with SLA requirements
• Setting up monitoring, alerting, and observability for AI agents
• Managing agent security, permissions, and audit trails
• Debugging production agent failures or behavioral drift

Operational Domains

1. Runtime Lifecycle

2. Observability (Trace-Level)

Modern agent observability requires trace-level visibility — not just request/response logging.

What to capture per agent step:

• Reasoning chain / thinking content (if available)
• Tool call inputs, outputs, and latency
• RAG retrieval queries and returned chunks
• Token consumption (input/output/cache) per step
• Model selection decisions (if using model routing)

Recommended stack:

• OpenTelemetry for standardized tracing — use semantic conventions for GenAI
• LangSmith or Logfire for agent-specific trace visualization
• Grafana + Prometheus for metrics dashboards
• Structured JSON logs with correlation IDs linking traces to business transactions

# OpenTelemetry trace example for an agent step
from opentelemetry import trace

tracer = trace.get_tracer("agent.ops")

with tracer.start_as_current_span("agent.tool_call") as span:
    span.set_attribute("agent.model", "claude-sonnet-4-6")
    span.set_attribute("agent.tool", tool_name)
    span.set_attribute("agent.tokens.input", input_tokens)
    span.set_attribute("agent.tokens.output", output_tokens)
    span.set_attribute("agent.cost_usd", step_cost)
    result = execute_tool(tool_name, tool_input)
    span.set_attribute("agent.tool.success", result.success)

3. Safety Controls

• Least-privilege credentials: Each agent gets only the permissions it needs
• Scope boundaries: Define which tools, APIs, and file paths an agent can access
• Kill switches: Remote disable via feature flag or admin API
• Rate limiting: Per-agent and per-tool rate limits to prevent runaway loops
• Hard budget caps: Circuit breakers that halt execution when cost exceeds threshold
• Human-in-the-loop gates: Require approval for destructive actions (delete, deploy, payment)

4. Change Management

• Immutable deployment artifacts: Never patch running agents in-place
• Canary rollouts: Test new agent versions on subset of traffic before full deploy
• Rollback plan: Every deployment must have a one-command rollback path
• Audit log: Record all configuration changes, permission grants, and agent actions
• Version pinning: Pin model versions and tool schemas; don't auto-update in production

Metrics to Track

Quality Metrics

Operational Metrics

Cost Metrics

Drift Detection

Agent behavior can drift silently due to model updates, prompt changes, or data shifts.

Detection methods:

1. Output schema validation: Verify structured outputs match expected schemas
2. Behavioral regression tests: Run golden-set tasks on every deployment
3. Statistical monitoring: Track output distribution changes (token length, tool call frequency, error patterns)
4. LLM-as-judge sampling: Periodically evaluate a random sample of outputs for quality

# Simple drift detection: compare output distributions
def detect_drift(recent_metrics: list[float], baseline_metrics: list[float], threshold: float = 2.0) -> bool:
    """Alert if recent metrics deviate from baseline by more than threshold standard deviations."""
    import statistics
    baseline_mean = statistics.mean(baseline_metrics)
    baseline_stdev = statistics.stdev(baseline_metrics) or 0.01
    recent_mean = statistics.mean(recent_metrics)
    z_score = abs(recent_mean - baseline_mean) / baseline_stdev
    return z_score > threshold

Incident Pattern

When failure spikes:

1. Freeze — halt new rollout, stop canary promotion
2. Capture — collect representative traces with full reasoning chains
3. Isolate — identify failing route (model? tool? prompt? data?)
4. Triage — classify: transient (retry), systematic (fix), external (escalate)
5. Patch — apply smallest safe change, never batch fixes during incidents
6. Verify — run regression + security checks on the fix
7. Resume — gradual traffic increase with monitoring

Automated Remediation

For known failure classes, implement automated recovery:

REMEDIATION_MAP = {
    "rate_limit": lambda: backoff_and_retry(multiplier=2),
    "model_overloaded": lambda: failover_to_secondary_model(),
    "tool_timeout": lambda: retry_with_extended_timeout(),
    "budget_exceeded": lambda: pause_and_alert_ops(),
    "auth_expired": lambda: refresh_credentials_and_retry(),
}

Deployment Integrations

Observe → Evaluate → Optimize → Deploy Loop

Production agent ops follows a continuous improvement cycle:

1. Observe: Collect traces, metrics, and user feedback
2. Evaluate: Score agent performance against quality benchmarks
3. Optimize: Tune prompts, adjust routing, update tools based on findings
4. Deploy: Push changes through canary → staged → full rollout

Each cycle iteration should be tracked with a timestamp, evidence, and measurable improvement.