Agent Ops

Agent Operations: Build, Evaluate, and Monitor AI Agents

This skill covers the agent lifecycle beyond basic tracing: architecture patterns, evaluation, metrics, and production monitoring. All examples use Opik for observability — for SDK details (tracing, integrations, span types), load the opik skill.

The Agent Lifecycle

1. Instrument — Add Opik tracing to make your agent's behavior visible (see opik skill)
2. Evaluate — Measure performance with datasets, metrics, and experiments
3. Monitor — Track quality, cost, and reliability in production
4. Optimize — Improve based on data from evaluation and production traces

Agent Architecture Patterns

Trace every component of your agent with appropriate span types:

import opik

@opik.track(name="research_agent")
def agent(query: str) -> str:
    plan = plan_action(query)        # general span
    results = execute_tool(plan)     # tool span
    return generate_response(results) # llm span

@opik.track(type="tool")
def execute_tool(action: dict) -> str:
    return search_web(action["query"])

@opik.track(type="llm")
def generate_response(context: str) -> str:
    return llm_call(context)

What to Trace

Evaluation

Evaluate agents at multiple levels — end-to-end and per-component:

from opik.evaluation import evaluate
from opik.evaluation.metrics import AnswerRelevance, Hallucination, AgentTaskCompletion

results = evaluate(
    experiment_name="agent-v2",
    dataset=dataset,
    task=lambda item: {"output": agent(item["input"])},
    scoring_metrics=[
        AnswerRelevance(),
        Hallucination(),
        AgentTaskCompletion(),
    ]
)

Built-in Agent Metrics

41 Total Built-in Metrics

Heuristic (Equals, Contains, BLEU, ROUGE, BERTScore, IsJson, etc.), LLM-as-Judge (AnswerRelevance, Hallucination, Usefulness, GEval, etc.), RAG (ContextPrecision, ContextRecall, Faithfulness), and conversation metrics. See references/evaluation.md for the full list.

Production Monitoring

• Dashboards — Visualize quality, cost, latency, and error trends
• Online evaluation — Automatically score production traces with LLM-as-Judge
• Alerts — Get notified when metrics deviate (quality drops, cost spikes, error rates)
• Guardrails — PII detection, topic validation, custom safety checks
• Opik Assist — AI-powered root cause analysis for failed traces

Common Anti-Patterns

Detailed References