AI and LLM Telemetry
Overview
This skill turns the telemetry emitted by LLM applications (traces, spans, spend logs, GenAI metrics) into answers about cost, usage, latency, tool behaviour and privacy. It knows three sources well: Opik from Comet for traces and evaluations, the LiteLLM proxy for spend, budgets and routing, and the OpenTelemetry GenAI semantic conventions used by vendor-neutral instrumentations. The default posture is read-only and evidence-based: query and export, compute from the data, cite request ids, trace ids and timestamps, and never change a budget, a key limit, a callback or a retention setting on a target system without the user's explicit approval.
When it applies
- A bill from OpenAI, Anthropic, Bedrock or Azure OpenAI is higher than expected and the user wants to know which model, team, user, key or day is responsible.
- An agent behaves strangely: too many tool calls, repeated identical calls, long chains, high failure rate on one tool, slow time to first token.
- A team wants an observability setup for a new LLM feature: Opik project, LiteLLM callbacks, OTel collector pipeline, budgets and alerts.
- A compliance or privacy review asks what prompt content is stored, where, for how long, and whether personal data is redacted.
- Evaluation results (feedback scores, experiments on datasets) must be correlated with production traces.
Hand over to observability-agent when the problem is infrastructure latency
unrelated to model calls, and to finops-analyst when the question is cloud
spend outside LLM providers.
Engines and sources
Opik (Comet)
- Instrumentation:
@opik.trackon functions,typeinllm,tool,general,guardrail; integrations for OpenAI, Anthropic, Bedrock, LangChain, LlamaIndex, CrewAI, OpenAI Agents. Configuration throughOPIK_API_KEY,OPIK_WORKSPACE,OPIK_PROJECT_NAME,OPIK_URL_OVERRIDE. - REST:
GET /api/v1/private/traces?project_name=<p>&page=1&size=500&filters=<json>,GET /api/v1/private/spans?project_name=<p>&type=tool&trace_id=<id>,POST /api/v1/private/traces/searchand/spans/searchwithlast_retrieved_idcursor,GET /api/v1/private/traces/statsfor aggregates. Responses are{page, size, total, content: [...]}. - Key fields:
usage.prompt_tokens,usage.completion_tokens,usage.total_tokens,total_estimated_cost,model,provider,error_info,thread_id,feedback_scores[].nameand.value,tags,metadata. - Full field list, filter grammar and the Langfuse, Arize Phoenix and Helicone
equivalents:
references/opik-api.md.
LiteLLM proxy
GET /spend/logs?start_date=YYYY-MM-DD&end_date=YYYY-MM-DDwith rowsrequest_id,call_type,api_key(hashed),spend,total_tokens,prompt_tokens,completion_tokens,startTime,endTime,completionStartTime,model,user,team_id,end_user,metadata,cache_hit,request_tags.GET /global/spend/report?start_date=&end_date=&group_by=team|customer|api_key,GET /global/spend/tags,GET /spend/tags,GET /global/activity.GET /key/info?key=,GET /team/info?team_id=,GET /user/info?user_id=returnspend,max_budget,budget_duration,budget_reset_at,soft_budget,rpm_limit,tpm_limit,model_max_budget.GET /model/infoforinput_cost_per_tokenandoutput_cost_per_token.config.yaml:litellm_settings.success_callback: ["opik", "langfuse", "otel"],failure_callback,turn_off_message_logging,cache,max_budget,budget_duration;environment_variables.OPIK_API_KEY,OPIK_PROJECT_NAME.- Details:
references/litellm-spend-api.md.
OpenTelemetry GenAI conventions
- Spans
chat <model>,embeddings <model>,execute_tool <tool>,invoke_agent <agent>; attributesgen_ai.system(orgen_ai.provider.name),gen_ai.operation.name,gen_ai.request.model,gen_ai.response.model,gen_ai.usage.input_tokens,gen_ai.usage.output_tokens,gen_ai.response.finish_reasons,gen_ai.tool.name,gen_ai.tool.call.id,error.type. - Content events (
gen_ai.content.prompt,gen_ai.user.message, ...) exist only whenOTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true. - Metrics
gen_ai.client.token.usage,gen_ai.client.operation.duration,gen_ai.server.time_to_first_token. - Query in Tempo with TraceQL, in Prometheus with PromQL, in Datadog or
Grafana through the
connectorstool when the connector is configured. - Attribute tables, PromQL and collector redaction:
references/otel-genai-conventions.md.
Navin integration
- Kubernetes or cloud CLIs (reading the proxy config map, collector logs,
secrets metadata) go through the
infratool at observe level; prepare and execute levels are reserved for changes the user approved. - Datadog, Grafana, Slack, GitHub, Notion: use the
connectorstool for dashboards, alert history and delivery of the report. - Tokens (
OPIK_API_KEY,LITELLM_MASTER_KEY,LANGFUSE_SECRET_KEY) come from environment variables; print only their presence, never their value. - A missing CLI (
curl,jq,promtool) is installed with thetoolbeltskill. - Exports above a few megabytes are searched with the
greptool ("error_info", arequest_id, atrace_id) before anyread_file.
Workflow
- Frame the question. Record the period (dates in UTC), the scope (project, team, key, model) and the metric asked for (cost, tokens, latency, tool failures, loops). Name the source of truth: LiteLLM for money when the proxy is in front of every provider, Opik or OTel for behaviour.
- Check access without printing secrets:
env | grep -c OPIK_API_KEY,GET /health/livelinesson the proxy,GET /api/v1/private/projectson Opik. Stop and ask when a source is missing; do not guess numbers. - Export the data to files in the workspace, one file per source and day
range:
spend_logs_<from>_<to>.json,opik_spans_<project>_<date>.json(keep the raw{content: [...]}page). Paginate untilpage * size >= total. For OTel, export OTLP JSON from the collectorfileexporter or TempoGET /api/traces/<id>. - Compute spend:
python3 scripts/spend_report.py examples/spend_logs.json --budget 5(replace the sample by the export). The report gives cost and tokens per model, team, user and day, p50/p95 tokens per request, cache hit ratio, the top 10 requests and daily anomalies by median absolute deviation. Exit 1 means the budget is exceeded or a day is anomalous: open the flagged day and list its top requests before drawing conclusions. - Reconcile with the proxy's own view:
GET /global/spend/reportfor the same range must match the script's totals within rounding. A mismatch points to rows withspend: 0for an unknown model (checkGET /model/infoprices) or to cache hits counted differently. - Audit agent behaviour:
python3 scripts/tool_calls_audit.py examples/opik_spans.json --fail-on-loopsfor an Opik span page, orpython3 scripts/tool_calls_audit.py examples/otel_genai_spans.json --jsonfor OTLP JSON. The audit gives per-tool call count, failure rate, latency p50/p95, repeated identical calls per trace (default threshold 3) and the longest chains. For each loop, open the trace and read theerror_infoorstatus.messageof the repeated span: a loop is almost always the model retrying a failing tool with the same arguments. - Derive the remaining metrics from the same exports: time to first token
=
completionStartTime - startTimeon streamed LiteLLM rows orgen_ai.server.time_to_first_token; error rate = rows with a non-2xxstatusor spans witherror_infodivided by all rows; retry rate = extra CLIENT spans under one parent; cost per 1000 requests per team. - Check privacy: confirm whether prompt bodies are stored (
inputandoutputon Opik spans,gen_ai.*.messageevents, LiteLLMturn_off_message_logging), whether personal data appears in them (grep for emails, phone numbers, ids), what the retention is (Opik cloud plan, self-hosted ClickHouse TTL, Tempo retention). Report gaps as findings; propose the redaction config but apply nothing unprompted. - Write the deliverable, cite request ids, trace ids and timestamps, and
propose budget or limit changes as a separate approval list with the
exact LiteLLM call (
POST /key/update,POST /team/update) for each.
Rules
spendin LiteLLM is computed at request time from the price table; a price update does not rewrite old rows. Recompute from tokens and/model/infowhen the user disputes a figure.- Rows with
cache_hit: "True"cost 0 and keep their tokens: compute the cache hit ratio on rows, not on spend, and report the saved amount as tokens times the model price. api_keyin spend logs is a SHA-256 hash. Map it to an alias withGET /key/info?key=<hash>(returnskey_alias), never ask for raw keys.- Opik
total_estimated_costisnullfor models missing from its price table; treat null as unknown, not as free, and list those models. - Opik
typeis set by the instrumentation; frameworks sometimes log tools asgeneral. When the audit finds no tool spans, filter onnameprefixes (tool:,function:) and report the mapping used. - OTel
execute_toolspans are children of the agent span, not of thechatspan that requested them; link them throughgen_ai.tool.call.idmatching the assistant messagetool_calls[].id. gen_ai.response.finish_reasons = ["length"]on many spans meansmax_tokenstruncation, usually followed by a retry: count it as a cost driver.- The MAD anomaly detector needs at least three days; with fewer, say so instead of flagging nothing silently.
- A budget breach returned by the script (exit 1) is a finding, not a failure of the run.
- Never export or paste prompt bodies into the report; quote at most the tool name and a hash of the arguments. Redact emails and ids in any excerpt.
- Timezones: LiteLLM stores
startTimein UTC; Opikstart_timeis UTC withZ. Convert the user's local dates before filtering. - Changes to budgets,
rpm_limit,tpm_limit, callbacks or retention are prepared as commands and executed only after the user approves each one.
Deliverable
Produce a Markdown report with these sections and tables:
- Scope: period, sources, projects, keys, counts of rows and traces.
- Spend summary: table
| Dimension | Key | Requests | Tokens | Spend (USD) | Share |for model, team and user; table| Day | Requests | Tokens | Spend (USD) | Anomaly |. - Top requests:
| Request id | Model | Team | User | Tokens | Spend (USD) | Start (UTC) |. - Tool behaviour:
| Tool | Calls | Failures | Failure rate | p50 ms | p95 ms |and| Trace id | Tool | Repeats | Arguments hash | Likely cause |. - Latency and errors:
| Metric | Value | Source | Note |for time to first token p50/p95, error rate, retry rate, cache hit ratio. - Privacy:
| Store | Prompt bodies kept | Personal data seen | Retention | Gap |. - Findings and proposed actions:
| # | Finding | Evidence (id, timestamp) | Proposed action | Needs approval |.
When the user wants an HTML report, render the same content with the
studio-html-report skill. Deliver through the connectors tool (Slack,
Notion, GitHub issue) when asked.
Resources
scripts/spend_report.py: LiteLLM/spend/logsexport (JSON array,{data: [...]}or JSONL) to cost and tokens per model, team, user and day, top requests, cache hit ratio, token percentiles and MAD anomalies;--budgetsets the exit 1 threshold,--jsonfor machine output. Run it in step 4.scripts/tool_calls_audit.py: Opik span page or OTLP GenAI JSON to per-tool frequency, failure rate, latency p50/p95, repeated identical calls and longest chains;--loop-threshold,--fail-on-loops,--json. Run it in step 6.examples/spend_logs.json: seven days of synthetic LiteLLM spend rows with one anomalous day and cache hits, for testing the spend script.examples/opik_spans.json: an Opik/spanspage withtoolandllmspans, including a six-repeat loop onget_order.examples/otel_genai_spans.json: OTLP JSON withexecute_toolandchatspans following the GenAI conventions.references/litellm-spend-api.md: spend log fields, aggregate endpoints, key, team and user budget fields, callback configuration and alerting. Open it before querying the proxy or proposing a budget change.references/opik-api.md: Opik objects, SDK usage, REST endpoints, filter grammar, online evaluation rules, and the Langfuse, Phoenix and Helicone equivalents. Open it before exporting traces.references/otel-genai-conventions.md: span names, attributes, content capture opt-in, metrics, PromQL and collector redaction. Open it when the data comes from an OTel pipeline or when designing instrumentation.
