How do I install the Instrument LLM Observability skill?

Run npx skillmds@latest add contextjet-ai/instrument-llm-observability in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Instrument LLM Observability safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Instrument LLM Observability?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Instrument LLM Observability free to use?

Yes. Installing skills from SkillMD is free. This skill is licensed under CC0-1.

Who published Instrument LLM Observability?

ContextJet-ai (@contextjet-ai) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Instrument LLM Observability by contextjet-ai

Name: Instrument LLM Observability
Author: ContextJet-ai

Instrument LLM Observability

Use this when adding tracing/observability to an LLM or AI-agent application - capturing prompts, tool calls, token usage, latency, and cost per step. Trigger whenever someone wants to "add tracing", "instrument", "monitor", "see what my agent is doing", or debug an LLM app in production. Prefer vendor-neutral OpenTelemetry unless a specific platform is already in use.

ContextJet-ai a2365df 2 files · 5.0 KB Updated Sep 17, 2026

File contents

Instrument an LLM app for observability

Add production-grade tracing to an LLM/agent app so every run is inspectable: prompts, tool calls, retrievals, token usage, latency, and cost per span.

Decide the approach first (2 questions)

Is a backend already chosen? (Langfuse, Arize Phoenix, Comet Opik, Datadog, Grafana…). If yes, use its SDK/auto-instrumentation. If no, default to OpenTelemetry GenAI semantic conventions so traces export to any OTel backend and you avoid lock-in.
Lowest-friction path available? If the app calls LLMs through a gateway/proxy (LiteLLM, Helicone, Portkey), you may get tracing + cost with zero code by changing the base URL. Check that before adding an SDK.

Vendor-neutral path (recommended default): OpenTelemetry

Emit gen_ai.* spans via an OTel instrumentation library, then point at any collector.

Use OpenLLMetry (traceloop-sdk) or OpenInference (Arize) - both emit OTel-compatible LLM spans.
Minimal shape:
1. Install the instrumentation lib + an OTLP exporter.
2. Initialize once at startup (set service name + OTLP endpoint via OTEL_EXPORTER_OTLP_ENDPOINT).
3. Auto-instrument the LLM SDK (OpenAI/Anthropic/etc.); wrap custom agent steps in manual spans (gen_ai.operation.name, tool spans).
Result: spans carry model, prompt/completion, token counts, latency; cost is derived downstream.

See references/opentelemetry.md for the exact span attributes to set and a copy-paste init.

Platform SDK path (if a backend is already picked)

Wrap the LLM client with the platform's tracer (e.g. track_openai(...), @observe, an OTel exporter to that backend). Prefer their auto-instrumentation over hand-rolled spans; add manual spans only for custom tool/retrieval steps the SDK can't see.

Instrument the right things (checklist)

Each LLM call: model, input messages, output, prompt_tokens/completion_tokens, latency.
Tool/function calls: name, args, result, error.
Retrieval steps (RAG): query, retrieved doc IDs/scores.
Agent/chain boundaries: one parent span per user request, child spans per step.
Errors: record exceptions on the span (don't swallow them - a silent except that returns empty output hides the failure).
Cost: derive from token counts × model price (most platforms do this for you).

Verify

Run one representative request and confirm the trace tree shows the full chain (parent → LLM/tool/retrieval children) with tokens + latency populated. If spans are missing, the instrumentation isn't wrapping that code path - instrument it manually.

Anti-patterns

Logging raw prints instead of structured spans (no trace tree, no cost rollup).
Instrumenting only the top-level call, missing tool/retrieval steps (the interesting failures hide there).
Hard-coding a vendor SDK when OTel would keep you portable.
Sending full prompts/PII to a third-party backend without redaction - mask sensitive fields first.

Tool picker

Not sure which backend? See the sibling choose-observability-stack skill, or the curated list in this repo's README.

---name: instrument-llm-observabilitydescription: Use this when adding tracing/observability to an LLM or AI-agent application - capturing prompts, tool calls, token usage, latency, and cost per step. Trigger whenever someone wants to "add tracing", "instrument", "monitor", "see what my agent is doing", or debug an LLM app in production. Prefer vendor-neutral OpenTelemetry unless a specific platform is already in use.license: CC0-1.0---# Instrument an LLM app for observabilityAdd production-grade tracing to an LLM/agent app so every run is inspectable: prompts, tool calls, retrievals, token usage, latency, and cost per span.## Decide the approach first (2 questions)1. **Is a backend already chosen?** (Langfuse, Arize Phoenix, Comet Opik, Datadog, Grafana…). If yes, use its SDK/auto-instrumentation. If no, default to **OpenTelemetry GenAI semantic conventions** so traces export to *any* OTel backend and you avoid lock-in.2. **Lowest-friction path available?** If the app calls LLMs through a **gateway/proxy** (LiteLLM, Helicone, Portkey), you may get tracing + cost with *zero code* by changing the base URL. Check that before adding an SDK.## Vendor-neutral path (recommended default): OpenTelemetryEmit `gen_ai.*` spans via an OTel instrumentation library, then point at any collector.- Use **OpenLLMetry** (`traceloop-sdk`) or **OpenInference** (Arize) - both emit OTel-compatible LLM spans.- Minimal shape: 1. Install the instrumentation lib + an OTLP exporter. 2. Initialize once at startup (set service name + OTLP endpoint via `OTEL_EXPORTER_OTLP_ENDPOINT`). 3. Auto-instrument the LLM SDK (OpenAI/Anthropic/etc.); wrap custom agent steps in manual spans (`gen_ai.operation.name`, tool spans).- Result: spans carry model, prompt/completion, token counts, latency; cost is derived downstream.See `references/opentelemetry.md` for the exact span attributes to set and a copy-paste init.## Platform SDK path (if a backend is already picked)Wrap the LLM client with the platform's tracer (e.g. `track_openai(...)`, `@observe`, an OTel exporter to that backend). Prefer their **auto-instrumentation** over hand-rolled spans; add manual spans only for custom tool/retrieval steps the SDK can't see.## Instrument the right things (checklist)- [ ] Each LLM call: model, input messages, output, `prompt_tokens`/`completion_tokens`, latency.- [ ] Tool/function calls: name, args, result, error.- [ ] Retrieval steps (RAG): query, retrieved doc IDs/scores.- [ ] Agent/chain boundaries: one parent span per user request, child spans per step.- [ ] Errors: record exceptions on the span (don't swallow them - a silent `except` that returns empty output hides the failure).- [ ] Cost: derive from token counts × model price (most platforms do this for you).## VerifyRun one representative request and confirm the trace tree shows the full chain (parent → LLM/tool/retrieval children) with tokens + latency populated. If spans are missing, the instrumentation isn't wrapping that code path - instrument it manually.## Anti-patterns- Logging raw prints instead of structured spans (no trace tree, no cost rollup).- Instrumenting only the top-level call, missing tool/retrieval steps (the interesting failures hide there).- Hard-coding a vendor SDK when OTel would keep you portable.- Sending full prompts/PII to a third-party backend without redaction - mask sensitive fields first.## Tool pickerNot sure which backend? See the sibling `choose-observability-stack` skill, or the curated list in this repo's README.

Frequently asked questions