DeepEval Tracing
Use this skill to instrument an AI application — an LLM app, agent, RAG
pipeline, or chatbot — with DeepEval's native tracing so its execution is
visible span by span in Confident AI's Observatory. The work is: pick a
supported integration when one exists, fall back to manual @observe
otherwise, give each span a meaningful type, and add tags and metadata.
This skill stops at producing well-formed traces. Attaching evaluation metrics
and running evals is the deepeval skill's job.
Scope: AI Applications Only
Instrument only the AI parts of the system — agent loops and planning, LLM
calls, retrieval / vector search, and tool calls. The span types (llm,
retriever, tool, agent) describe AI components. Do not trace non-AI
software (web servers, CRUD backends, infrastructure). If the target has no
LLM, agent, retrieval, or tool-calling component, this skill does not apply.
When to Use vs the deepeval and deepeval-otel Skills
- This skill (
deepeval-tracing) — instrument an app with the DeepEval SDK
(@observe, framework integrations) so traces reach Confident AI.
deepeval skill — build pytest eval suites: datasets, metrics, traced
evals, deepeval test run, iteration. It runs evals against an app this
skill instrumented.
deepeval-otel skill — instrument with the vendor-neutral OpenTelemetry
SDK instead of the DeepEval SDK (raw OTLP, including non-Python apps).
The three are complementary. If unsure between this skill and deepeval-otel:
use this one when the app is Python and you want the DeepEval SDK; use
deepeval-otel when you want raw OpenTelemetry or the app is not Python.
Prerequisites
- An AI application in Python with
pip install deepeval.
- For traces to reach Confident AI:
deepeval login, or an exported
CONFIDENT_API_KEY (preferred for CI and non-interactive runs).
Workflow
- Confirm the target is an AI application (it has LLM calls, an agent loop,
retrieval, or tool calls). If it has none of these, stop — this skill does
not apply.
- Detect the framework, model provider, agent SDK, and vector database in use.
- Read
references/integrations.md and the exact integration doc for what was
detected. Prefer a native integration over manual instrumentation.
- If no native integration fits, instrument manually with
@observe. Read
references/tracing.md.
- Give each span a meaningful
type (llm, retriever, tool, agent) and
capture inputs/outputs.
- Add trace-level tags and metadata where they help diagnose failure patterns.
Never trace secrets, credentials, or raw sensitive data.
- Confirm
deepeval login or CONFIDENT_API_KEY, then verify traces appear
in the Confident AI Observatory.
Core Principles
- Instrument AI components only —
llm, retriever, tool, agent spans.
Never trace non-AI software.
- Prefer a supported integration over manual
@observe. Manual tracing is the
fallback for unsupported frameworks and app-owned wrapper boundaries.
- Read the exact integration doc before writing tracing code.
- Give spans meaningful types; let names default to function names unless
there is a strong reason to override.
- Never trace secrets, credentials, API keys, or raw sensitive user data.
- Producing traces is the scope. Attaching metrics and running evals belong to
the
deepeval skill; raw OpenTelemetry export belongs to deepeval-otel.
References
| Topic |
File |
Manual instrumentation: @observe, span types, tags, metadata |
references/tracing.md |
| Integration selection rule and framework / model / vector-DB doc index |
references/integrations.md |
1---2name: deepeval-tracing3description: >4 Instrument an AI application with DeepEval's native tracing so its behavior5 is visible in Confident AI. TRIGGER when the user wants to add DeepEval6 tracing or @observe to an LLM app, agent, RAG pipeline, or chatbot; wire a7 framework, model-provider, or vector-database integration (LangGraph,8 LangChain, OpenAI Agents, LlamaIndex, Pydantic AI, CrewAI, and others);9 choose between a native integration and manual instrumentation; set span10 types, tags, or metadata; or send DeepEval-SDK traces to Confident AI's11 Observatory. DO NOT TRIGGER for building DeepEval pytest eval suites,12 datasets, goldens, metrics, or deepeval test run (use the `deepeval` skill),13 or for raw OpenTelemetry / OTLP export without the deepeval package (use the14 `deepeval-otel` skill). This skill is purely DeepEval-SDK instrumentation —15 producing well-formed traces, not running evals.16license: Apache-2.017metadata:18 source: https://skillmd.com/skills/confident-ai/deepeval-tracing19 author: Confident AI20 version: "1.0.0"21 category: observability22 tags: "deepeval, tracing, observe, instrumentation, integrations, spans, confident-ai"23 compatibility: "Python with `pip install deepeval`. Instrumentation uses the DeepEval SDK (`deepeval.tracing`). Sending traces to Confident AI requires `deepeval login` or an exported `CONFIDENT_API_KEY`."24---2526# DeepEval Tracing2728Use this skill to instrument an **AI application** — an LLM app, agent, RAG29pipeline, or chatbot — with **DeepEval's native tracing** so its execution is30visible span by span in **Confident AI's Observatory**. The work is: pick a31supported integration when one exists, fall back to manual `@observe`32otherwise, give each span a meaningful type, and add tags and metadata.3334This skill stops at producing well-formed traces. Attaching evaluation metrics35and running evals is the `deepeval` skill's job.3637## Scope: AI Applications Only3839Instrument only the AI parts of the system — agent loops and planning, LLM40calls, retrieval / vector search, and tool calls. The span types (`llm`,41`retriever`, `tool`, `agent`) describe AI components. Do not trace non-AI42software (web servers, CRUD backends, infrastructure). If the target has no43LLM, agent, retrieval, or tool-calling component, this skill does not apply.4445## When to Use vs the `deepeval` and `deepeval-otel` Skills4647- **This skill (`deepeval-tracing`)** — instrument an app with the DeepEval SDK48 (`@observe`, framework integrations) so traces reach Confident AI.49- **`deepeval` skill** — build pytest eval suites: datasets, metrics, traced50 evals, `deepeval test run`, iteration. It runs evals *against* an app this51 skill instrumented.52- **`deepeval-otel` skill** — instrument with the vendor-neutral OpenTelemetry53 SDK instead of the DeepEval SDK (raw OTLP, including non-Python apps).5455The three are complementary. If unsure between this skill and `deepeval-otel`:56use this one when the app is Python and you want the DeepEval SDK; use57`deepeval-otel` when you want raw OpenTelemetry or the app is not Python.5859## Prerequisites6061- An AI application in Python with `pip install deepeval`.62- For traces to reach Confident AI: `deepeval login`, or an exported63 `CONFIDENT_API_KEY` (preferred for CI and non-interactive runs).6465## Workflow66671. Confirm the target is an AI application (it has LLM calls, an agent loop,68 retrieval, or tool calls). If it has none of these, stop — this skill does69 not apply.702. Detect the framework, model provider, agent SDK, and vector database in use.713. Read `references/integrations.md` and the exact integration doc for what was72 detected. Prefer a native integration over manual instrumentation.734. If no native integration fits, instrument manually with `@observe`. Read74 `references/tracing.md`.755. Give each span a meaningful `type` (`llm`, `retriever`, `tool`, `agent`) and76 capture inputs/outputs.776. Add trace-level tags and metadata where they help diagnose failure patterns.78 Never trace secrets, credentials, or raw sensitive data.797. Confirm `deepeval login` or `CONFIDENT_API_KEY`, then verify traces appear80 in the Confident AI Observatory.8182## Core Principles83841. Instrument AI components only — `llm`, `retriever`, `tool`, `agent` spans.85 Never trace non-AI software.862. Prefer a supported integration over manual `@observe`. Manual tracing is the87 fallback for unsupported frameworks and app-owned wrapper boundaries.883. Read the exact integration doc before writing tracing code.894. Give spans meaningful types; let names default to function names unless90 there is a strong reason to override.915. Never trace secrets, credentials, API keys, or raw sensitive user data.926. Producing traces is the scope. Attaching metrics and running evals belong to93 the `deepeval` skill; raw OpenTelemetry export belongs to `deepeval-otel`.9495## References9697| Topic | File |98| --- | --- |99| Manual instrumentation: `@observe`, span types, tags, metadata | `references/tracing.md` |100| Integration selection rule and framework / model / vector-DB doc index | `references/integrations.md` |