# Lemma Tracing

> Integrate Lemma AI observability tracing into a codebase. Use when adding Lemma tracing, fixing missing or malformed traces, adding tool calls, generations, trace handles, thread/user context, Vercel AI SDK, OpenAI Agents SDK, LangChain, LangGraph, Langfuse side-by-side installs, or debugging Lemma trace delivery and trace shape.

- Skill: `uselemma/lemma-tracing` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add uselemma/lemma-tracing`
- Raw SKILL.md: https://api.skillmd.com/api/skills/uselemma/lemma-tracing/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: uselemma (https://skillmd.com/u/uselemma)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/uselemma/lemma-tracing

---


# Lemma Tracing

## Operating Mode

Integrate Lemma with the direct tracing SDK by default. The SDK sends trace JSON to Lemma and emits the trace shape Lemma understands.

Work in this order:

1. Detect the app language, framework, runtime, agent entry points, model calls, tool calls, streaming/finalization path, and any existing tracing.
2. Choose the smallest integration path that produces the Lemma trace contract.
3. Read the relevant docs/reference files before editing.
4. Present a concise plan when the user asks for one or when the integration touches multiple files.
5. Implement, verify with tests or a smoke trace, and use debug mode for delivery or shape issues.

## Contract First

Every integration must satisfy this product contract:

- One agent execution becomes one Lemma root trace.
- The root trace has a stable `name`, user input, final output or error, and `threadId` / `userId` when available.
- When the user supplies an agent name (Lemma's onboarding prompt does: "Name the agent `…`"), that exact string is the root trace `name` — the `functionId` / `agentName` / `agent_name` / `name` the integration reads. Lemma files the agent's context and artifacts under it, so do not rename or restyle it. The precedence between a user-supplied name, existing instrumentation, and names already in Lemma is owned by [lemma-artifacts › discover › The agent's name](../lemma-artifacts/references/discover.md#the-agents-name).
- LLM calls are generation children: `recordGeneration(...)` / `record_generation(...)` or `startGeneration(...)` / `start_generation(...)`.
- Tool invocations are tool children: `recordTool(...)` / `record_tool(...)` or `startTool(...)` / `start_tool(...)`.
- Retrieval, ranking, planning, routing, and app logic are spans: `recordSpan(...)` / `record_span(...)` or `startSpan(...)` / `start_span(...)`.
- Nested work is recorded on the parent span handle when it should appear under that parent.

If the app cannot produce this shape at the exact call site, pass IDs and record from the coordinator that knows the full trace. For a host and sandbox that are one turn, use `startTurn` / `attachTurn` / `apply`. See [references/direct-sdk.md](references/direct-sdk.md) and [Cross-process turns](https://docs.uselemma.ai/tracing/instrumentation/cross-process-turns).

When you assemble a trace yourself and deliver it with `ingest()` (queues, workers, backfills), send one complete payload (root input/output, thread/user, and all child spans) when the turn finishes. This is required; patching a trace over time is not currently supported. Retries of the same payload are safe (stable span IDs are skipped). `ingest()` is not an incremental merge API: omitted root fields do not preserve prior values, and analysis runs once after an idle debounce.

## Choose the Integration Path

| Situation | Preferred path |
| --- | --- |
| New or manually instrumented TypeScript app | Use `@uselemma/tracing` directly. See [references/direct-sdk.md](references/direct-sdk.md). |
| New or manually instrumented Python app | Use `uselemma-tracing` directly. See [references/direct-sdk.md](references/direct-sdk.md). |
| Vercel AI SDK v7 or v6 | Use `vercelAI()` in AI SDK telemetry and let it create/finalize the Lemma trace. Do not wrap normal AI SDK runs in `lemma.trace(...)`. See [references/vercel-ai-sdk.md](references/vercel-ai-sdk.md). |
| OpenAI Agents SDK | Use the built-in Lemma OpenAI Agents integration. Do not wrap normal agent runs in `lemma.trace(...)`; the processor/instrumentor creates the Lemma trace from Agents SDK events. |
| LangChain | Use the built-in Lemma LangChain callback handler. |
| LangGraph | Use the built-in Lemma LangGraph callback handler; it follows LangChain callback semantics with LangGraph defaults. |
| Mastra | Use the built-in Lemma Mastra exporter (`LemmaMastraExporter` / `mastra()`). Do not wrap normal Mastra agent/workflow runs in `lemma.trace(...)`; register the exporter in Mastra `Observability` and let it create one Lemma trace per run. See [Mastra](https://docs.uselemma.ai/integrations/mastra). |
| Streaming or callbacks where one function does not own the whole run | In TypeScript, use a trace handle and call/await `trace.end(...)` from the terminal callback or finalization path. In Python, prefer a callback trace around the owned run and record `start_*` handles inside that callback. |
| Host and sandbox such as E2B that are one user turn | TypeScript `startTurn` / `attachTurn` / `apply` (Python `start_turn` / `attach_turn`). Child records a journal; host `ingest()`s one root. See [references/direct-sdk.md](references/direct-sdk.md) and [Cross-process turns](https://docs.uselemma.ai/tracing/instrumentation/cross-process-turns). |
| App already has Langfuse | Keep Langfuse only if the customer still needs it, and add Lemma SDK tracing alongside it. Langfuse instrumentation is not sufficient for Lemma because it usually does not produce the Lemma trace contract. Do not route new Lemma work through Langfuse. |
| Existing OpenTelemetry only | Do not tear it out. Keep it if the user needs it, but use the Lemma SDK for the product trace contract unless the user explicitly asks for OTel export compatibility work. |

## Docs

Base URL: `https://docs.uselemma.ai`

Use docs in this order:

1. Fetch `https://docs.uselemma.ai/llms.txt` when live docs are needed.
2. Read the relevant current page before editing:
   - Quickstart: `https://docs.uselemma.ai/tracing/instrumentation/setup.md`
   - Step-by-step agent instrumentation: `https://docs.uselemma.ai/tracing/instrumentation/instrument-an-agent.md`
   - Traces and handles: `https://docs.uselemma.ai/tracing/instrumentation/traces.md`
   - Cross-process turns: `https://docs.uselemma.ai/tracing/instrumentation/cross-process-turns.md`
   - Generations: `https://docs.uselemma.ai/tracing/instrumentation/generations.md`
   - Tool calls: `https://docs.uselemma.ai/tracing/instrumentation/tool-calls.md`
   - Spans: `https://docs.uselemma.ai/tracing/instrumentation/spans.md`
   - Context: `https://docs.uselemma.ai/tracing/instrumentation/context.md`
   - Vercel AI SDK: `https://docs.uselemma.ai/integrations/vercel-ai.md`
   - OpenAI Agents SDK: `https://docs.uselemma.ai/integrations/openai-agents.md`
   - LangChain: `https://docs.uselemma.ai/integrations/langchain.md`
   - LangGraph: `https://docs.uselemma.ai/integrations/langgraph.md`
   - Mastra: `https://docs.uselemma.ai/integrations/mastra.md`
   - Trace contract: `https://docs.uselemma.ai/reference/trace-contract.md`
   - Troubleshooting: `https://docs.uselemma.ai/tracing/troubleshooting/common-issues.md`
   - Debug mode: `https://docs.uselemma.ai/tracing/troubleshooting/debug-mode.md`

Prefer local package types/tests over memory when the SDK API is available in the workspace.

## Detection Checklist

Inspect imports, startup files, agent handlers, and model/tool call sites:

| Signal | How to detect |
| --- | --- |
| Lemma SDK already present | `@uselemma/tracing`, `uselemma_tracing`, `Lemma`, `vercelAI`, `lemma.trace` |
| Agent boundary | request handler, job processor, CLI command, workflow step, streaming route |
| Vercel AI SDK | `generateText`, `streamText`, `generateObject`, `tool`, `telemetry`, `experimental_telemetry` from `ai` |
| OpenAI Agents SDK | `@openai/agents`, `openai-agents`, `Agent`, `Runner`, `run`, `addTraceProcessor` |
| LangChain | `langchain`, `@langchain/*`, `ChatOpenAI`, chains, callbacks |
| LangGraph | `langgraph`, `@langchain/langgraph`, graph `invoke`/`stream`, callbacks |
| Mastra | `@mastra/core`, `@mastra/observability`, `new Mastra(...)`, `Observability`, `agent.generate` |
| Model call | `openai`, `anthropic`, provider adapters, AI SDK model calls |
| Tool call | functions passed as tools, MCP calls, retrieval/search/order/payment helpers |
| Trace finalization | callback return, `onEnd`, `onFinish`, SSE close, queue completion, background job completion |
| Existing tracing | Langfuse, OpenTelemetry, OpenInference, Arize/Phoenix, Braintrust |

Ask one focused clarification question only when the agent boundary or finalization path is genuinely ambiguous.

## Implementation Rules

- Create one shared `Lemma` client on the server side.
- Use `LEMMA_API_KEY` and `LEMMA_PROJECT_ID`. The default endpoint is `https://api.uselemma.ai`; pass `baseUrl` / `base_url` only for staging or self-hosted deployments.
- Never expose `LEMMA_API_KEY` in browser code or `NEXT_PUBLIC_*` variables.
- Never write the value of `LEMMA_API_KEY` into any file, tracked or not, and do not ask the user to paste it into the conversation. A placeholder line such as `LEMMA_API_KEY=` in an env example is fine. `LEMMA_PROJECT_ID` is not a secret; fill it in when you know it.
- Use callback traces when one function owns the whole run.
- In TypeScript, use trace handles when the run is coordinated across callbacks, streaming, or helpers; do not set final duration until `trace.end(...)`.
- In Python, use callback traces as the root boundary and use `start_span`, `start_tool`, and `start_generation` on the active trace context for work in progress.
- For Vercel AI SDK and OpenAI Agents SDK, do not wrap normal runs in `lemma.trace(...)`; use the integration so it creates and closes the trace from framework lifecycle events.
- Record completed work with `recordSpan`, `recordTool`, and `recordGeneration`.
- Use `startSpan`, `startTool`, and `startGeneration` when you need a handle before the work finishes.
- Use `traceId` and `parentSpanId` for detached recording by ID in the **same** process.
- For a host and sandbox that are one user turn, use `startTurn` / `attachTurn` / `apply` (Python `start_turn` / `attach_turn`). The child records a journal with no API key; the host `ingest()`s once.
- Prefer native contract props such as `toolParameters`, `llmInputMessages`, `llmInvocationParameters`, and `model`; use raw `attributes` only as an escape hatch.
- Redact secrets and sensitive payloads before recording inputs or outputs.

## Debugging

Use debug mode whenever traces are missing, delayed, split into separate roots, individual spans show up as separate traces, spans are not nested under the expected parent, trace children are missing, tools/generations are missing, or input/output is blank. For the full diagnostic workflow, read [references/debug-mode.md](references/debug-mode.md).

Enable it before creating the client:

```typescript
import { enableDebugMode, Lemma } from "@uselemma/tracing";

enableDebugMode();
const lemma = new Lemma();
```

or:

```bash
LEMMA_DEBUG=1
```

(`LEMMA_DEBUG=true` also works.)

Expected successful sequence:

```text
[LEMMA:client] trace started
[LEMMA:client] sending trace
[LEMMA:client] trace sent
```

For trace handles, expect `trace handle created`, span-level debug logs as each handle starts/ends, then `sending trace` when the handle ends. Use the `spanCount` (`span_count` in Python) on `sending trace` to confirm children are actually included.

Common reads:

- No Lemma logs: debug mode is not enabled in the running process, or the traced path is not executing.
- `trace started` but no `sending trace`: callback did not finish, or a handle was never ended.
- `sending trace` then `trace ingest failed`: check credentials, project ID, `baseUrl`, status code, and network policy.
- Individual spans show up as traces: the child work is being recorded outside the root trace context; move it inside the callback or pass `traceId` / `parentSpanId`.
- Host and sandbox show up as two traces: use `startTurn` / `attachTurn` / `apply` so the child's journal nests under one root.
- Spans are not nested properly: record nested work on the parent span handle or pass the correct `parentSpanId` to detached helpers.
- Trace is missing spans: compare expected children to debug `span started` / `span ended` / `span recorded` logs and the final `spanCount`.
- `trace sent` but dashboard shape is wrong: compare the trace against the contract and record missing children with typed helpers.

## Verification

Use the strongest available evidence:

- Type-check and run focused tests for changed code.
- Run a smoke trace from the same runtime that handles real traffic.
- With debug mode enabled, confirm `trace sent` and the expected child count.

Validation checklist before considering an integration complete:

- One trace per agent run: one root trace, not sibling traces per model/tool call.
- Root trace has a stable `name`.
- Root trace records the user input.
- Root trace records the final output or error.
- LLM calls are typed generation children with `model`, input/messages, and output text when available.
- Tool calls are typed tool children with input arguments and output/error.
- App work is recorded as spans, nested under the correct parent when relevant.
- Related conversation turns share `threadId` / `thread_id`.
- Host and sandbox work that is one user turn is one root via `startTurn` / `attachTurn` / `apply`.
- TypeScript trace handles are ended from the terminal callback, `finally` block, or job completion path.
- Debug logs show `trace sent` and a final `spanCount` / `span_count` that matches the expected child records.
- The dashboard trace shape matches the code path: root, generations, tools, spans, parent/child nesting, and thread/user context when expected.

## Handing Back

End by telling the user what to set before they run the app: `LEMMA_API_KEY`, an ingest-scoped key from the **Connect your agent** step in Lemma onboarding or from **Settings → API keys** in the dashboard, and `LEMMA_PROJECT_ID`. Nothing is sent until both are set. If the user arrived here from `lemma-artifacts`, the agent name you used is the one they chose in onboarding; say so, so they can see the two line up.

## Skill Feedback

If this skill gives incorrect guidance, references a page that does not exist, or misses a scenario, offer to submit feedback. See [references/skill-feedback.md](references/skill-feedback.md).