Lemma Tracing

Lemma Tracing

Operating Mode

Integrate Lemma with the direct tracing SDK by default. The SDK sends trace JSON to Lemma and emits the trace shape Lemma understands.

Work in this order:

1. Detect the app language, framework, runtime, agent entry points, model calls, tool calls, streaming/finalization path, and any existing tracing.
2. Choose the smallest integration path that produces the Lemma trace contract.
3. Read the relevant docs/reference files before editing.
4. Present a concise plan when the user asks for one or when the integration touches multiple files.
5. Implement, verify with tests or a smoke trace, and use debug mode for delivery or shape issues.

Contract First

Every integration must satisfy this product contract:

• One agent execution becomes one Lemma root trace.
• The root trace has a stable name, user input, final output or error, and threadId / userId when available.
• When the user supplies an agent name (Lemma's onboarding prompt does: "Name the agent …"), that exact string is the root trace name — the functionId / agentName / agent_name / name the integration reads. Lemma files the agent's context and artifacts under it, so do not rename or restyle it. The precedence between a user-supplied name, existing instrumentation, and names already in Lemma is owned by lemma-artifacts › discover › The agent's name.
• LLM calls are generation children: recordGeneration(...) / record_generation(...) or startGeneration(...) / start_generation(...).
• Tool invocations are tool children: recordTool(...) / record_tool(...) or startTool(...) / start_tool(...).
• Retrieval, ranking, planning, routing, and app logic are spans: recordSpan(...) / record_span(...) or startSpan(...) / start_span(...).
• Nested work is recorded on the parent span handle when it should appear under that parent.

If the app cannot produce this shape at the exact call site, pass IDs and record from the coordinator that knows the full trace. For a host and sandbox that are one turn, use startTurn / attachTurn / apply. See references/direct-sdk.md and Cross-process turns.

When you assemble a trace yourself and deliver it with ingest() (queues, workers, backfills), send one complete payload (root input/output, thread/user, and all child spans) when the turn finishes. This is required; patching a trace over time is not currently supported. Retries of the same payload are safe (stable span IDs are skipped). ingest() is not an incremental merge API: omitted root fields do not preserve prior values, and analysis runs once after an idle debounce.

Choose the Integration Path

Docs

Base URL: https://docs.uselemma.ai

Use docs in this order:

1. Fetch https://docs.uselemma.ai/llms.txt when live docs are needed.
2. Read the relevant current page before editing:
   • Quickstart: https://docs.uselemma.ai/tracing/instrumentation/setup.md
   • Step-by-step agent instrumentation: https://docs.uselemma.ai/tracing/instrumentation/instrument-an-agent.md
   • Traces and handles: https://docs.uselemma.ai/tracing/instrumentation/traces.md
   • Cross-process turns: https://docs.uselemma.ai/tracing/instrumentation/cross-process-turns.md
   • Generations: https://docs.uselemma.ai/tracing/instrumentation/generations.md
   • Tool calls: https://docs.uselemma.ai/tracing/instrumentation/tool-calls.md
   • Spans: https://docs.uselemma.ai/tracing/instrumentation/spans.md
   • Context: https://docs.uselemma.ai/tracing/instrumentation/context.md
   • Vercel AI SDK: https://docs.uselemma.ai/integrations/vercel-ai.md
   • OpenAI Agents SDK: https://docs.uselemma.ai/integrations/openai-agents.md
   • LangChain: https://docs.uselemma.ai/integrations/langchain.md
   • LangGraph: https://docs.uselemma.ai/integrations/langgraph.md
   • Mastra: https://docs.uselemma.ai/integrations/mastra.md
   • Trace contract: https://docs.uselemma.ai/reference/trace-contract.md
   • Troubleshooting: https://docs.uselemma.ai/tracing/troubleshooting/common-issues.md
   • Debug mode: https://docs.uselemma.ai/tracing/troubleshooting/debug-mode.md

Prefer local package types/tests over memory when the SDK API is available in the workspace.

Detection Checklist

Inspect imports, startup files, agent handlers, and model/tool call sites:

Ask one focused clarification question only when the agent boundary or finalization path is genuinely ambiguous.

Implementation Rules

• Create one shared Lemma client on the server side.
• Use LEMMA_API_KEY and LEMMA_PROJECT_ID. The default endpoint is https://api.uselemma.ai; pass baseUrl / base_url only for staging or self-hosted deployments.
• Never expose LEMMA_API_KEY in browser code or NEXT_PUBLIC_* variables.
• Never write the value of LEMMA_API_KEY into any file, tracked or not, and do not ask the user to paste it into the conversation. A placeholder line such as LEMMA_API_KEY= in an env example is fine. LEMMA_PROJECT_ID is not a secret; fill it in when you know it.
• Use callback traces when one function owns the whole run.
• In TypeScript, use trace handles when the run is coordinated across callbacks, streaming, or helpers; do not set final duration until trace.end(...).
• In Python, use callback traces as the root boundary and use start_span, start_tool, and start_generation on the active trace context for work in progress.
• For Vercel AI SDK and OpenAI Agents SDK, do not wrap normal runs in lemma.trace(...); use the integration so it creates and closes the trace from framework lifecycle events.
• Record completed work with recordSpan, recordTool, and recordGeneration.
• Use startSpan, startTool, and startGeneration when you need a handle before the work finishes.
• Use traceId and parentSpanId for detached recording by ID in the same process.
• For a host and sandbox that are one user turn, use startTurn / attachTurn / apply (Python start_turn / attach_turn). The child records a journal with no API key; the host ingest()s once.
• Prefer native contract props such as toolParameters, llmInputMessages, llmInvocationParameters, and model; use raw attributes only as an escape hatch.
• Redact secrets and sensitive payloads before recording inputs or outputs.

Debugging

Use debug mode whenever traces are missing, delayed, split into separate roots, individual spans show up as separate traces, spans are not nested under the expected parent, trace children are missing, tools/generations are missing, or input/output is blank. For the full diagnostic workflow, read references/debug-mode.md.

Enable it before creating the client:

import { enableDebugMode, Lemma } from "@uselemma/tracing";

enableDebugMode();
const lemma = new Lemma();

or:

LEMMA_DEBUG=1

(LEMMA_DEBUG=true also works.)

Expected successful sequence:

[LEMMA:client] trace started
[LEMMA:client] sending trace
[LEMMA:client] trace sent

For trace handles, expect trace handle created, span-level debug logs as each handle starts/ends, then sending trace when the handle ends. Use the spanCount (span_count in Python) on sending trace to confirm children are actually included.

Common reads:

• No Lemma logs: debug mode is not enabled in the running process, or the traced path is not executing.
• trace started but no sending trace: callback did not finish, or a handle was never ended.
• sending trace then trace ingest failed: check credentials, project ID, baseUrl, status code, and network policy.
• Individual spans show up as traces: the child work is being recorded outside the root trace context; move it inside the callback or pass traceId / parentSpanId.
• Host and sandbox show up as two traces: use startTurn / attachTurn / apply so the child's journal nests under one root.
• Spans are not nested properly: record nested work on the parent span handle or pass the correct parentSpanId to detached helpers.
• Trace is missing spans: compare expected children to debug span started / span ended / span recorded logs and the final spanCount.
• trace sent but dashboard shape is wrong: compare the trace against the contract and record missing children with typed helpers.

Verification

Use the strongest available evidence:

• Type-check and run focused tests for changed code.
• Run a smoke trace from the same runtime that handles real traffic.
• With debug mode enabled, confirm trace sent and the expected child count.

Validation checklist before considering an integration complete:

• One trace per agent run: one root trace, not sibling traces per model/tool call.
• Root trace has a stable name.
• Root trace records the user input.
• Root trace records the final output or error.
• LLM calls are typed generation children with model, input/messages, and output text when available.
• Tool calls are typed tool children with input arguments and output/error.
• App work is recorded as spans, nested under the correct parent when relevant.
• Related conversation turns share threadId / thread_id.
• Host and sandbox work that is one user turn is one root via startTurn / attachTurn / apply.
• TypeScript trace handles are ended from the terminal callback, finally block, or job completion path.
• Debug logs show trace sent and a final spanCount / span_count that matches the expected child records.
• The dashboard trace shape matches the code path: root, generations, tools, spans, parent/child nesting, and thread/user context when expected.

Handing Back

End by telling the user what to set before they run the app: LEMMA_API_KEY, an ingest-scoped key from the Connect your agent step in Lemma onboarding or from Settings → API keys in the dashboard, and LEMMA_PROJECT_ID. Nothing is sent until both are set. If the user arrived here from lemma-artifacts, the agent name you used is the one they chose in onboarding; say so, so they can see the two line up.

Skill Feedback

If this skill gives incorrect guidance, references a page that does not exist, or misses a scenario, offer to submit feedback. See references/skill-feedback.md.