Sentry Setup AI Monitoring

SOURCE: getsentry/sentry-for-claude

PATH: skills/sentry-setup-ai-monitoring/SKILL.md

DO NOT EDIT: This file is synced from external source

Setup Sentry AI Agent Monitoring

This skill helps configure Sentry's AI Agent Monitoring to track LLM calls, agent executions, tool usage, and token consumption.

When to Use This Skill

Invoke this skill when:

• User asks to "setup AI monitoring" or "add AI agent tracking"
• User wants to "monitor LLM calls" or "track OpenAI/Anthropic usage"
• User requests "AI observability" or "agent monitoring"
• User mentions tracking token usage, model latency, or AI costs
• User asks about instrumenting their AI/LLM code with Sentry

CRITICAL: Detection-First Approach

ALWAYS detect installed AI SDKs before suggesting configuration. Do not assume which AI library the user is using.

Step 1: Detect Platform and AI SDKs

For JavaScript/TypeScript Projects

Run these commands to detect installed AI packages:

# Check package.json for AI SDKs
grep -E '"(openai|@anthropic-ai/sdk|ai|@langchain|@google/genai|@langchain/langgraph)"' package.json

Supported JavaScript AI SDKs:

For Python Projects

# Check requirements.txt or pyproject.toml
grep -E '(openai|anthropic|langchain|huggingface)' requirements.txt pyproject.toml 2>/dev/null

Supported Python AI SDKs:

Step 2: Verify Sentry SDK Version

JavaScript

grep -E '"@sentry/(nextjs|react|node)"' package.json

Check version meets minimum for detected AI SDK (see table above).

Upgrade if needed:

npm install @sentry/nextjs@latest  # or appropriate package

Python

pip show sentry-sdk | grep Version

Upgrade if needed:

pip install --upgrade "sentry-sdk[openai]"  # include detected extras

Step 3: Verify Tracing is Enabled

AI Agent Monitoring requires tracing. Check that tracesSampleRate is set in Sentry.init().

If not enabled, inform the user:

AI Agent Monitoring requires tracing to be enabled. I'll add tracesSampleRate to your Sentry configuration.

Step 4: Configure Based on Detected SDK

IMPORTANT: Present Detection Results First

Before configuring, tell the user what you found:

I detected the following AI SDK(s) in your project:
- [SDK NAME] (version X.X.X)

Sentry has automatic integration support for this SDK. I'll configure the appropriate integration.

If no supported SDK is detected:

I didn't detect any AI SDKs with automatic Sentry integration support in your project.

Your options:
1. Manual Instrumentation - I can help you set up custom spans for your AI calls
2. Install a supported SDK - If you're planning to add one of the supported SDKs

Would you like to proceed with manual instrumentation? If so, please describe where your AI/LLM calls are located.

JavaScript Integration Configurations

OpenAI Integration

Automatic - Enabled by default in SDK 10.2.0+

Explicit configuration (to enable input/output recording):

import * as Sentry from "@sentry/nextjs"; // or @sentry/node, @sentry/react

Sentry.init({
  dsn: "YOUR_DSN_HERE",
  tracesSampleRate: 1.0,

  integrations: [
    Sentry.openAIIntegration({
      recordInputs: true,  // Capture prompts/messages
      recordOutputs: true, // Capture responses
    }),
  ],
});

Supported methods (auto-instrumented):

• chat.completions.create()
• responses.create()

Example usage (no changes needed):

import OpenAI from "openai";

const openai = new OpenAI();
const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Hello!" }],
});
// Automatically captured by Sentry

Anthropic Integration

Automatic - Enabled by default in SDK 10.12.0+

Explicit configuration:

import * as Sentry from "@sentry/nextjs";

Sentry.init({
  dsn: "YOUR_DSN_HERE",
  tracesSampleRate: 1.0,

  integrations: [
    Sentry.anthropicAIIntegration({
      recordInputs: true,
      recordOutputs: true,
    }),
  ],
});

Supported methods (auto-instrumented):

• messages.create() / messages.stream()
• messages.countTokens()
• completions.create()
• beta.messages.create()

Vercel AI SDK Integration

Automatic in Node runtime - Enabled by default in SDK 10.6.0+

For Edge runtime (Next.js) - Must be explicitly enabled in sentry.edge.config.ts:

import * as Sentry from "@sentry/nextjs";

Sentry.init({
  dsn: "YOUR_DSN_HERE",
  tracesSampleRate: 1.0,
  integrations: [Sentry.vercelAIIntegration()],
});

IMPORTANT: Telemetry must be enabled per-call:

import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";

const result = await generateText({
  model: openai("gpt-4o"),
  prompt: "Hello!",
  experimental_telemetry: {
    isEnabled: true,
    recordInputs: true,
    recordOutputs: true,
    functionId: "my-generation", // Optional but recommended
  },
});

Configuration options:

Sentry.vercelAIIntegration({
  recordInputs: true,
  recordOutputs: true,
  force: false, // Force enable even if AI module not detected
})

LangChain Integration

Requires explicit integration in SDK 10.22.0+

import * as Sentry from "@sentry/nextjs";

Sentry.init({
  dsn: "YOUR_DSN_HERE",
  tracesSampleRate: 1.0,

  integrations: [
    Sentry.langChainIntegration({
      recordInputs: true,
      recordOutputs: true,
    }),
  ],
});

Auto-instrumented operations:

• Chat model calls (gen_ai.request)
• LLM pipeline executions (gen_ai.pipeline)
• Chain invocations (gen_ai.invoke_agent)
• Tool calls (gen_ai.execute_tool)
• invoke(), stream(), batch() methods

LangGraph Integration

Requires explicit integration in SDK 10.25.0+

import * as Sentry from "@sentry/nextjs";

Sentry.init({
  dsn: "YOUR_DSN_HERE",
  tracesSampleRate: 1.0,

  integrations: [
    Sentry.langGraphIntegration({
      recordInputs: true,
      recordOutputs: true,
    }),
  ],
});

Auto-instrumented:

• Agent creation (StateGraph compilation)
• Agent invocation (invoke() method)

Google GenAI Integration

Automatic - Enabled by default in SDK 10.14.0+

Explicit configuration:

import * as Sentry from "@sentry/nextjs";

Sentry.init({
  dsn: "YOUR_DSN_HERE",
  tracesSampleRate: 1.0,

  integrations: [
    Sentry.googleGenAIIntegration({
      recordInputs: true,
      recordOutputs: true,
    }),
  ],
});

Supported methods:

• models.generateContent() / models.generateContentStream()
• chats.create()
• sendMessage() / sendMessageStream()

Python Integration Configurations

OpenAI Integration (Python)

Install with extra:

pip install "sentry-sdk[openai]"

Configuration:

import sentry_sdk
from sentry_sdk.integrations.openai import OpenAIIntegration

sentry_sdk.init(
    dsn="YOUR_DSN_HERE",
    traces_sample_rate=1.0,
    send_default_pii=True,  # Required for input/output capture
    integrations=[
        OpenAIIntegration(
            include_prompts=True,  # Capture inputs/outputs
            tiktoken_encoding_name="cl100k_base",  # For streaming token count
        ),
    ],
)

Usage (auto-instrumented):

from openai import OpenAI

client = OpenAI()

with sentry_sdk.start_transaction(name="AI inference"):
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Hello!"}]
    )

Anthropic Integration (Python)

Install with extra:

pip install "sentry-sdk[anthropic]"

Configuration:

import sentry_sdk
from sentry_sdk.integrations.anthropic import AnthropicIntegration

sentry_sdk.init(
    dsn="YOUR_DSN_HERE",
    traces_sample_rate=1.0,
    send_default_pii=True,
    integrations=[
        AnthropicIntegration(
            include_prompts=True,
        ),
    ],
)

Usage:

from anthropic import Anthropic

client = Anthropic()

with sentry_sdk.start_transaction(name="Claude inference"):
    response = client.messages.create(
        model="claude-3-5-sonnet-20240620",
        max_tokens=1024,
        messages=[{"role": "user", "content": "Hello!"}]
    )

LangChain Integration (Python)

Install with extra:

pip install "sentry-sdk[langchain]"

Configuration:

import sentry_sdk
from sentry_sdk.integrations.langchain import LangchainIntegration

sentry_sdk.init(
    dsn="YOUR_DSN_HERE",
    traces_sample_rate=1.0,
    send_default_pii=True,  # Required for LLM input/output capture
    integrations=[
        LangchainIntegration(
            include_prompts=True,
        ),
    ],
)

Usage:

with sentry_sdk.start_transaction(name="LangChain agent"):
    agent_executor = AgentExecutor(agent=agent, tools=tools)
    result = agent_executor.invoke({"input": "your prompt"})

Hugging Face Hub Integration (Python)

Install with extra:

pip install "sentry-sdk[huggingface_hub]"

Configuration:

import sentry_sdk
from sentry_sdk.integrations.huggingface_hub import HuggingfaceHubIntegration

sentry_sdk.init(
    dsn="YOUR_DSN_HERE",
    traces_sample_rate=1.0,
    send_default_pii=True,
    integrations=[
        HuggingfaceHubIntegration(
            include_prompts=True,
        ),
    ],
)

Manual Instrumentation (Last Resort)

IMPORTANT: Only use manual instrumentation when:

1. No supported AI SDK is detected
2. User explicitly confirms they want manual instrumentation
3. User describes where their AI calls are located

Ask User Before Proceeding

I'll help you set up manual AI instrumentation. To do this effectively, I need to know:

1. Where are your AI/LLM calls located? (file paths)
2. What AI provider/model are you using?
3. What operations do you want to track?
   - LLM requests (prompts, completions)
   - Agent executions
   - Tool calls
   - Agent handoffs

Manual Span Types

Four span types are required for AI Agents Insights:

1. AI Request Span (gen_ai.request)

Tracks individual LLM calls:

import * as Sentry from "@sentry/nextjs";

async function callLLM(prompt, model = "custom-model") {
  return await Sentry.startSpan(
    {
      op: "gen_ai.request",
      name: `LLM request ${model}`,
      attributes: {
        "gen_ai.request.model": model,
        "gen_ai.request.temperature": 0.7,
        "gen_ai.request.max_tokens": 1000,
      },
    },
    async (span) => {
      // Record input messages
      span.setAttribute(
        "gen_ai.request.messages",
        JSON.stringify([{ role: "user", content: prompt }])
      );

      const startTime = performance.now();

      // Your actual LLM call here
      const result = await yourLLMClient.complete(prompt);

      // Record output and metrics
      span.setAttribute("gen_ai.response.text", result.text);
      span.setAttribute("gen_ai.usage.input_tokens", result.inputTokens || 0);
      span.setAttribute("gen_ai.usage.output_tokens", result.outputTokens || 0);

      return result;
    }
  );
}

2. Invoke Agent Span (gen_ai.invoke_agent)

Tracks full agent execution lifecycle:

async function runAgent(task) {
  return await Sentry.startSpan(
    {
      op: "gen_ai.invoke_agent",
      name: "Execute AI Agent",
      attributes: {
        "gen_ai.agent.name": "my-agent",
        "gen_ai.agent.available_tools": JSON.stringify(["search", "calculate"]),
      },
    },
    async (span) => {
      span.setAttribute("gen_ai.agent.input", task);

      // Agent execution logic (may include multiple LLM calls and tool uses)
      const result = await agent.execute(task);

      span.setAttribute("gen_ai.agent.output", JSON.stringify(result));
      span.setAttribute("gen_ai.usage.total_tokens", result.totalTokens || 0);

      return result;
    }
  );
}

3. Execute Tool Span (gen_ai.execute_tool)

Tracks tool/function calls:

async function executeTool(toolName, toolInput) {
  return await Sentry.startSpan(
    {
      op: "gen_ai.execute_tool",
      name: `Tool: ${toolName}`,
      attributes: {
        "gen_ai.tool.name": toolName,
        "gen_ai.tool.description": getToolDescription(toolName),
      },
    },
    async (span) => {
      span.setAttribute("gen_ai.tool.input", JSON.stringify(toolInput));

      const result = await tools[toolName](toolInput);

      span.setAttribute("gen_ai.tool.output", JSON.stringify(result));

      return result;
    }
  );
}

4. Handoff Span (gen_ai.handoff)

Tracks agent-to-agent control transitions:

async function handoffToAgent(fromAgent, toAgent, context) {
  return await Sentry.startSpan(
    {
      op: "gen_ai.handoff",
      name: `Handoff: ${fromAgent} -> ${toAgent}`,
      attributes: {
        "gen_ai.handoff.from_agent": fromAgent,
        "gen_ai.handoff.to_agent": toAgent,
      },
    },
    async (span) => {
      span.setAttribute("gen_ai.handoff.context", JSON.stringify(context));

      // Perform handoff
      const result = await agents[toAgent].receive(context);

      return result;
    }
  );
}

Python Manual Instrumentation

import sentry_sdk
import json

def call_llm(prompt, model="custom-model"):
    with sentry_sdk.start_span(
        op="gen_ai.request",
        name=f"LLM request {model}",
    ) as span:
        span.set_data("gen_ai.request.model", model)
        span.set_data("gen_ai.request.messages", json.dumps([
            {"role": "user", "content": prompt}
        ]))

        # Your actual LLM call
        result = your_llm_client.complete(prompt)

        span.set_data("gen_ai.response.text", result.text)
        span.set_data("gen_ai.usage.input_tokens", result.input_tokens)
        span.set_data("gen_ai.usage.output_tokens", result.output_tokens)

        return result

Span Attributes Reference

Required Attributes

Token Usage Attributes

Request Attributes

Response Attributes

Agent Attributes

Tool Attributes

Note: All complex data must be JSON stringified. Span attributes only allow primitive types.

Framework-Specific Notes

Next.js

• Configure in instrumentation-client.ts, sentry.server.config.ts, sentry.edge.config.ts
• Vercel AI SDK requires explicit edge runtime configuration
• Server-side LLM calls captured in server config
• Client-side calls (if any) in client config

Node.js / Express

• Configure in entry point or dedicated sentry config
• All integrations work in Node runtime

React (Client-Only)

• Limited usefulness - most LLM calls should be server-side
• Consider moving AI calls to API routes

Python (Django/Flask/FastAPI)

• Configure in settings or app initialization
• Wrap LLM calls in transactions for proper span hierarchy

Privacy and PII Considerations

Prompts and outputs are considered PII. To capture them:

JavaScript

Sentry.init({
  sendDefaultPii: true,
  // OR configure per-integration:
  integrations: [
    Sentry.openAIIntegration({
      recordInputs: true,
      recordOutputs: true,
    }),
  ],
});

Python

sentry_sdk.init(
    send_default_pii=True,
    # OR configure per-integration:
    integrations=[
        OpenAIIntegration(include_prompts=True),
    ],
)

To exclude prompts/outputs:

• Set recordInputs: false / recordOutputs: false (JS)
• Set include_prompts=False (Python)

Verification Steps

After setup, verify AI monitoring is working:

JavaScript

// Trigger a test LLM call
const response = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Say 'Sentry test successful'" }],
});
console.log("Test complete:", response.choices[0].message.content);

Python

with sentry_sdk.start_transaction(name="AI Test"):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Say 'Sentry test'"}]
    )
    print("Test complete:", response.choices[0].message.content)

Check in Sentry:

1. Go to Traces > AI Spans tab
2. Look for spans with gen_ai.* operations
3. Verify token usage and latency are captured

Common Issues and Solutions

Issue: AI spans not appearing

Solutions:

1. Verify tracing is enabled (tracesSampleRate > 0)
2. Check SDK version meets minimum requirements
3. Ensure integration is properly configured
4. For Vercel AI: Check experimental_telemetry.isEnabled: true

Issue: Token counts missing

Solutions:

1. Some providers don't return token counts for streaming
2. For Python OpenAI streaming, install tiktoken
3. Manual instrumentation: Ensure you're setting token attributes

Issue: Prompts/outputs not captured

Solutions:

1. Enable sendDefaultPii: true or per-integration recording
2. Check recordInputs/recordOutputs or include_prompts settings

Issue: Integration not detected

Solutions:

1. Verify AI package is installed and version is compatible
2. For Vercel AI edge runtime, explicitly add integration
3. Use force: true option if module detection fails

Summary Checklist

## Sentry AI Agent Monitoring Setup Complete

### Detection Results:
- [ ] AI SDK(s) detected: [LIST DETECTED SDKS]
- [ ] Sentry SDK version verified
- [ ] Tracing enabled

### Configuration Applied:
- [ ] Integration(s) added to Sentry.init()
- [ ] Input/output recording configured
- [ ] PII settings reviewed

### For Vercel AI SDK (if applicable):
- [ ] Edge runtime integration added
- [ ] experimental_telemetry enabled in function calls

### Next Steps:
1. Make an AI/LLM call to verify
2. Check Sentry > Traces > AI Spans
3. Review AI Agents Dashboard for metrics

Quick Reference