Langfuse Incident Runbook

Langfuse Incident Runbook

Overview

Step-by-step procedures for responding to Langfuse-related incidents.

Prerequisites

• Access to Langfuse dashboard
• Application logs access
• Metrics/monitoring dashboards
• Escalation contacts

Incident Severity Levels

Quick Diagnostics

Step 1: Initial Assessment (2 minutes)

#!/bin/bash
# quick-diagnosis.sh

echo "=== Langfuse Quick Diagnosis ==="
echo "Time: $(date)"
echo ""

# 1. Check Langfuse status
echo "1. Langfuse Status:"
curl -s https://status.langfuse.com/api/v2/status.json | jq '.status.description'

# 2. Check API connectivity
echo ""
echo "2. API Connectivity:"
curl -s -o /dev/null -w "HTTP %{http_code} in %{time_total}s\n" \
  https://cloud.langfuse.com/api/public/health

# 3. Check authentication
echo ""
echo "3. Auth Test:"
AUTH=$(echo -n "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" | base64)
curl -s -o /dev/null -w "HTTP %{http_code}\n" \
  -H "Authorization: Basic $AUTH" \
  "https://cloud.langfuse.com/api/public/traces?limit=1"

# 4. Check application health
echo ""
echo "4. Application Metrics:"
curl -s http://localhost:3000/api/metrics | grep langfuse | head -5

Step 2: Determine Incident Type

Section A: Traces Not Appearing

Symptoms

• Dashboard shows no new traces
• No errors in application logs
• Application functioning normally

Diagnosis Steps

// 1. Verify SDK is enabled
console.log("Langfuse enabled:", process.env.LANGFUSE_ENABLED !== "false");
console.log("Environment:", process.env.NODE_ENV);

// 2. Check for pending events
// Add this to your code temporarily
const langfuse = getLangfuse();
console.log("Pending events:", langfuse.pendingItems?.length || "unknown");

// 3. Force flush and check for errors
try {
  await langfuse.flushAsync();
  console.log("Flush successful");
} catch (error) {
  console.error("Flush failed:", error);
}

Resolution Steps

1. Check shutdown handlers

   // Ensure shutdown is registered
   process.on("beforeExit", async () => {
     await langfuse.shutdownAsync();
   });
   

2. Reduce batch size temporarily

   const langfuse = new Langfuse({
     flushAt: 1,        // Immediate flush
     flushInterval: 1000,
   });
   

3. Enable debug logging

   DEBUG=langfuse* npm start
   

Section B: Authentication Errors

Symptoms

• 401 Unauthorized errors
• 403 Forbidden errors
• "Invalid API key" messages

Diagnosis Steps

# 1. Verify environment variables
echo "Public key starts with: ${LANGFUSE_PUBLIC_KEY:0:10}"
echo "Secret key is set: ${LANGFUSE_SECRET_KEY:+yes}"
echo "Host: ${LANGFUSE_HOST:-https://cloud.langfuse.com}"

# 2. Test credentials directly
curl -v -X GET \
  -H "Authorization: Basic $(echo -n "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" | base64)" \
  "${LANGFUSE_HOST:-https://cloud.langfuse.com}/api/public/traces?limit=1"

Resolution Steps

1. Verify keys match project

   • Go to Langfuse Dashboard > Settings > API Keys
   • Ensure keys are from the correct project
   • Check keys haven't been revoked

2. Check for key rotation

   • If keys were recently rotated, update all environments
   • Verify secret manager has latest values

3. Verify host URL

   • Cloud: https://cloud.langfuse.com
   • Self-hosted: Your instance URL (no trailing slash)

Section C: High Latency / Timeouts

Symptoms

• Slow API responses
• Request timeouts
• 429 Rate limit errors

Diagnosis Steps

// Check flush timing
const start = Date.now();
await langfuse.flushAsync();
console.log(`Flush took ${Date.now() - start}ms`);

// Check batch sizes
console.log("Current batch size:", langfuse.pendingItems?.length);

# Network latency test
curl -w "DNS: %{time_namelookup}s\nConnect: %{time_connect}s\nTLS: %{time_appconnect}s\nTotal: %{time_total}s\n" \
  -o /dev/null -s https://cloud.langfuse.com/api/public/health

Resolution Steps

1. For rate limits

   // Increase batching
   const langfuse = new Langfuse({
     flushAt: 50,
     flushInterval: 10000,
   });
   

2. For network issues

   • Check firewall rules allow outbound HTTPS
   • Verify DNS resolution
   • Consider using a closer region (self-hosted)

3. Implement circuit breaker

   class CircuitBreaker {
     private failures = 0;
     private lastFailure?: Date;
     private readonly threshold = 5;
     private readonly resetMs = 60000;
   
     async execute<T>(operation: () => Promise<T>): Promise<T | null> {
       if (this.isOpen()) {
         console.warn("Circuit breaker open, skipping Langfuse");
         return null;
       }
   
       try {
         const result = await operation();
         this.reset();
         return result;
       } catch (error) {
         this.recordFailure();
         throw error;
       }
     }
   
     private isOpen(): boolean {
       if (this.failures < this.threshold) return false;
       if (!this.lastFailure) return false;
       return Date.now() - this.lastFailure.getTime() < this.resetMs;
     }
   
     private recordFailure() {
       this.failures++;
       this.lastFailure = new Date();
     }
   
     private reset() {
       this.failures = 0;
       this.lastFailure = undefined;
     }
   }
   

Section D: Missing/Partial Data

Symptoms

• Some traces appear, others don't
• Missing spans or generations
• Incomplete trace data

Diagnosis Steps

// Check for errors in trace operations
const trace = langfuse.trace({ name: "test" });
console.log("Trace ID:", trace.id);

const span = trace.span({ name: "test-span" });
console.log("Span ID:", span.id);

// Verify end() is called
span.end({ output: { test: true } });
console.log("Span ended");

await langfuse.flushAsync();
console.log("Flushed");

Resolution Steps

1. Ensure all spans are ended

   const span = trace.span({ name: "operation" });
   try {
     return await doWork();
   } finally {
     span.end(); // Always end in finally
   }
   

2. Check for exceptions swallowing

   try {
     await langfuse.flushAsync();
   } catch (error) {
     console.error("Langfuse flush error:", error);
     // Don't swallow - log for debugging
   }
   

Section E: Langfuse Service Outage

Symptoms

• status.langfuse.com shows issues
• All API calls failing
• Multiple users affected

Immediate Actions

1. Check status page: https://status.langfuse.com

2. Enable fallback mode

   // Graceful degradation
   const langfuse = new Langfuse({
     enabled: false, // Disable during outage
   });
   

3. Queue events locally

   // Store events to file during outage
   const pendingEvents: any[] = [];
   
   function queueEvent(event: any) {
     pendingEvents.push({
       ...event,
       timestamp: new Date().toISOString(),
     });
   
     if (pendingEvents.length > 1000) {
       // Write to file
       fs.writeFileSync(
         `langfuse-backup-${Date.now()}.json`,
         JSON.stringify(pendingEvents)
       );
       pendingEvents.length = 0;
     }
   }
   

4. Monitor for recovery

   # Watch status
   watch -n 30 'curl -s https://status.langfuse.com/api/v2/status.json | jq .status'
   

Post-Incident Checklist

• Verify traces are appearing in dashboard
• Check no data was lost during incident
• Review error rates returning to normal
• Update incident documentation
• Schedule post-mortem if P1/P2
• Update runbook with learnings

Escalation Contacts

Resources

• Langfuse Status
• Langfuse Discord
• Langfuse GitHub Issues

Next Steps

For data export and retention, see langfuse-data-handling.