Error Handler

--- name: error-handler description: Design error handling, structured logging, and observability with OpenTelemetry (traces, metrics, logs), error classification, recovery patterns (retry with jitter, circuit breaker, bulkhead, timeout), error budgets/SLOs with burn rate alerts, and production incident triage. Use when user asks to implement error handling, logging, monitoring, observability, OpenTelemetry, error boundaries, circuit breakers, retry logic, or SLO tracking. Do NOT use for incident runbooks (use runbook-gen), vendor-specific APM setup (Datadog, Sentry agent config), or K8s debugging. triggers: - "error handling" - "logging" - "observability" - "OpenTelemetry" - "error boundary" - "circuit breaker" - "retry logic" - "SLO" - "error budget" - "structured logging" - "monitoring" - "tracing" negatives: - "incident runbook" - "on-call" - "Datadog setup" - "Sentry config" - "K8s debugging" - "APM vendor setup" license: MIT compatibility: opencode metadata: workflow: backend audience: developers version: "4.0.0" author: shokunin allowed-tools: Read Bash Write Grep

Error Handler

Observability that makes debugging fast and production predictable. Based on Google SRE (Site Reliability Engineering), OpenTelemetry semantic conventions, and production patterns from Sentry, Honeycomb, and Datadog.

Sub-Commands

The 3 Signals (OpenTelemetry)

All three signals share trace_id + span_id for correlation.

Workflow

Step 1: Classify errors

Step 2: Set up OpenTelemetry (exact configuration)

import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { Resource } from '@opentelemetry/resources'
import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions'

const sdk = new NodeSDK({
  resource: new Resource({
    [ATTR_SERVICE_NAME]: 'api-service',
    'deployment.environment': process.env.NODE_ENV,
  }),
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
  }),
  instrumentations: [
    new HttpInstrumentation(),
    new ExpressInstrumentation(),
    new PgInstrumentation(),
    new RedisInstrumentation(),
  ],
})

sdk.start()

process.on('SIGTERM', async () => {
  await sdk.shutdown()
})

Sampling strategy:

• Dev/staging: AlwaysOn (100%)
• Production: ParentBased + TraceIdRatioBased(0.1) (10% head-based)
• Never sample errors out. Use AlwaysOn for traces with status=ERROR.

Import note: ParentBasedSampler is imported from @opentelemetry/sdk-trace-base, NOT from @opentelemetry/sdk-node. If using @opentelemetry/sdk-node, configure it via the sampler option in NodeSDK constructor.

Step 3: Implement structured error middleware

import { trace, SpanStatusCode } from '@opentelemetry/api'
import { randomUUID } from 'crypto'

app.use((err: Error, req: Request, res: Response, next: NextFunction) => {
  const span = trace.getActiveSpan()
  const requestId = req.headers['x-request-id'] || randomUUID()

  span?.recordException(err)
  span?.setStatus({ code: SpanStatusCode.ERROR, message: err.message })

  logger.error({
    message: err.message,
    errorName: err.name,
    stack: err.stack,
    requestId,
    trace_id: span?.spanContext().traceId,
    span_id: span?.spanContext().spanId,
    path: req.path,
    method: req.method,
    userId: req.user?.id,
  })

  if (err instanceof ValidationError) {
    return res.status(400).json({
      error: {
        code: 'VALIDATION_ERROR',
        message: err.message,
        details: err.details,
        requestId,
      },
    })
  }

  if (err instanceof NotFoundError) {
    return res.status(404).json({
      error: { code: 'NOT_FOUND', message: err.message, requestId },
    })
  }

  // Generic fallback - never leak internal state
  res.status(500).json({
    error: {
      code: 'INTERNAL_ERROR',
      message: 'An unexpected error occurred',
      requestId,
    },
  })
})

Step 4: Recovery patterns (exact implementations)

Retry with exponential backoff + jitter

async function withRetry<T>(
  fn: () => Promise<T>,
  options: { maxRetries?: number; baseMs?: number; maxMs?: number } = {}
): Promise<T> {
  const { maxRetries = 3, baseMs = 200, maxMs = 10000 } = options

  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await fn()
    } catch (error) {
      if (attempt === maxRetries) throw error
      if (!isRetryable(error)) throw error

      const exponentialDelay = baseMs * Math.pow(2, attempt)
      const jitter = Math.random() * 0.3 * exponentialDelay
      const delay = Math.min(exponentialDelay + jitter, maxMs)

      await new Promise(r => setTimeout(r, delay))
    }
  }
  throw new Error('Unreachable')
}

function isRetryable(error: any): boolean {
  const retryableStatuses = [408, 429, 502, 503, 504]
  if (error?.status && retryableStatuses.includes(error.status)) return true
  if (error?.code === 'ETIMEDOUT' || error?.code === 'ECONNRESET') return true
  return false
}

// Never retry: 400, 401, 403, 404, 409, 422

Circuit breaker

class CircuitBreaker {
  private failures = 0
  private state: 'closed' | 'open' | 'half-open' = 'closed'
  private lastFailureTime = 0

  constructor(
    private failureThreshold: number = 5,
    private resetTimeout: number = 30000
  ) {}

  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === 'open') {
      if (Date.now() - this.lastFailureTime > this.resetTimeout) {
        this.state = 'half-open'
      } else {
        throw new Error('Circuit breaker is open')
      }
    }

    try {
      const result = await fn()
      if (this.state === 'half-open') {
        this.state = 'closed'
        this.failures = 0
      }
      return result
    } catch (error) {
      this.failures++
      this.lastFailureTime = Date.now()

      if (this.failures >= this.failureThreshold) {
        this.state = 'open'
      }
      throw error
    }
  }
}

Timeout

function withTimeout<T>(promise: Promise<T>, timeoutMs: number): Promise<T> {
  const timeout = new Promise<T>((_, reject) =>
    setTimeout(() => reject(new Error(`Operation timed out after ${timeoutMs}ms`)), timeoutMs)
  )
  return Promise.race([promise, timeout])
}

// Usage
const result = await withTimeout(fetch('https://api.example.com/data'), 5000)

Step 5: Error budgets + SLO

Burn rate alert (PromQL for 99.9% SLO):

sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
> 0.001 * 6

Multi-window, multi-burn-rate approach (Google SRE):

• Short window (5m) + high burn rate (6-10x) ? Page on-call immediately
• Long window (1h) + low burn rate (2-3x) ? Ticket for next business day

Structured Logging Schema

Every log line must be JSON with these fields:

{
  "timestamp": "2026-05-16T10:21:27.000Z",
  "level": "error",
  "message": "Failed to process payment",
  "trace_id": "0af7651916cd43dd8448eb211c80319c",
  "span_id": "b7ad6b7169203331",
  "request_id": "req_a1b2c3d4",
  "service": "payment-service",
  "environment": "production",
  "userId": "user_abc123",
  "path": "/api/payments",
  "method": "POST",
  "duration_ms": 234,
  "error": {
    "name": "PaymentFailedError",
    "message": "Card declined",
    "code": "CARD_DECLINED"
  }
}

Rules:

• Never log PII, tokens, passwords, or full credit card numbers
• Always include trace_id, span_id, request_id for correlation
• duration_ms on every request. p95 > 200ms ? investigate.

Production Checklist

• OpenTelemetry SDK initialized at app startup (before any imports)
• Auto-instrumentations for HTTP, DB, messaging, Redis
• Every route handler wrapped in try/catch or error middleware
• Structured JSON logging (never plain text)
• trace_id + span_id + request_id in every log line
• Recovery: retry with jitter on external calls. Timeout on every external call.
• Circuit breaker for critical downstream services
• SLO defined for every service (99.9% critical, 99.5% standard)
• Burn rate alerts: multi-window (5m short, 1h long)
• No PII, tokens, or secrets in logs or error messages
• Error classification by type (not generic catch-all)
• Sampling: 10% production, 100% for errors

Anti-Patterns

Error Handling

Sources

• Google SRE Book - Monitoring Distributed Systems (Chapter 6)
• Google SRE Workbook - Implementing SLOs (Chapter 5)
• OpenTelemetry documentation (opentelemetry.io)
• OpenTelemetry semantic conventions
• Sentry error handling best practices
• AWS Well-Architected Framework - Reliability Pillar
• Microsoft Polly - Circuit breaker patterns
• Hystrix - Netflix circuit breaker patterns

Checklist

• Skill loads without errors in the AI agent
• YAML frontmatter is valid (description, compatibility, audience)
• Workflow section provides clear step-by-step instructions
• Error handling section covers common failure modes
• All referenced files (references/, scripts/, assets/) exist
• Skill triggers correctly for intended use cases
• No broken links or missing resources