Opentelemetry Instrumentation

OpenTelemetry Instrumentation for Node.js

This skill guides you through instrumenting Node.js/TypeScript applications with OpenTelemetry spans, following best practices for distributed tracing.

When to Instrument

✅ DO Instrument:

• External service calls (HTTP, gRPC)
• Database operations (queries, transactions)
• Critical business operations (order processing, payment flows)
• Long-running operations (file processing, batch jobs)
• Integration points (third-party APIs, message queues)

❌ DON'T Instrument:

• Simple property getters/setters
• In-memory operations (object lookups, array operations)
• Trivial calculations
• Operations that complete in <1ms consistently

Installation

npm install @opentelemetry/api @opentelemetry/sdk-trace-base @opentelemetry/sdk-trace-node
npm install @opentelemetry/instrumentation-http @opentelemetry/instrumentation-express
npm install @opentelemetry/exporter-otlp-http

Core Concepts

Tracer and Span

OpenTelemetry uses Tracer and Span for creating traces:

import { trace } from '@opentelemetry/api';

const tracer = trace.getTracer('MyService');

// Create a span
const span = tracer.startSpan('operation.name');
try {
  // Your code here
  span.setAttributes({
    'key': 'value',
  });
  span.setStatus({ code: SpanStatusCode.OK });
} catch (error) {
  span.setStatus({ 
    code: SpanStatusCode.ERROR, 
    message: error.message 
  });
  span.recordException(error);
  throw error;
} finally {
  span.end();
}

Span Kinds

import { SpanKind } from '@opentelemetry/api';

SpanKind.SERVER    // Incoming HTTP request
SpanKind.CLIENT    // Outgoing HTTP/gRPC call
SpanKind.INTERNAL  // Internal operation
SpanKind.PRODUCER  // Message queue producer
SpanKind.CONSUMER  // Message queue consumer

Instrumentation Patterns

Pattern 1: HTTP Server (Express)

Automatic instrumentation:

import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-otlp-http';

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318/v1/traces',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

Manual instrumentation:

import { trace, SpanKind, SpanStatusCode } from '@opentelemetry/api';
import express, { Request, Response, NextFunction } from 'express';

const tracer = trace.getTracer('MyService');

export function tracingMiddleware(
  req: Request,
  res: Response,
  next: NextFunction
): void {
  const span = tracer.startSpan(`HTTP ${req.method} ${req.path}`, {
    kind: SpanKind.SERVER,
    attributes: {
      'http.method': req.method,
      'http.url': req.url,
      'http.route': req.route?.path || req.path,
    },
  });

  res.on('finish', () => {
    span.setAttributes({
      'http.status_code': res.statusCode,
    });
    
    if (res.statusCode >= 400) {
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: `HTTP ${res.statusCode}`,
      });
    } else {
      span.setStatus({ code: SpanStatusCode.OK });
    }
    
    span.end();
  });

  trace.setSpan(trace.active(), span);
  next();
}

Pattern 2: HTTP Client Calls

import { trace, SpanKind, SpanStatusCode } from '@opentelemetry/api';
import axios, { AxiosRequestConfig } from 'axios';

const tracer = trace.getTracer('MyService');

export async function callExternalApi(
  url: string,
  config?: AxiosRequestConfig
) {
  const span = tracer.startSpan('http.client.call', {
    kind: SpanKind.CLIENT,
    attributes: {
      'http.method': config?.method?.toUpperCase() || 'GET',
      'http.url': url,
    },
  });

  try {
    const response = await axios(url, config);
    
    span.setAttributes({
      'http.status_code': response.status,
    });
    span.setStatus({ code: SpanStatusCode.OK });
    
    return response;
  } catch (error: any) {
    span.recordException(error);
    span.setStatus({
      code: SpanStatusCode.ERROR,
      message: error.message,
    });
    throw error;
  } finally {
    span.end();
  }
}

Pattern 3: Database Operations

import { trace, SpanKind, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('MyService');

export async function queryDatabase<T>(
  query: string,
  params?: any[]
): Promise<T[]> {
  const span = tracer.startSpan('db.query', {
    kind: SpanKind.INTERNAL,
    attributes: {
      'db.system': 'postgresql',
      'db.name': databaseName,
      'db.operation': 'SELECT',
      // Note: Sanitize db.statement if contains sensitive data
    },
  });

  try {
    const result = await db.query(query, params);
    
    span.setAttributes({
      'db.rows_affected': result.rowCount || 0,
    });
    span.setStatus({ code: SpanStatusCode.OK });
    
    return result.rows;
  } catch (error: any) {
    span.recordException(error);
    span.setStatus({
      code: SpanStatusCode.ERROR,
      message: error.message,
    });
    throw error;
  } finally {
    span.end();
  }
}

Pattern 4: Business Operations

export async function processOrder(orderId: string, amount: number) {
  const span = tracer.startSpan('order.process', {
    kind: SpanKind.INTERNAL,
    attributes: {
      'order.id': orderId,
      'order.amount': amount,
    },
  });

  try {
    // Child spans for sub-operations
    const paymentSpan = tracer.startSpan('payment.process', {
      kind: SpanKind.INTERNAL,
    });
    
    try {
      await processPayment(orderId, amount);
      paymentSpan.setStatus({ code: SpanStatusCode.OK });
    } catch (error: any) {
      paymentSpan.recordException(error);
      paymentSpan.setStatus({
        code: SpanStatusCode.ERROR,
        message: error.message,
      });
      throw error;
    } finally {
      paymentSpan.end();
    }
    
    span.setStatus({ code: SpanStatusCode.OK });
    return { success: true };
  } catch (error: any) {
    span.recordException(error);
    span.setStatus({
      code: SpanStatusCode.ERROR,
      message: error.message,
    });
    throw error;
  } finally {
    span.end();
  }
}

Configuration

Bootstrap TracerProvider

import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-otlp-http';
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';

const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: process.env.SERVICE_NAME || 'MyService',
  }),
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318/v1/traces',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

// Start SDK
sdk.start();

// Graceful shutdown
process.on('SIGTERM', () => {
  sdk.shutdown()
    .then(() => console.log('Tracing terminated'))
    .catch((error) => console.error('Error terminating tracing', error))
    .finally(() => process.exit(0));
});

Environment Variables

# Enable/disable OpenTelemetry
OTEL_ENABLED=true

# OTLP endpoint
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces

# Service name
SERVICE_NAME=MyService

# Sampling configuration
OTEL_TRACES_SAMPLER=traceidratio
OTEL_TRACES_SAMPLER_ARG=1.0  # 1.0 = 100%, 0.1 = 10%

Semantic Tags (Best Practices)

Follow OpenTelemetry semantic conventions:

HTTP: http.method, http.url, http.status_code, http.route, net.peer.name

Database: db.system, db.name, db.operation, db.statement (sanitized!), db.rows_affected

RPC: rpc.system, rpc.service, rpc.method, rpc.status_code

Custom Business: order.id, user.id, payment.amount, etc.

Never include: Passwords, tokens, API keys, credit cards, full request/response bodies

Helper Pattern (Recommended)

Create a centralized helper for consistent instrumentation:

import { trace, SpanKind, SpanStatusCode, Span } from '@opentelemetry/api';

export class TraceabilityHelper {
  private static tracer = trace.getTracer('MyService');

  static startSpan(
    name: string,
    kind: SpanKind = SpanKind.INTERNAL
  ): Span | undefined {
    if (!this.isOtelEnabled()) return undefined;
    return this.tracer.startSpan(name, { kind });
  }

  static setHttpTags(
    span: Span | undefined,
    method?: string,
    url?: string,
    statusCode?: number
  ): void {
    if (!span) return;
    if (method) span.setAttribute('http.method', method);
    if (url) span.setAttribute('http.url', url);
    if (statusCode !== undefined) {
      span.setAttribute('http.status_code', statusCode);
    }
  }

  static setDbTags(
    span: Span | undefined,
    system: string,
    name: string,
    operation: string
  ): void {
    if (!span) return;
    span.setAttribute('db.system', system);
    span.setAttribute('db.name', name);
    span.setAttribute('db.operation', operation);
  }

  static setSuccessStatus(span: Span | undefined): void {
    span?.setStatus({ code: SpanStatusCode.OK });
  }

  static setErrorStatus(
    span: Span | undefined,
    errorMessage?: string
  ): void {
    span?.setStatus({
      code: SpanStatusCode.ERROR,
      message: errorMessage,
    });
  }

  static recordException(span: Span | undefined, error: Error): void {
    if (!span || !error) return;
    span.recordException(error);
  }

  private static isOtelEnabled(): boolean {
    const env = process.env.OTEL_ENABLED;
    return env === 'true' || env === '1';
  }
}

Best Practices

1. Feature Flag: Use OTEL_ENABLED to disable without code changes
2. Never Crash: Wrap initialization in try/catch - misconfiguration shouldn't bring down the app
3. Null Checks: Always check if span exists (may be null if disabled or sampled out)
4. Semantic Conventions: Use standard attribute names (http.*, db.*, rpc.*, error.*)
5. Sensitive Data: Never include passwords, tokens, credit cards, full payloads
6. Span Hierarchy: Create logical parent-child relationships (Server → Internal → Client)
7. Error Handling: Always record exceptions and set error status
8. Sampling: Dev 100% (SAMPLER_ARG=1.0), Production 10% or less (0.1 or 0.01)

Common Issues

Spans Not Appearing:

• Check OTEL_ENABLED=true, OTLP endpoint reachable, sampling rate, tracer registered

Memory Leaks:

• Ensure spans are properly ended in finally blocks
• Use automatic instrumentation when possible

Performance Impact:

• Use sampling in production (10% or less)
• Avoid instrumenting high-frequency operations

Key Principles

1. Instrument at boundaries: External calls, database operations, critical business flows
2. Use semantic conventions: Standard attribute names ensure compatibility with observability tools
3. Fail gracefully: Misconfiguration should never crash the application
4. Respect sampling: Don't instrument operations that complete in <1ms
5. Protect sensitive data: Never log passwords, tokens, or full payloads
6. Create logical hierarchies: Parent-child spans show operation flow clearly