Cross Platform Structured Logging

Cross-Platform Structured Logging

Establish unified, machine-readable JSON logging across all platforms for centralized aggregation, powerful querying, and correlation with distributed traces.

Why Structured Logging

Problem with traditional logging:

console.log('User ' + userId + ' logged in from ' + ip)

• Not machine-readable
• Hard to query
• No type safety
• Can't aggregate metrics

Structured logging solution:

logger.info('User logged in', {
  user_id: userId,
  ip_address: ip,
  auth_method: 'oauth'
})

Output (JSON):

{
  "level": "info",
  "timestamp": "2024-01-15T10:30:00.000Z",
  "message": "User logged in",
  "user_id": "user_123",
  "ip_address": "192.168.1.1",
  "auth_method": "oauth",
  "trace_id": "abc123...",
  "service": "api"
}

Benefits:

• Query: user_id:user_123 AND level:error
• Aggregate: Count logins by auth_method
• Correlate: Find all logs for trace_id

Unified Log Schema

Use consistent field names across all platforms:

Standard Fields

Context Fields (as needed)

{
  "http": {
    "method": "POST",
    "url": "/api/users",
    "status_code": 201,
    "user_agent": "Mozilla/5.0..."
  },
  "error": {
    "type": "ValidationError",
    "message": "Invalid email",
    "stack": "Error: Invalid email\n  at..."
  }
}

Node.js: Pino

Pino is the fastest JSON logger for Node.js.

Basic Setup

// lib/logger.ts
import pino from 'pino'

const logger = pino({
  level: process.env.LOG_LEVEL || 'info',

  // Pretty print in development
  transport: process.env.NODE_ENV === 'development'
    ? { target: 'pino-pretty', options: { colorize: true } }
    : undefined,

  // Base fields included in every log
  base: {
    service: {
      name: process.env.SERVICE_NAME || 'api',
      version: process.env.SERVICE_VERSION || '1.0.0'
    },
    environment: process.env.NODE_ENV || 'development'
  },

  // Redact sensitive fields
  redact: {
    paths: ['password', 'api_key', 'credit_card', '*.password', '*.token'],
    censor: '[REDACTED]'
  }
})

export default logger

Usage

import logger from '@/lib/logger'

// Simple message
logger.info('Server started')

// With context
logger.info({ user_id: '123', action: 'login' }, 'User logged in')

// Child logger (adds context to all subsequent logs)
const requestLogger = logger.child({ request_id: req.id })
requestLogger.info('Processing request')
requestLogger.error({ err }, 'Request failed')

Middleware Integration (Next.js API Routes)

// lib/logger-middleware.ts
import { NextRequest, NextResponse } from 'next/server'
import logger from '@/lib/logger'

export function withLogging(
  handler: (req: NextRequest) => Promise<NextResponse>
) {
  return async (req: NextRequest) => {
    const requestLogger = logger.child({
      request_id: crypto.randomUUID(),
      http: {
        method: req.method,
        url: req.url
      }
    })

    requestLogger.info('Request started')

    try {
      const response = await handler(req)
      requestLogger.info({ status: response.status }, 'Request completed')
      return response
    } catch (error) {
      requestLogger.error({ err: error }, 'Request failed')
      throw error
    }
  }
}

// Usage in API route
export const GET = withLogging(async (req) => {
  // Your handler logic
})

Production Configuration (Async Transport)

// lib/logger.ts (production)
import pino from 'pino'

const logger = pino({
  level: 'info',

  // Don't use pino-pretty in production (performance)
  transport: process.env.NODE_ENV === 'production'
    ? {
        target: 'pino/file',
        options: { destination: 1 } // stdout
      }
    : { target: 'pino-pretty' },

  // Async logging (don't block event loop)
  destination: pino.destination({
    sync: false // Async mode
  })
})

export default logger

Python: structlog

structlog integrates with Python's standard logging library.

Basic Setup

# logging_config.py
import structlog
import logging
import sys

def configure_logging():
    # Configure standard library logging
    logging.basicConfig(
        format="%(message)s",
        stream=sys.stdout,
        level=logging.INFO,
    )

    # Configure structlog
    structlog.configure(
        processors=[
            # Add log level
            structlog.stdlib.add_log_level,

            # Add timestamp
            structlog.processors.TimeStamper(fmt="iso"),

            # Add calling location (file:line)
            structlog.processors.CallsiteParameterAdder(
                parameters=[
                    structlog.processors.CallsiteParameter.FILENAME,
                    structlog.processors.CallsiteParameter.LINENO,
                ]
            ),

            # Format exceptions
            structlog.processors.format_exc_info,

            # Render as JSON
            structlog.processors.JSONRenderer()
        ],
        wrapper_class=structlog.stdlib.BoundLogger,
        context_class=dict,
        logger_factory=structlog.stdlib.LoggerFactory(),
        cache_logger_on_first_use=True,
    )

# Call at app startup
configure_logging()

Usage

import structlog

logger = structlog.get_logger()

# Simple message
logger.info("server_started", port=8000)

# With context
logger.info(
    "user_logged_in",
    user_id="user_123",
    auth_method="oauth"
)

# Error logging
try:
    risky_operation()
except Exception as e:
    logger.error("operation_failed", exc_info=e)

FastAPI Integration

# main.py
import structlog
from fastapi import FastAPI, Request
from uuid import uuid4

app = FastAPI()

@app.middleware("http")
async def logging_middleware(request: Request, call_next):
    request_id = str(uuid4())

    # Bind context for this request
    logger = structlog.get_logger().bind(
        request_id=request_id,
        method=request.method,
        path=request.url.path
    )

    logger.info("request_started")

    try:
        response = await call_next(request)
        logger.info("request_completed", status_code=response.status_code)
        return response
    except Exception as e:
        logger.error("request_failed", exc_info=e)
        raise

Adding Trace Context

# Integrate with OpenTelemetry
from opentelemetry import trace

logger = structlog.get_logger()

# Get current span
span = trace.get_current_span()
context = span.get_span_context()

# Log with trace context
logger.info(
    "database_query",
    trace_id=format(context.trace_id, '032x'),
    span_id=format(context.span_id, '016x'),
    query="SELECT * FROM users"
)

Firebase Functions: Native Logger

Firebase provides a built-in logger optimized for Cloud Logging.

Setup

// src/utils/logger.ts
import { logger } from 'firebase-functions'

export function log(message: string, data?: Record<string, any>) {
  logger.info(message, {
    ...data,
    service: 'firebase-functions',
    environment: process.env.GCLOUD_PROJECT
  })
}

export function logError(message: string, error: Error, data?: Record<string, any>) {
  logger.error(message, {
    ...data,
    error: {
      type: error.name,
      message: error.message,
      stack: error.stack
    }
  })
}

Usage

import * as functions from 'firebase-functions'
import { log, logError } from './utils/logger'

export const myFunction = functions.https.onCall(async (data, context) => {
  log('Function called', {
    user_id: context.auth?.uid,
    data
  })

  try {
    const result = await processData(data)
    log('Function completed', { result })
    return result
  } catch (error) {
    logError('Function failed', error as Error, { data })
    throw error
  }
})

Automatic Execution ID (2nd Gen Functions)

Cloud Run-based 2nd gen functions automatically inject execution_id:

import { onRequest } from 'firebase-functions/v2/https'
import { logger } from 'firebase-functions'

export const myHttpFunction = onRequest(async (req, res) => {
  // execution_id automatically added by Cloud Run
  logger.info('Processing request', {
    path: req.path,
    method: req.method
  })
  // Output includes: execution_id, trace, span_id
})

Security: PII Redaction

Never log sensitive data:

Pino Redaction

const logger = pino({
  redact: {
    paths: [
      'password',
      'api_key',
      'credit_card',
      'ssn',
      '*.password',
      '*.token',
      'req.headers.authorization'
    ],
    censor: '[REDACTED]'
  }
})

logger.info({ password: 'secret123' }, 'User created')
// Output: { "password": "[REDACTED]", "message": "User created" }

structlog Redaction

def redact_sensitive_data(logger, method_name, event_dict):
    """Processor to redact sensitive fields"""
    sensitive_fields = ['password', 'api_key', 'credit_card', 'ssn']

    for field in sensitive_fields:
        if field in event_dict:
            event_dict[field] = '[REDACTED]'

    return event_dict

structlog.configure(
    processors=[
        redact_sensitive_data,  # Add as first processor
        # ... other processors
    ]
)

Anti-Patterns

String Interpolation

// BAD
logger.info(`User ${userId} logged in from ${ip}`)

// GOOD
logger.info('User logged in', { user_id: userId, ip_address: ip })

console.log in Production

// BAD
console.log('User logged in', userId)

// GOOD
logger.info('User logged in', { user_id: userId })

Logging Sensitive Data

// BAD
logger.info('Login attempt', { password: req.body.password })

// GOOD
logger.info('Login attempt', { email: req.body.email })

Synchronous Logging (Node.js Production)

// BAD (blocks event loop)
const logger = pino({ sync: true })

// GOOD (async)
const logger = pino({ sync: false })

Log Levels

Use appropriate levels:

Best Practices

1. Use structured data, not string interpolation
2. Redact PII and secrets automatically
3. Include trace_id in every log for correlation
4. Use child loggers for request-scoped context
5. Set appropriate log levels per environment
6. Use async transports in production (Node.js)
7. Include service.name for multi-service aggregation

Verification

Test structured logging:

# Start service and make request
curl http://localhost:3000/api/test

# Check logs (should be JSON)
cat logs.json | jq '.'

# Verify fields present
cat logs.json | jq 'select(.trace_id != null)'

Query in Google Cloud Logging:

resource.type="cloud_run_revision"
jsonPayload.user_id="user_123"
jsonPayload.trace_id="abc123..."