Observability

Observability Agent

Standardises logging, metrics, tracing, and health checks across BC Gov projects.

Logging — Serilog

Package install

dotnet add package Serilog.AspNetCore
dotnet add package Serilog.Formatting.Compact
dotnet add package Serilog.Sinks.Console

Program.cs bootstrap

// LOGGING — structured JSON to stdout (picked up by OpenShift log aggregation)
Log.Logger = new LoggerConfiguration()
    .ReadFrom.Configuration(builder.Configuration)
    .Enrich.FromLogContext()
    .Enrich.WithMachineName()
    .Enrich.WithEnvironmentName()
    .WriteTo.Console(new CompactJsonFormatter())
    .CreateLogger();

builder.Host.UseSerilog();

appsettings.json log levels

{
  "Serilog": {
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "Microsoft.EntityFrameworkCore.Database.Command": "Warning",
        "System": "Warning"
      }
    }
  }
}

Log level Warning for EF Core SQL commands — never log SQL at Information in production (may expose query parameters containing PII).

Log level standards

PII-free logging rules

NEVER log:

• User names, email addresses
• SIN, DL number, student number, or other personal identifiers
• Connection strings, tokens, passwords, or Vault values
• Full HTTP request/response bodies (may contain form data)
• Query string parameters that may carry tokens (?code=, ?token=)

Safe to log:

• User ID (GUID / sub claim) — opaque identifier only
• HTTP method + path (no query string)
• HTTP status code
• Duration (ms)
• Correlation ID / trace ID

Structured logging pattern

// Use structured properties, not string interpolation
_logger.LogInformation("Created work item {WorkItemId} for project {ProjectId}",
    item.Id, item.ProjectId);

// NOT:
_logger.LogInformation($"Created work item {item.Id}");

Health Checks

Package

dotnet add package AspNetCore.HealthChecks.MySql

Registration in Program.cs

// HEALTH CHECKS — /api/health (liveness) and /api/health/details (readiness)
builder.Services.AddHealthChecks()
    .AddMySql(
        connectionString: builder.Configuration.GetConnectionString("DefaultConnection")!,
        name: "database",
        tags: ["ready"]);

// ...
app.MapHealthChecks("/api/health");
app.MapHealthChecks("/api/health/details", new HealthCheckOptions
{
    ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse,
    Predicate      = _ => true,
});

Containerfile health check

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
  CMD wget -qO- http://localhost:8080/api/health || exit 1

Metrics — Prometheus

Emerald uses a Prometheus pull model. Annotate pods so Prometheus discovers the metrics endpoint.

Pod annotations (Helm values.yaml)

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/path:   "/metrics"
  prometheus.io/port:   "8080"

.NET metrics endpoint (ASP.NET Core)

dotnet add package prometheus-net.AspNetCore

// METRICS — expose /metrics for Prometheus scrape
app.UseHttpMetrics();  // request duration, count, in-flight
app.MapMetrics();      // GET /metrics

Custom metrics example

// create once at class level (static) — Prometheus counters are global
private static readonly Counter _itemsCreated =
    Metrics.CreateCounter("app_items_created_total", "Number of work items created.",
        labelNames: ["project_id"]);

// in service method — increment counter when an item is created
_itemsCreated.WithLabels(projectId.ToString()).Inc();

Tracing — OpenTelemetry

Packages

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Exporter.Console   # dev only

Registration

// TRACING — OpenTelemetry with OTLP export (or console in dev)
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing
            .SetResourceBuilder(ResourceBuilder.CreateDefault()
                .AddService(serviceName: "my-app-api"))
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddEntityFrameworkCoreInstrumentation();

        if (builder.Environment.IsDevelopment())
            tracing.AddConsoleExporter();
        else
            tracing.AddOtlpExporter(opt =>
                opt.Endpoint = new Uri(builder.Configuration["Otlp:Endpoint"]!));
    });

Alert Baseline (document in securityNextSteps.md)

PII-in-Logs Validation Checklist

Systems handling personal information (Protected B) must not write PII to logs. FOIPPA and the Privacy Act treat PII in application logs as a disclosure risk — logs are often retained longer than business records and may be accessed by ops staff without data steward approval.

Common PII Categories to Check

• Full names, addresses, dates of birth
• SIN, health card numbers, driver's licence numbers
• BCeID / IDIR usernames (these are identifiers linked to real people)
• Email addresses
• IP addresses (context-dependent — usually PII for Protected B systems)
• FOI request content (subject matter, request description, requester details)
• Other program-area-specific sensitive fields (configmap keys, request payloads, document content)

Python Checklist

• No logging.debug(request.json()) or logging.info(vars(model)) — these dump full objects
• No f"Processing request for {user.name}" — use opaque user ID only (e.g. user.id)
• No full request body logging at INFO or DEBUG level
• Flask: never log request.data or request.form at INFO
• SQLAlchemy: confirm LOG_SQLALCHEMY=ERROR (not DEBUG — DEBUG logs full queries with bound parameter values)
• Structured log format (JSON output) confirmed — verify with LOG_FORMAT=json or equivalent env var

Go Checklist

• No log.Printf("%+v", document) on full document structs — %+v dumps all fields including content
• Use structured logging (slog or zap) with an explicit field allowlist
• Never log document content (raw bytes, extracted text, OCR output)
• Confirm log output is JSON-formatted (structured) not plain text

Node.js Checklist

• No console.log(req.body) in middleware or request handlers
• No JSON.stringify(user) in log statements
• Morgan (HTTP logger): use :method :url :status :res[content-length] - :response-time ms format only — no body logging
• No winston.debug(JSON.stringify(requestPayload)) patterns

Audit Command — Scan for PII-Risk Log Statements

# Python — flag logging calls that reference request bodies, user objects, or email/name fields
grep -rn "logging\.\(info\|debug\|warning\)(.*\(request\|body\|user\|email\|name\))" \
  --include="*.py" .

# Go — flag log calls that reference document, user, email, or body variables
grep -rn "log\.\(Print\|Info\|Debug\)(.*\(document\|user\|email\|body\))" \
  --include="*.go" .

# Node.js — flag console.log or logger calls referencing req.body or req.user
grep -rn "console\.log\|logger\.\(info\|debug\)(.*req\.\(body\|user\))" \
  --include="*.js" --include="*.ts" .

Review all matches manually — not all are violations, but each needs a justification.

OBSERVABILITY_KNOWLEDGE

confirmed_facts:
  - "CompactJsonFormatter writes single-line JSON to stdout — required for OpenShift log aggregation"
  - "Prometheus on Emerald uses pod annotations to discover /metrics endpoints"
  - "EF Core SQL logging at Information level may expose query parameters — use Warning"
  - "OpenTelemetry AddEntityFrameworkCoreInstrumentation requires EFCore instrumentation package"
  - "Health check at /api/health is used by OpenShift liveness probe; /api/health/details for readiness"
  - "2026-06-05: [multi-app-engagement] Python services using LOG_SQLALCHEMY=ERROR is correct but structured log format must be confirmed — validate JSON output in all Python/Go services"
common_pitfalls:
  - "Never log PII — user emails, names, or identifying numbers in structured properties"
  - "Metrics.CreateCounter must be static — creating per-request instances causes memory leaks"
  - "prometheus-net endpoint conflicts with ASP.NET Core minimal API route if MapMetrics() called after MapControllers()"