Golang Observability

Go Observability

Create telemetry that answers a defined operational question without leaking data or destabilizing the system. Start from the service's existing conventions and backend.

Frame the question

Inspect current logging, metrics, tracing, profiling, middleware, dashboards, alerts, privacy rules, and deployment topology. Identify:

• the user-visible behavior or failure being investigated;
• service boundaries and asynchronous hops;
• existing service-level indicators and objectives;
• the telemetry backend's naming, cardinality, retention, and sampling constraints;
• data classified as secrets, personal information, or regulated content.

Do not introduce OpenTelemetry, Prometheus, a vendor agent, or another backend merely to follow a default architecture.

Instrument stable boundaries

Prefer instrumentation at request, job, queue, database, and external-client boundaries over logs at every function.

Logs

• Use the project's structured logger and stable field names.
• Include correlation identifiers when already available, but do not treat a request ID as authentication.
• Record an error once at the layer that can add useful outcome context; avoid repeated stack traces at every return.
• Keep secrets, credentials, authorization headers, session data, raw request bodies, and sensitive identifiers out of logs.
• Avoid attacker-controlled text in unstructured log lines; structured fields reduce log-forging ambiguity.

Choose levels by operator action, not by code location. Expected invalid input is not automatically an error-level incident.

Metrics

• Name measurements by the behavior they represent and define units.
• Keep labels bounded. User IDs, request IDs, URLs, raw errors, SQL, and unbounded tenant or object identifiers are unsafe label values.
• Count outcomes and measure latency or size at meaningful boundaries.
• Choose histogram buckets from expected ranges and observed data rather than copying universal defaults.
• Avoid duplicate metric registration and ensure concurrent callbacks are safe.

For request-driven services, rate, errors, and duration are often a useful starting model. Resource-oriented components may need utilization, saturation, and error signals. Apply these as heuristics, not mandatory dashboards.

Traces

Propagate context across supported process and messaging boundaries. Create spans around meaningful remote or asynchronous operations, not every helper. Record bounded attributes and useful status while excluding payloads and secrets.

Sampling is an operational policy. Preserve upstream sampling decisions and the existing backend configuration unless the task explicitly changes them.

Profiling and runtime diagnostics

Use runtime metrics, execution traces, or profiles for a specific hypothesis. Profiling endpoints and debug handlers can expose topology, arguments, memory contents, or credentials:

• bind them to an internal interface or separate authenticated listener;
• keep them out of public routing by default;
• restrict access and retention;
• collect for a bounded period and account for overhead.

Do not enable continuous profiling or runtime knobs in production without the user's operational scope.

Dashboards and alerts

A dashboard should connect traffic or workload, outcomes, latency, and saturation to a service boundary. Avoid panels that duplicate data without supporting a decision.

Alert on user-impacting symptoms or a well-understood imminent failure. Each alert needs:

• a defined owner and response;
• a duration that filters expected noise;
• a link to relevant context or runbook;
• validation against historical or staged data;
• no dependency on labels too granular to aggregate reliably.

Do not invent an SLO or paging threshold. If none exists, present the decision and available baseline evidence.

Verification

Test instrumentation without coupling behavior tests to a particular exporter where possible. Verify:

• success, failure, cancellation, and retry outcomes;
• context propagation across boundaries;
• cardinality under representative inputs;
• redaction of sensitive data;
• cleanup and shutdown flushing with bounded deadlines;
• disabled or unavailable backends do not break core behavior unless telemetry is explicitly required.

Use existing local collectors or test exporters. Do not send test telemetry to production services or create external dashboards and alerts without authorization.