Consul Knowledge Patch

HashiCorp Consul

Use this skill when designing, configuring, upgrading, or troubleshooting Consul deployments whose behavior depends on recent discovery, service-mesh, gateway, Kubernetes, security, Enterprise, or operational changes.

Treat the reference files as task-oriented supplements. Inspect the deployment's actual Consul and Envoy versions, edition, platform, topology, and configuration before applying guidance. Release-specific behavior does not replace an existing tested constraint.

Reference index

Start with compatibility and security changes

Validate KV names before rollout

The KV endpoint rejects invalid key names by default. Audit writers and stored keys before upgrading. Use DisableKVKeyValidation only as a deliberate compatibility escape hatch, with a plan to remove invalid names.

Keep HTTP paths normalized before L7 authorization

API Gateway and terminating-gateway HTTP listeners normalize paths before intention RBAC checks. Custom envoy_public_listener_json HTTP Connection Manager filter chains receive the same normalization. Do not set InsecureDisablePathNormalization unless the bypass risk is explicitly accepted.

Account for bounded agent requests

Check updates, check and service registration, and Connect authorization requests are limited to 512 KiB before decoding or ACL resolution. This includes chunked requests. Split oversized registrations and treat HTTP 413 as a size failure, not an ACL failure.

Bound external gRPC clients

External gRPC and gRPC-TLS listeners default to 100 connections per source IP, and their handshake timeout is 20 seconds. Tune limits.grpc_max_conns_per_client only after checking client fan-out and shared-NAT behavior.

limits {
  grpc_max_conns_per_client = 100
}

Match Consul and Envoy deliberately

The bundled Envoy line changed to 1.35.3 while 1.31.10 support was removed, and Consul 2.0 service mesh expects Envoy 1.37.2 or newer. Verify separately installed or pinned Envoy binaries before rolling agents. For Envoy 1.35+, generated configuration adds a TLS transport socket only when a CA bundle exists.

Migrate OpenShift gateway resources

OpenShift 4.19+ requires the newer resource types in the consul.hashicorp.com API group. Inventory and migrate older Kubernetes Gateway API v1alpha resources as part of the OpenShift upgrade.

High-value configuration changes

Preserve blocking queries with agent HTTP defaults

Agent http_config.read_timeout and write_timeout now default to 15 minutes. read_header_timeout remains 10 seconds and idle_timeout remains 120 seconds. Set all four explicitly when load balancers, proxies, or clients need a different timeout contract.

Register and route named ports

Use service ports for multi-port catalog registration. Kubernetes Service sync understands multi-port Services, and DNS can select a port with its port field.

Enterprise sidecars advertise named local ports with proxy.local_service_ports; upstreams choose one with proxy.upstreams[].destination_port. Direct-mode callers still use localhost:<bind-port>, while transparent-proxy callers use <port-name>.<service>.virtual.consul.

Change Enterprise RPC limits without restarts

Use the Raft-replicated rate-limit configuration entry for cluster-wide RPC limits and critical-method exemptions. Discover targetable method names with:

curl -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
  https://consul.example/v1/internal/rpc/methods

The token needs operator:read.

Rotate gateway certificates through SDS

API Gateway listeners can use a default SDS certificate. HTTP or TCP route services may override it and otherwise inherit the listener's SDS cluster; conflicting override mappings are rejected. Terminating-gateway upstream TLS also uses SDS, so certificates can change without restarting the gateway.

Tune gateway upstream pressure

Apply gateway-wide defaults or route-service overrides for MaxConnections, MaxPendingRequests, and MaxConcurrentRequests. Keep overrides consistent with the capacity of each destination.

Configure passive failure detection

Enterprise PassiveHealthCheck supports Consecutive5xx, ConsecutiveGatewayFailure, and EnforcingConsecutiveGatewayFailure. Gateway failures cover HTTP 502, 503, and 504; choose thresholds that distinguish a bad upstream from brief transients.

Safe upgrade workflow

1. Confirm the source and target Consul versions support a direct upgrade. Routine upgrades should cross no more than two major jumps; Enterprise LTS operators may cross at most three.
2. Confirm that the installed Envoy version is supported by both old and new Consul versions. Centralize sidecar and mesh-gateway configuration before WAN-federated mesh upgrades.
3. Snapshot state, verify quorum, and identify the Raft leader with consul operator raft list-peers.
4. Upgrade primary-datacenter server followers one at a time, then the leader, waiting for each server to rejoin and become healthy. Upgrade clients afterward.
5. Repeat servers then clients for each secondary datacenter. Preserve redundant service instances because a client is undiscoverable between consul leave and restart.
6. If the release requires an incompatible protocol transition, run the new binary with -protocol=PREVIOUS everywhere first, then restart all agents without the override.
7. Restore non-persisted agent and default ACL tokens when needed. Verify LAN and WAN membership, agent builds, and protocol versions.
8. Query /v1/acl/replication from a secondary datacenter after the rollout; the primary reports replication as disabled even when replication works.

See Upgrades and rollouts for exact Enterprise license order, Envoy stop/start sequencing, commands, and Autopilot replacement behavior.

Platform decisions

Choose one address family per datacenter

Agents and services on VMs and Kubernetes can use IPv4 or IPv6, but one family per datacenter is recommended. IPv6 is unavailable on OpenShift, Nomad, and ECS for the relevant behavior. Envoy uses 127.0.0.1 in IPv4-only environments and ::1 in IPv6 or dual-stack environments; IPv6 agent binds also default local upstream and proxy service addresses to ::1.

Apply namespace pod-security controls

Use Kubernetes Pod Security Admission per namespace instead of PodSecurityPolicy. Validate Consul components against the namespace policy before enforcing it.

Select snapshot authentication by platform

The Enterprise Kubernetes snapshot-agent sidecar can store snapshots locally, in Amazon S3, Azure Blob Storage, or Google Cloud Storage. Azure Blob can use Azure Managed Service Identity instead of static credentials.

Scale and migrate gateways consciously

Enterprise Kubernetes API Gateways can exceed eight replicas and use Horizontal Pod Autoscaling enabled with Gateway annotations. On OpenShift upgrades, coordinate scaling changes with the required resource migration.

Identity and certificate choices

PKCE is enabled by default for UI OIDC login. Providers may authenticate the OIDC client with a JWT assertion rather than a client secret; align the provider registration with the selected method.

Enterprise service-mesh certificate signing can be delegated to CyberArk Workload Identity Manager (Venafi Firefly) with:

connect {
  ca_provider = "pan-distributed-issuer"
}

Use the Connect CA API NotAfter values plus /agent/metrics certificate metrics to monitor roots, intermediates, signing CAs, agent certificates, and leaf renewal. Structured certificate-expiration logs support configurable severity thresholds.

Observability and reporting

Self-managed Enterprise product-usage telemetry is opt-in and disabled by default. Keep the choice explicit in configuration and change review.

Census metrics collection is always enabled, while export for license reporting is configurable. For manual or air-gapped reporting, use /v1/operator/utilization or generate a census bundle:

consul operator utilization [-today-only] [-message] [-y]

Certificate metrics carry datacenter, partition, and namespace labels. Design alerts with those dimensions so the affected trust domain is identifiable.

Troubleshooting checklist

• HTTP 413 during registration or authorization: measure the encoded body and split it below 512 KiB.
• Long-poll query ends too early: inspect all four agent HTTP timeout values and every intermediary timeout.
• External gRPC clients cannot connect: check per-source-IP counts, NAT aggregation, and the 20-second handshake window.
• Envoy fails after an agent upgrade: compare the Envoy version with both Consul versions and verify whether a CA bundle exists.
• L7 intentions behave unexpectedly: confirm path normalization applies to standard and custom public listeners.
• A route receives the wrong certificate: check listener defaults, route overrides, SDS-cluster inheritance, and conflicting mappings.
• Multi-port traffic reaches the wrong endpoint: align catalog port names, local sidecar ports, upstream destination_port, and the application dial mode.
• ACL replication appears disabled: make the check against a secondary datacenter, not the primary.
• A restarted server cannot rejoin: restore non-persisted agent and default tokens.
• Gateway responses expose an unwanted server header: use envoy_suppress_envoy_headers, or rename it with envoy_server_header_name; suppression wins when both are set.

Detailed lookup

Read only the reference relevant to the task, then validate against the live configuration:

• Service registration, DNS, IP families, Envoy, or outlier detection: Discovery, networking, and mesh.
• Support policy, license reporting, runtime limits, or certificate monitoring: Enterprise operations and telemetry.
• Authentication, gateway TLS, L7 enforcement, gRPC, or proxy headers: Gateways, security, and identity.
• Kubernetes security, snapshots, OpenShift, or gateway controllers: Kubernetes and snapshots.
• Rolling, federated, automated, license, or protocol upgrades: Upgrades and rollouts.