Consul

HashiCorp Consul AI Skill Guide

Overview

Consul provides service discovery, health checking, a hierarchical KV store, and (with Consul Service Mesh) mTLS intentions between services. Agents register services; clients resolve healthy instances via DNS (service.consul) or HTTP API/Catalog. Operators should separate client agents from server quorum and treat ACLs as mandatory in any shared environment.

Consul servers (quorum / Raft)
        ^
        | gossip + RPC
        v
Client agents on nodes
        |
        +--> service registration + health checks
        +--> DNS / API queries
        +--> Connect proxies + intentions (mesh)

When to use

• Registering services with TCP/HTTP health checks
• Resolving healthy backends without hard-coded IPs
• Storing non-secret config in Consul KV (secrets belong in Vault)
• Defining mesh intentions (allow/deny service-to-service)

Operational directives

1. Run an odd-numbered server quorum (3 or 5); never one server in prod.
2. Enable ACLs; bootstrap once and store the bootstrap token offline.
3. Keep health checks honest - failing checks remove instances from discovery.
4. Do not put passwords in Consul KV; integrate Vault for secrets.
5. Prefer prepared queries / service resolvers for multi-DC failover designs.

Concrete examples

Service registration (JSON)

{
  "service": {
    "name": "api",
    "id": "api-1",
    "port": 8080,
    "tags": ["prod", "v1"],
    "check": {
      "http": "http://127.0.0.1:8080/healthz",
      "interval": "10s",
      "timeout": "2s"
    }
  }
}

consul services register api.json
consul catalog services
consul health service api -passing
dig @127.0.0.1 -p 8600 api.service.consul

KV config

consul kv put apps/api/log_level info
consul kv get apps/api/log_level
consul kv export apps/api/ > api-kv-backup.json

Intention (service mesh allow)

consul intention create -allow web api
consul intention list

Operator checks

consul members
consul operator raft list-peers
consul info

Discovery failure matrix

Best practices

1. Tag services with environment and version for canary routing.
2. Use health check deregister-after for crash-looping tasks.
3. Backup Raft/KV on a schedule; test restore.
4. Document gossip encryption key rotation separately from ACLs.

Limitations

• Consul is not a replacement for full east-west observability (pair with metrics/tracing).
• Mesh features add proxy sidecar operational cost.
• Multi-DC WAN federation needs careful network and ACL planning.

Related skills

• nomad - often co-deployed for scheduling with Consul discovery
• vault - secrets companion to Consul KV
• opentelemetry - telemetry beyond Consul health checks
• nginx-hardening - edge proxy in front of discovered backends