Managing Cluster Capacity

Managing Cluster Capacity

Manages cluster capacity across all CockroachDB deployment tiers. What "capacity" means varies by tier — Self-Hosted manages individual nodes, Advanced/BYOC manage node count and machine size, Standard manages provisioned vCPUs, and Basic auto-scales with cost controls.

When to Use This Skill

• Permanently removing a node from a cluster (Self-Hosted)
• Adding nodes to increase capacity (Self-Hosted)
• Scaling cluster node count or machine size (Advanced, BYOC)
• Adjusting provisioned compute (Standard)
• Managing costs on a serverless cluster (Basic)
• Replacing hardware or migrating infrastructure (Self-Hosted, BYOC)
• Replacing a failed or dead node (Self-Hosted)
• Managing storage utilization and disk pressure (Self-Hosted)

For temporary maintenance (not capacity changes): Use performing-cluster-maintenance. For pre-operation health check: Use reviewing-cluster-health.

Step 1: Gather Context

Required Context

Additional Context (by tier)

If Self-Hosted (scaling down):

If Advanced or BYOC:

If Standard:

If Basic: Gather cost management goals — Basic auto-scales with no manual capacity control.

Context-Driven Routing

Self-Hosted Capacity Management

Applies when: Tier = Self-Hosted

Scaling Down: Decommission Nodes

Pre-Decommission Validation

# All nodes live, version-consistent, with replication and per-node range counts
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>

Inspect the output for:

• is_live = true for every node
• ranges_underreplicated is 0 everywhere (all ranges fully replicated)

-- Replication factor (and other zone-level settings)
SHOW ZONE CONFIGURATION FOR RANGE default;

For per-store capacity (so you can verify remaining nodes won't exceed 60% utilization after absorbing the decommissioned node's data), use the DB Console Overview → Storage page or scrape the Prometheus metrics endpoint:

curl -ks https://<node>:8080/_status/vars | grep '^capacity'

Node count after decommission must be ≥ the zone's num_replicas.

If Node Is Alive: Drain Then Decommission

# Step 1: Drain
cockroach node drain <node_id> --certs-dir=<certs-dir> --host=<any-live-node>

# Step 2: Decommission (single node)
cockroach node decommission <node_id> --certs-dir=<certs-dir> --host=<any-live-node>

# Step 2: Decommission (multiple nodes — more efficient, do simultaneously)
cockroach node decommission <id_1> <id_2> <id_3> --certs-dir=<certs-dir> --host=<any-live-node>

If Node Is Dead: Replace Failed Node

When a node has been dead longer than server.time_until_store_dead (default 5m), CockroachDB automatically re-replicates its data to surviving nodes. Use this procedure to clean up the dead node and optionally add a replacement.

Step 1: Confirm the node is dead and data is safe

# Confirm the dead node and verify replication has caught up
cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>

In the output: the dead node should show is_live = false, and every surviving node should show ranges_underreplicated = 0. For per-store capacity on the surviving nodes, use the DB Console Overview → Storage page.

If under-replicated ranges exist, wait for re-replication to complete before proceeding.

Step 2: Decommission the dead node (metadata cleanup)

cockroach node decommission <dead_node_id> --certs-dir=<certs-dir> --host=<any-live-node>

Step 3: Add a replacement node (recommended)

If remaining nodes are above 60% utilization, provision a replacement node using the Scaling Up: Add Nodes procedure.

Multiple dead nodes: Decommission all dead nodes simultaneously:

cockroach node decommission <id_1> <id_2> --certs-dir=<certs-dir> --host=<any-live-node>

See replacing-failed-nodes reference for detailed failure scenarios and recovery procedures.

Monitor Decommission Progress

cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>

Wait for gossiped_replicas = 0 and membership = 'decommissioned'. Then stop the process on the decommissioned node.

Cancel a Decommission

cockroach node recommission <node_id> --certs-dir=<certs-dir> --host=<any-live-node>

Only works while still in decommissioning state.

Scaling Up: Add Nodes

1. Provision new hardware/VM with same specs as existing nodes
2. Install same CockroachDB version (cockroach version to confirm)
3. Start node with --join pointing to existing cluster nodes
4. Verify join and monitor rebalancing:

   cockroach node status --certs-dir=<certs-dir> --host=<any-live-node>
   

   The new node should appear in the output with is_live = true. The ranges column climbs as data rebalances toward the new node.

Post-Scaling Verification

cockroach node status --decommission --certs-dir=<certs-dir> --host=<any-live-node>

Expect ranges_underreplicated = 0 on every node and a balanced ranges count across nodes. For per-store capacity utilization, use the DB Console Overview → Storage page.

Advanced Scaling

Applies when: Tier = Advanced

Advanced clusters are managed by Cockroach Labs. Capacity is adjusted by changing node count or machine size.

Via Cloud Console

1. Cluster → Capacity
2. Adjust node count or machine type (vCPUs per node)
3. CRL handles all node operations (drain, decommission, provisioning) safely
4. Monitor progress in Cloud Console

Via Cloud API

# Scale node count
curl -X PATCH -H "Authorization: Bearer $COCKROACH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"config": {"num_nodes": <new_count>}}' \
  "https://cockroachlabs.cloud/api/v1/clusters/<cluster-id>"

Via Terraform

resource "cockroach_cluster" "example" {
  dedicated {
    num_virtual_cpus = 8     # vCPUs per node
    storage_gib      = 150
    num_nodes        = 5     # total nodes
  }
}

Pre-Scaling Check

-- Ensure no disruptive jobs are running before scaling down
WITH j AS (SHOW JOBS)
SELECT job_type, status, COUNT(*) FROM j WHERE status = 'running' GROUP BY 1, 2;

Constraints

• Minimum: 3 nodes x 4 vCPUs (12 vCPUs total)
• Scale down: Data must fit on remaining nodes; zone configs must be satisfiable
• Scale up: Additional nodes available within your plan limits

BYOC Scaling

Applies when: Tier = BYOC

Follow all Advanced Scaling steps. BYOC scaling is managed through the same Cloud Console/API/Terraform interfaces.

Cloud Provider Verification (after scaling down)

If AWS:

aws ec2 describe-instances --filters "Name=tag:cockroach-cluster,Values=<cluster-name>" \
  --query 'Reservations[].Instances[].{ID:InstanceId,State:State.Name}'

If GCP:

gcloud compute instances list --filter="labels.cockroach-cluster=<cluster-name>"

If Azure:

az vm list --resource-group <rg> --query "[?tags.cockroachCluster=='<name>']"

Additional BYOC Considerations

• Verify security groups/firewall rules after scaling
• Update reserved instance or committed use discount allocations
• Verify network connectivity (PrivateLink/PSC/VPC Peering) is unaffected
• Check cloud billing reflects the new instance count

Standard Compute Management

Applies when: Tier = Standard

Standard is a multi-tenant managed service. There are no individual nodes. Capacity is managed by adjusting provisioned compute (vCPUs).

Adjust Provisioned vCPUs

1. Cloud Console → Cluster → Capacity
2. Increase or decrease provisioned vCPUs
3. Change takes effect without downtime

Before Scaling Down

• Review CPU utilization in Cloud Console — ensure workload fits within reduced compute
• Storage is usage-based and unaffected by compute changes

After Scaling

Monitor P99 latency and QPS in Cloud Console for 24-48 hours. If latency increases after scaling down, scale compute back up.

Basic Cost Management

Applies when: Tier = Basic

Basic is a serverless offering that auto-scales. There are no nodes or provisioned compute to manage. Capacity scales automatically based on demand. Cost is managed through spending controls.

Manage Spending

• Set spending limits: Cloud Console → Cluster → Settings → configure monthly spending cap
• Review usage: Cloud Console shows Request Unit (RU) consumption over time
• Optimize queries: Reduce RU consumption through query tuning and indexing
• Archive data: Delete unused tables or databases to reduce storage costs

When to Consider Upgrading

If you need explicit control over compute capacity (guaranteed vCPUs), consider upgrading to Standard. If you need dedicated infrastructure, consider Advanced.

Safety Considerations

Critical (Self-Hosted):

• Never decommission below the replication factor
• Always drain before decommission (for live nodes)
• Decommission multiple nodes simultaneously (not sequentially)
• Verify remaining capacity can absorb the data
• For dead nodes: wait for re-replication to complete before decommissioning
• Monitor storage utilization — nodes above 80% risk performance degradation

Troubleshooting

References

Skill references:

• Replacing failed nodes
• Storage management

Related skills:

• reviewing-cluster-health — Pre/post health checks
• performing-cluster-maintenance — Drain procedure (SH)
• upgrading-cluster-version — Upgrades and lifecycle

Official CockroachDB Documentation:

• cockroach node decommission
• Node Shutdown
• Manage Advanced Cluster
• Manage Standard Cluster
• Cloud API