Cloud Networking Design

Cloud Networking Design (Verbose)

Core Patterns

VPC and Subnet Topology

A VPC is a private IP space you own inside a region. Everything else — routing, isolation, egress control — is built from how you subdivide it. Two rules pay off for years: choose a CIDR big enough that you never renumber, and never let two networks you might one day connect share an address range.

VPC 10.20.0.0/16                         (one per region per account)

  AZ-a                        AZ-b                        AZ-c
  public  10.20.0.0/20        public  10.20.16.0/20       public  10.20.32.0/20
  private 10.20.64.0/20       private 10.20.80.0/20       private 10.20.96.0/20
  data    10.20.128.0/20      data    10.20.144.0/20      data    10.20.160.0/20

Three subnet tiers, spread across the AZs the region offers:

• public — a route to the internet gateway. Load balancers and, if you must, bastions. Nothing stateful.
• private — application/compute. Egress via NAT; no inbound from the internet.
• data — databases, caches. No internet route at all, reached only from the private tier via security-group references.

Availability zones are the failure domain. Spreading subnets across AZs is what lets a load balancer survive a zone outage — but see the cost pattern below, because that same spread is where inter-AZ charges accrue.

Routing and Egress

"Public" and "private" are not attributes of a subnet; they are consequences of its route table.

public route table                 private route table
  10.20.0.0/16 -> local              10.20.0.0/16 -> local
  0.0.0.0/0    -> igw                0.0.0.0/0    -> nat-gw
                                     s3-prefix    -> vpce (gateway endpoint)

A NAT gateway lets private instances reach out (package installs, third-party APIs) while blocking all unsolicited inbound. It is also metered on both throughput and hours, which makes it a quiet cost center. Route provider-managed services (object storage, etc.) through a VPC endpoint so that traffic bypasses the NAT path entirely.

Security Groups vs NACLs

Two mechanisms, deliberately different, and best used together.

Security groups are the workhorse. Reference other groups instead of IP ranges so the policy tracks the topology, not addresses:

lb-sg:    inbound 443 from 0.0.0.0/0
app-sg:   inbound 8080 from lb-sg          # only the LB may reach the app
data-sg:  inbound 5432 from app-sg         # only the app may reach the DB

Reserve NACLs for coarse, subnet-wide statements — "this data subnet never talks to the internet" — where a deny rule adds a guardrail a security group cannot express. Do not try to run fine-grained policy in NACLs; the stateless return-port bookkeeping will bite you.

Connecting Networks

Non-transitivity is the concept people rediscover painfully. If A peers with B and B peers with C, A still cannot reach C. Peering is a point-to-point relationship, and n VPCs fully meshed need n(n-1)/2 connections.

Move to a transit gateway before the peering mesh grows past a handful of VPCs. PrivateLink is not a network join — it publishes a single service endpoint into a consumer VPC, so the consumer reaches exactly that service and nothing else, with no CIDR coordination required. That property makes it the clean answer for overlapping-CIDR situations peering cannot solve.

Private Access to Managed Services

Every hop to a managed service over a public endpoint is a hop over the internet (or through NAT) that you are paying for and exposing. Two private paths:

• Gateway endpoint — a route-table entry for storage-class services. No ENI, typically no hourly charge, keeps that traffic on the backbone.
• Interface endpoint / PSC — an ENI with a private IP in your subnet fronting the service. Hourly + data cost, but private and DNS-integrated.

The payoff is threefold: data never traverses the public internet, you drop the NAT hop, and you usually avoid the egress charge that the public path would incur.

The Cross-AZ and Egress Cost Trap

This is the pattern that surprises teams at scale. Three costs compound:

1. Inter-AZ traffic is billed, generally in both directions, even inside one VPC. A service whose replicas gossip across three AZs pays per hop.
2. NAT gateway charges per hour and per gigabyte processed.
3. Internet egress dwarfs everything else — data leaving the cloud is the expensive direction; inbound is usually free.

The tension is real: spreading across AZs buys availability, and concentrating in one AZ buys cheapness and latency. Resolve it deliberately. Keep high-volume, latency-sensitive chatter (cache reads, replica sync) zone-local with topology-aware routing, while still placing enough capacity in each AZ to survive losing one. See cloud-cost-optimization for turning these into budgets and alerts, and cloud-provider-tradeoffs for how the billing shapes differ.

Common Anti-Patterns

❌ Overlapping CIDR ranges across VPCs or with on-prem. Peering and VPN both refuse to route between overlapping ranges, and you cannot fix it without renumbering a live network. ✅ Allocate non-overlapping blocks from a central plan (an IPAM) up front.

❌ Databases in public subnets "behind a security group". One misordered rule and the data tier is internet-reachable. ✅ Data tier in subnets with no internet route at all; reach it only from the app tier via group references.

❌ Security groups written with 0.0.0.0/0 or wide CIDRs internally. The policy no longer reflects who actually talks to whom. ✅ Reference source security groups by id, so rules track topology.

❌ A full mesh of peering connections. It grows quadratically and becomes unauditable. ✅ Hub-and-spoke through a transit gateway once past a few VPCs.

❌ Routing managed-service and inter-service traffic over public endpoints/NAT. You pay egress and NAT for traffic that could stay on the backbone. ✅ Private endpoints for managed services; keep intra-cloud traffic private.

❌ Ignoring which AZ traffic lands in. Chatty cross-AZ paths bill silently. ✅ Make placement and routing topology-aware; measure inter-AZ bytes.

Network Design Checklist

• VPC CIDRs allocated from a non-overlapping, room-to-grow plan
• Three-tier subnets (public / private / data) across multiple AZs
• Only load balancers and bastions have an internet-gateway route
• Stateful and application workloads sit in private/data subnets
• Security groups reference source groups, not IP ranges, internally
• NACLs used only for coarse, subnet-wide guardrails
• Peering vs transit gateway chosen by VPC count; non-transitivity understood
• Managed services reached via VPC/interface endpoints, not the internet
• Inter-AZ, NAT, and egress traffic measured and budgeted
• Latency-sensitive high-volume traffic kept zone-local where safe
• Hybrid links (VPN vs dedicated) chosen against throughput/latency needs