copyleftdev Jim Gettys Bufferbloat Style Guide
Overview
Jim Gettys, while working at Bell Labs and later on the One Laptop per Child project, discovered and named "bufferbloat"—the phenomenon where excessive buffering in network equipment causes massive latency spikes. Modern networks often have seconds of buffering, destroying interactive performance even when bandwidth is plentiful. Gettys' crusade to fix bufferbloat led to fq_codel and the understanding that network latency under load is the true measure of network quality.
Core Philosophy
"Latency is the new bandwidth. We have plenty of bandwidth; what we lack is low latency."
"The buffer is full of lies. Every packet in that buffer is a broken promise about when it will arrive."
"Good networks feel fast. Bufferbloated networks feel like wading through molasses."
Gettys realized that optimizing for throughput while ignoring latency creates terrible user experience. A network with 100ms idle RTT that spikes to 2000ms under load is fundamentally broken, even if it achieves high throughput. The solution is to keep queues short and managed.
Design Principles
Latency Under Load Matters: Measure RTT while the network is busy, not idle.
Buffers Lie About Bandwidth: Large buffers mask congestion, delaying feedback.
Queues Should Be Short: Aim for milliseconds of buffering, not seconds.
Flow Isolation: One greedy flow shouldn't destroy latency for others.
Active Queue Management: Don't just drop when full—manage proactively.
The Bufferbloat Problem
Without Bufferbloat (healthy network):
─────────────────────────────────────
Idle RTT: 20ms
Load RTT: 25ms (slight increase)
Difference: 5ms ✓ Good!
With Bufferbloat (broken network):
──────────────────────────────────
Idle RTT: 20ms
Load RTT: 2000ms (100x increase!)
Difference: 1980ms ✗ Terrible!
Why does this happen?
┌─────────────────────────────────────────────────────────────┐
│ │
│ Sender Router Receiver │
│ ────── ────── ──────── │
│ │
│ 100 Mbps ─────────► ┌─────────┐ ─────────► 10 Mbps │
│ │ BUFFER │ │
│ │█████████│ ← 2 seconds of packets! │
│ │█████████│ │
│ │█████████│ │
│ └─────────┘ │
│ │
│ Packets queue up waiting for the slow link. │
│ TCP doesn't know—it sees ACKs arriving (eventually). │
│ User sees lag, even with "good bandwidth." │
│ │
└─────────────────────────────────────────────────────────────┘
When Engineering Low-Latency Networks
Always
- Measure latency UNDER LOAD, not just idle
- Use fq_codel or similar AQM on bottleneck queues
- Size buffers based on BDP, not maximum possible
- Test with realistic traffic patterns
- Monitor queue depth, not just throughput
- Prioritize latency for interactive traffic
Never
- Assume more buffering is better
- Measure only idle RTT as "ping time"
- Optimize only for throughput benchmarks
- Use deep buffers "just in case"
- Ignore latency complaints with "bandwidth is fine"
- Conflate bandwidth with network quality
Prefer
- Shallow queues over deep buffers
- Fair queuing over FIFO
- AQM over tail-drop
- Latency metrics over throughput
- Per-flow isolation
- Measuring under load
Code Patterns
Bufferbloat Detection
class BufferbloatDetector:
"""
Detect bufferbloat by comparing idle vs loaded RTT.
Gettys' insight: the difference tells you everything.
"""
def __init__(self, target_host: str):
self.target = target_host
self.idle_samples = []
self.loaded_samples = []
def measure_idle_rtt(self, samples: int = 20) -> float:
"""
Measure RTT when network is idle.
"""
rtts = []
for _ in range(samples):
rtt = self._ping(self.target)
if rtt is not None:
rtts.append(rtt)
time.sleep(0.1)
self.idle_samples = rtts
return min(rtts) if rtts else None
def measure_loaded_rtt(self,
samples: int = 20,
load_generator: Callable = None) -> float:
"""
Measure RTT while generating load.
"""
# Start background load
if load_generator:
load_thread = threading.Thread(target=load_generator)
load_thread.start()
time.sleep(1) # Let load stabilize
rtts = []
for _ in range(samples):
rtt = self._ping(self.target)
if rtt is not None:
rtts.append(rtt)
time.sleep(0.1)
self.loaded_samples = rtts
return sum(rtts) / len(rtts) if rtts else None
def diagnose(self) -> BufferbloatDiagnosis:
"""
Diagnose bufferbloat severity.
"""
if not self.idle_samples or not self.loaded_samples:
return BufferbloatDiagnosis(status='insufficient_data')
baseline = min(self.idle_samples)
loaded_avg = sum(self.loaded_samples) / len(self.loaded_samples)
loaded_max = max(self.loaded_samples)
bloat = loaded_avg - baseline
bloat_ratio = loaded_avg / baseline if baseline > 0 else float('inf')
# Gettys' thresholds
if bloat < 5:
grade = 'A'
status = 'excellent'
recommendation = 'Network is well-tuned'
elif bloat < 30:
grade = 'B'
status = 'good'
recommendation = 'Minor bufferbloat, acceptable for most uses'
elif bloat < 100:
grade = 'C'
status = 'moderate'
recommendation = 'Noticeable lag under load, enable fq_codel'
elif bloat < 300:
grade = 'D'
status = 'poor'
recommendation = 'Significant bufferbloat, enable AQM immediately'
else:
grade = 'F'
status = 'severe'
recommendation = 'Severe bufferbloat, network unusable for interactive use'
return BufferbloatDiagnosis(
grade=grade,
status=status,
baseline_rtt=baseline,
loaded_rtt=loaded_avg,
bloat_ms=bloat,
bloat_ratio=bloat_ratio,
recommendation=recommendation,
)
def _ping(self, host: str) -> Optional[float]:
"""Send ICMP ping and return RTT in ms."""
try:
result = subprocess.run(
['ping', '-c', '1', '-W', '1', host],
capture_output=True,
text=True
)
# Parse RTT from ping output
match = re.search(r'time=(\d+\.?\d*)', result.stdout)
if match:
return float(match.group(1))
except Exception:
pass
return None
def run_bufferbloat_test(target: str = '8.8.8.8') -> BufferbloatDiagnosis:
"""
Run a complete bufferbloat test.
"""
detector = BufferbloatDetector(target)
print("Measuring idle RTT...")
detector.measure_idle_rtt()
print("Measuring RTT under load...")
def generate_load():
# Download something large
subprocess.run(
['curl', '-o', '/dev/null', '-s',
'http://speedtest.tele2.net/100MB.zip'],
timeout=30
)
detector.measure_loaded_rtt(load_generator=generate_load)
return detector.diagnose()
fq_codel Implementation
class FQCoDel:
"""
Fair Queuing with Controlled Delay (fq_codel).
The solution to bufferbloat: per-flow fair queuing + CoDel AQM.
Key innovations:
1. Flow isolation: one flow can't bloat another
2. Per-flow AQM: CoDel applied to each flow
3. Fair sharing: all flows get equal share of bandwidth
"""
def __init__(self,
num_queues: int = 1024,
target_ms: float = 5.0,
interval_ms: float = 100.0,
quantum: int = 1514):
self.num_queues = num_queues
self.target = target_ms
self.interval = interval_ms
self.quantum = quantum # Bytes per round
self.queues = [FlowQueue(target_ms, interval_ms)
for _ in range(num_queues)]
self.active_list = [] # Flows with packets
self.flow_states = {} # Per-flow state
def hash_flow(self, packet: Packet) -> int:
"""
Hash packet to a queue based on flow (5-tuple).
"""
flow_id = (
packet.src_ip,
packet.dst_ip,
packet.src_port,
packet.dst_port,
packet.protocol
)
return hash(flow_id) % self.num_queues
def enqueue(self, packet: Packet, now_ms: float) -> bool:
"""
Enqueue a packet to its flow's queue.
"""
queue_idx = self.hash_flow(packet)
queue = self.queues[queue_idx]
packet.enqueue_time = now_ms
was_empty = queue.is_empty()
success = queue.enqueue(packet)
if success and was_empty:
# Flow became active, add to round-robin
self.active_list.append(queue_idx)
return success
def dequeue(self, now_ms: float) -> Optional[Packet]:
"""
Dequeue using deficit round-robin with CoDel.
"""
if not self.active_list:
return None
# Try each active queue in round-robin order
for _ in range(len(self.active_list)):
queue_idx = self.active_list[0]
queue = self.queues[queue_idx]
# Apply CoDel to this flow's queue
packet = queue.codel_dequeue(now_ms)
if packet is not None:
# Got a packet, update deficit
queue.deficit += self.quantum
queue.deficit -= len(packet.data)
if queue.deficit < 0:
# Exhausted quantum, move to back of active list
self.active_list.append(self.active_list.pop(0))
queue.deficit = 0
return packet
else:
# Queue empty, remove from active list
self.active_list.pop(0)
queue.deficit = 0
return None
class FlowQueue:
"""
Per-flow queue with CoDel.
"""
def __init__(self, target_ms: float, interval_ms: float, max_size: int = 10240):
self.packets = deque()
self.max_size = max_size
self.deficit = 0
# CoDel state
self.target = target_ms
self.interval = interval_ms
self.first_above_time = None
self.drop_next = 0
self.count = 0
self.dropping = False
def is_empty(self) -> bool:
return len(self.packets) == 0
def enqueue(self, packet: Packet) -> bool:
if len(self.packets) >= self.max_size:
return False
self.packets.append(packet)
return True
def codel_dequeue(self, now_ms: float) -> Optional[Packet]:
"""
Dequeue with CoDel logic.
"""
if not self.packets:
self.dropping = False
return None
packet = self.packets[0]
sojourn_time = now_ms - packet.enqueue_time
if sojourn_time < self.target:
# Good: below target
self.first_above_time = None
else:
if self.first_above_time is None:
self.first_above_time = now_ms + self.interval
elif now_ms >= self.first_above_time:
# Persistent delay: consider dropping
pass
if self.dropping:
if sojourn_time < self.target:
# Delay recovered, stop dropping
self.dropping = False
elif now_ms >= self.drop_next:
# Time to drop
self.packets.popleft() # Drop
self.count += 1
self.drop_next = now_ms + self.interval / (self.count ** 0.5)
return self.codel_dequeue(now_ms) # Try next
elif self.first_above_time and now_ms >= self.first_above_time:
# Start dropping
self.dropping = True
self.count = 1
self.drop_next = now_ms + self.interval
self.packets.popleft() # Drop
return self.codel_dequeue(now_ms) # Try next
return self.packets.popleft()
Network Quality Score
class NetworkQualityScore:
"""
Score network quality the Gettys way: latency under load.
"""
@staticmethod
def calculate_score(measurements: NetworkMeasurements) -> QualityScore:
"""
Calculate a network quality score.
Key insight: combine baseline latency, bloat, and jitter.
"""
baseline = measurements.baseline_rtt
loaded = measurements.loaded_rtt
jitter = measurements.jitter
loss = measurements.packet_loss
# Bloat penalty
bloat = loaded - baseline
bloat_factor = 1.0 / (1.0 + bloat / 50.0) # Penalize heavily
# Baseline penalty (prefer low latency)
baseline_factor = 1.0 / (1.0 + baseline / 100.0)
# Jitter penalty
jitter_factor = 1.0 / (1.0 + jitter / 20.0)
# Loss penalty (severe)
loss_factor = (1.0 - loss) ** 2
# Combined score (0-100)
raw_score = (bloat_factor * 0.5 +
baseline_factor * 0.2 +
jitter_factor * 0.2 +
loss_factor * 0.1)
score = int(raw_score * 100)
# Grade
if score >= 90:
grade = 'A'
elif score >= 75:
grade = 'B'
elif score >= 60:
grade = 'C'
elif score >= 40:
grade = 'D'
else:
grade = 'F'
return QualityScore(
score=score,
grade=grade,
baseline_rtt=baseline,
bloat=bloat,
jitter=jitter,
loss=loss,
bottleneck=identify_bottleneck(measurements),
)
def identify_bottleneck(measurements: NetworkMeasurements) -> str:
"""
Identify what's hurting network quality most.
"""
bloat = measurements.loaded_rtt - measurements.baseline_rtt
if bloat > 100:
return 'bufferbloat'
elif measurements.baseline_rtt > 100:
return 'high_base_latency'
elif measurements.jitter > 30:
return 'jitter'
elif measurements.packet_loss > 0.01:
return 'packet_loss'
else:
return 'none'
Buffer Sizing
class BufferSizing:
"""
Size buffers correctly to avoid bloat while maintaining throughput.
"""
@staticmethod
def calculate_optimal_buffer(bandwidth_mbps: float,
rtt_ms: float,
num_flows: int = 1) -> BufferRecommendation:
"""
Calculate optimal buffer size.
Rule of thumb (for N flows):
Buffer = BDP / sqrt(N)
Where BDP = Bandwidth × RTT
"""
# Bandwidth-Delay Product
bandwidth_bytes_per_sec = bandwidth_mbps * 1_000_000 / 8
rtt_sec = rtt_ms / 1000
bdp_bytes = bandwidth_bytes_per_sec * rtt_sec
# Buffer size
if num_flows == 1:
buffer_bytes = bdp_bytes
else:
# Appenzeller et al: BDP / sqrt(N)
buffer_bytes = bdp_bytes / (num_flows ** 0.5)
# Convert to practical units
buffer_packets = int(buffer_bytes / 1500) # MTU
buffer_ms = rtt_ms / (num_flows ** 0.5) if num_flows > 1 else rtt_ms
return BufferRecommendation(
bdp_bytes=int(bdp_bytes),
recommended_bytes=int(buffer_bytes),
recommended_packets=buffer_packets,
recommended_ms=buffer_ms,
explanation=(
f"For {bandwidth_mbps} Mbps link with {rtt_ms}ms RTT "
f"and ~{num_flows} flows, buffer {buffer_packets} packets "
f"(~{buffer_ms:.1f}ms of data)"
)
)
@staticmethod
def linux_buffer_settings(buffer_bytes: int) -> dict:
"""
Generate Linux sysctl settings for buffer sizes.
"""
return {
'net.core.rmem_max': buffer_bytes,
'net.core.wmem_max': buffer_bytes,
'net.ipv4.tcp_rmem': f'4096 87380 {buffer_bytes}',
'net.ipv4.tcp_wmem': f'4096 65536 {buffer_bytes}',
'net.core.netdev_max_backlog': 1000, # Reduce from default
}
@staticmethod
def enable_fq_codel(interface: str) -> str:
"""
Generate command to enable fq_codel on an interface.
"""
return f"""
# Enable fq_codel on {interface}
tc qdisc del dev {interface} root 2>/dev/null
tc qdisc add dev {interface} root fq_codel
# Verify
tc -s qdisc show dev {interface}
"""
Mental Model
Gettys approaches network performance by asking:
- What's the RTT under load? That's the true latency
- How deep are the buffers? Seconds of buffering = seconds of lag
- Is there flow isolation? One flow shouldn't ruin others
- Is AQM enabled? fq_codel should be everywhere
- Would I notice lag? User experience is the metric
The Bufferbloat Checklist
□ Measure RTT under load, not idle
□ Compare loaded RTT to baseline (>10x = severe bloat)
□ Enable fq_codel on all bottleneck queues
□ Size buffers based on BDP, not maximum
□ Test with interactive + bulk traffic together
□ Monitor queue depth, not just throughput
□ Grade with dslreports.com/speedtest or similar
□ Check router, modem, AND ISP equipment
Signature Gettys Moves
- Bufferbloat diagnosis (idle vs loaded RTT)
- fq_codel as the universal solution
- "Latency is the new bandwidth"
- Flow isolation requirement
- Queue depth monitoring
- BDP-based buffer sizing
- User experience as the metric
- Crusading for AQM everywhere