gettys-bufferbloat

name	gettys-bufferbloat
description	Engineer low-latency networks in the style of Jim Gettys, discoverer of bufferbloat. Emphasizes understanding excessive buffering, queue management, latency under load, and the fq_codel solution. Use when diagnosing network latency issues, optimizing for real-time applications, or implementing queue management.
tags	networking, latency, bufferbloat, tcp, congestion, aqm, queuing, real-time, bandwidth, packet-loss

Jim Gettys Bufferbloat Style Guide⁠‍⁠​‌​‌​​‌‌‍​‌​​‌​‌‌‍​​‌‌​​​‌‍​‌​​‌‌​​‍​​​​​​​‌‍‌​​‌‌​‌​‍‌​​​​​​​‍‌‌​​‌‌‌‌‍‌‌​​​‌​​‍‌‌‌‌‌‌​‌‍‌‌​‌​​​​‍​‌​‌‌‌‌‌‍​‌​​‌​‌‌‍​‌‌​‌​​‌‍‌​‌​‌‌‌​‍​​‌​‌​​​‍‌‌‌​‌​‌‌‍​‌‌​‌‌​‌‍​‌​‌​​‌​‍​​‌​​​​‌‍​​​​​‌​‌‍​​​​‌​​‌‍​‌​​‌‌‌‌⁠‍⁠

Overview

Jim Gettys, while working at Bell Labs and later on the One Laptop per Child project, discovered and named "bufferbloat"—the phenomenon where excessive buffering in network equipment causes massive latency spikes. Modern networks often have seconds of buffering, destroying interactive performance even when bandwidth is plentiful. Gettys' crusade to fix bufferbloat led to fq_codel and the understanding that network latency under load is the true measure of network quality.

Core Philosophy

"Latency is the new bandwidth. We have plenty of bandwidth; what we lack is low latency."

"The buffer is full of lies. Every packet in that buffer is a broken promise about when it will arrive."

"Good networks feel fast. Bufferbloated networks feel like wading through molasses."

Gettys realized that optimizing for throughput while ignoring latency creates terrible user experience. A network with 100ms idle RTT that spikes to 2000ms under load is fundamentally broken, even if it achieves high throughput. The solution is to keep queues short and managed.

Design Principles

1. Latency Under Load Matters: Measure RTT while the network is busy, not idle.

2. Buffers Lie About Bandwidth: Large buffers mask congestion, delaying feedback.

3. Queues Should Be Short: Aim for milliseconds of buffering, not seconds.

4. Flow Isolation: One greedy flow shouldn't destroy latency for others.

5. Active Queue Management: Don't just drop when full—manage proactively.

The Bufferbloat Problem

Without Bufferbloat (healthy network):
─────────────────────────────────────
Idle RTT:    20ms
Load RTT:    25ms  (slight increase)
Difference:   5ms  ✓ Good!

With Bufferbloat (broken network):
──────────────────────────────────
Idle RTT:    20ms
Load RTT:  2000ms  (100x increase!)
Difference: 1980ms ✗ Terrible!


Why does this happen?

┌─────────────────────────────────────────────────────────────┐
│                                                              │
│   Sender                    Router                Receiver   │
│   ──────                    ──────                ────────   │
│                                                              │
│   100 Mbps  ─────────►  ┌─────────┐  ─────────►  10 Mbps    │
│                         │ BUFFER  │                          │
│                         │█████████│ ← 2 seconds of packets! │
│                         │█████████│                          │
│                         │█████████│                          │
│                         └─────────┘                          │
│                                                              │
│   Packets queue up waiting for the slow link.               │
│   TCP doesn't know—it sees ACKs arriving (eventually).      │
│   User sees lag, even with "good bandwidth."                │
│                                                              │
└─────────────────────────────────────────────────────────────┘

When Engineering Low-Latency Networks

Always

• Measure latency UNDER LOAD, not just idle
• Use fq_codel or similar AQM on bottleneck queues
• Size buffers based on BDP, not maximum possible
• Test with realistic traffic patterns
• Monitor queue depth, not just throughput
• Prioritize latency for interactive traffic

Never

• Assume more buffering is better
• Measure only idle RTT as "ping time"
• Optimize only for throughput benchmarks
• Use deep buffers "just in case"
• Ignore latency complaints with "bandwidth is fine"
• Conflate bandwidth with network quality

Prefer

• Shallow queues over deep buffers
• Fair queuing over FIFO
• AQM over tail-drop
• Latency metrics over throughput
• Per-flow isolation
• Measuring under load

Code Patterns

Bufferbloat Detection

class BufferbloatDetector:
    """
    Detect bufferbloat by comparing idle vs loaded RTT.
    Gettys' insight: the difference tells you everything.
    """
    
    def __init__(self, target_host: str):
        self.target = target_host
        self.idle_samples = []
        self.loaded_samples = []
    
    def measure_idle_rtt(self, samples: int = 20) -> float:
        """
        Measure RTT when network is idle.
        """
        rtts = []
        for _ in range(samples):
            rtt = self._ping(self.target)
            if rtt is not None:
                rtts.append(rtt)
            time.sleep(0.1)
        
        self.idle_samples = rtts
        return min(rtts) if rtts else None
    
    def measure_loaded_rtt(self, 
                            samples: int = 20,
                            load_generator: Callable = None) -> float:
        """
        Measure RTT while generating load.
        """
        # Start background load
        if load_generator:
            load_thread = threading.Thread(target=load_generator)
            load_thread.start()
        
        time.sleep()  
        
        rtts = []
         _  (samples):
            rtt = ._ping(.target)
             rtt   :
                rtts.append(rtt)
            time.sleep()
        
        .loaded_samples = rtts
         (rtts) / (rtts)  rtts  
    
     () -> BufferbloatDiagnosis:
        
          .idle_samples   .loaded_samples:
             BufferbloatDiagnosis(status=)
        
        baseline = (.idle_samples)
        loaded_avg = (.loaded_samples) / (.loaded_samples)
        loaded_max = (.loaded_samples)
        
        bloat = loaded_avg - baseline
        bloat_ratio = loaded_avg / baseline  baseline >   ()
        
        
         bloat < :
            grade = 
            status = 
            recommendation = 
         bloat < :
            grade = 
            status = 
            recommendation = 
         bloat < :
            grade = 
            status = 
            recommendation = 
         bloat < :
            grade = 
            status = 
            recommendation = 
        :
            grade = 
            status = 
            recommendation = 
        
         BufferbloatDiagnosis(
            grade=grade,
            status=status,
            baseline_rtt=baseline,
            loaded_rtt=loaded_avg,
            bloat_ms=bloat,
            bloat_ratio=bloat_ratio,
            recommendation=recommendation,
        )
    
     () -> []:
        
        :
            result = subprocess.run(
                [, , , , , host],
                capture_output=,
                text=
            )
            
             = re.search(, result.stdout)
             :
                 (.group())
         Exception:
            
         


 () -> BufferbloatDiagnosis:
    
    detector = BufferbloatDetector(target)
    
    ()
    detector.measure_idle_rtt()
    
    ()
    
     ():
        
        subprocess.run(
            [, , , , 
             ],
            timeout=
        )
    
    detector.measure_loaded_rtt(load_generator=generate_load)
    
     detector.diagnose()

fq_codel Implementation

class FQCoDel:
    """
    Fair Queuing with Controlled Delay (fq_codel).
    The solution to bufferbloat: per-flow fair queuing + CoDel AQM.
    
    Key innovations:
    1. Flow isolation: one flow can't bloat another
    2. Per-flow AQM: CoDel applied to each flow
    3. Fair sharing: all flows get equal share of bandwidth
    """
    
    def __init__(self,
                 num_queues: int = 1024,
                 target_ms: float = 5.0,
                 interval_ms: float = 100.0,
                 quantum: int = 1514):
        self.num_queues = num_queues
        self.target = target_ms
        self.interval = interval_ms
        self.quantum = quantum  # Bytes per round
        
        self.queues = [FlowQueue(target_ms, interval_ms) 
                       for _ in range(num_queues)]
        self.active_list = []  # Flows with packets
        self.flow_states = {}  # Per-flow state
    
    def hash_flow(self, packet: Packet) -> int:
        """
        Hash packet to a queue based on flow (5-tuple).
        """
        flow_id = (
            packet.src_ip,
            packet.dst_ip,
            packet.src_port,
            packet.dst_port,
            packet.protocol
        )
        return hash(flow_id) % self.num_queues
    
    def enqueue() -> :
        
        queue_idx = .hash_flow(packet)
        queue = .queues[queue_idx]
        
        packet.enqueue_time = now_ms
        
        was_empty = queue.is_empty()
        success = queue.enqueue(packet)
        
         success  was_empty:
            
            .active_list.append(queue_idx)
        
         success
    
     () -> [Packet]:
        
          .active_list:
             
        
        
         _  ((.active_list)):
            queue_idx = .active_list[]
            queue = .queues[queue_idx]
            
            
            packet = queue.codel_dequeue(now_ms)
            
             packet   :
                
                queue.deficit += .quantum
                queue.deficit -= (packet.data)
                
                 queue.deficit < :
                    
                    .active_list.append(.active_list.pop())
                    queue.deficit = 
                
                 packet
            :
                
                .active_list.pop()
                queue.deficit = 
        
         


 :
    
    
     ():
        .packets = deque()
        .max_size = max_size
        .deficit = 
        
        
        .target = target_ms
        .interval = interval_ms
        .first_above_time = 
        .drop_next = 
        .count = 
        .dropping = 
    
     () -> :
         (.packets) == 
    
     () -> :
         (.packets) >= .max_size:
             
        .packets.append(packet)
         
    
     () -> [Packet]:
        
          .packets:
            .dropping = 
             
        
        packet = .packets[]
        sojourn_time = now_ms - packet.enqueue_time
        
         sojourn_time < .target:
            
            .first_above_time = 
        :
             .first_above_time  :
                .first_above_time = now_ms + .interval
             now_ms >= .first_above_time:
                
                
        
         .dropping:
             sojourn_time < .target:
                
                .dropping = 
             now_ms >= .drop_next:
                
                .packets.popleft()  
                .count += 
                .drop_next = now_ms + .interval / (.count ** )
                 .codel_dequeue(now_ms)  
         .first_above_time  now_ms >= .first_above_time:
            
            .dropping = 
            .count = 
            .drop_next = now_ms + .interval
            .packets.popleft()  
             .codel_dequeue(now_ms)  
        
         .packets.popleft()

Network Quality Score

class NetworkQualityScore:
    """
    Score network quality the Gettys way: latency under load.
    """
    
    @staticmethod
    def calculate_score(measurements: NetworkMeasurements) -> QualityScore:
        """
        Calculate a network quality score.
        
        Key insight: combine baseline latency, bloat, and jitter.
        """
        baseline = measurements.baseline_rtt
        loaded = measurements.loaded_rtt
        jitter = measurements.jitter
        loss = measurements.packet_loss
        
        # Bloat penalty
        bloat = loaded - baseline
        bloat_factor = 1.0 / (1.0 + bloat / 50.0)  # Penalize heavily
        
        # Baseline penalty (prefer low latency)
        baseline_factor = 1.0 / (1.0 + baseline / 100.0)
        
        # Jitter penalty
        jitter_factor = 1.0 / (1.0 + jitter / 20.0)
        
        # Loss penalty (severe)
        loss_factor = (1.0 - loss) ** 2
        
        # Combined score (0-100)
        raw_score = (bloat_factor * 0.5 + 
                     baseline_factor * 0.2 + 
                     jitter_factor * 0.2 + 
                     loss_factor * 0.1)
        
        score = int(raw_score * 100)
        
        # Grade
        if score >= 90:
            grade = 'A'
        elif score >= 75:
            grade = 'B'
        elif score >= :
            grade = 
         score >= :
            grade = 
        :
            grade = 
        
         QualityScore(
            score=score,
            grade=grade,
            baseline_rtt=baseline,
            bloat=bloat,
            jitter=jitter,
            loss=loss,
            bottleneck=identify_bottleneck(measurements),
        )


 () -> :
    
    bloat = measurements.loaded_rtt - measurements.baseline_rtt
    
     bloat > :
         
     measurements.baseline_rtt > :
         
     measurements.jitter > :
         
     measurements.packet_loss > :
         
    :
         

Buffer Sizing

class BufferSizing:
    """
    Size buffers correctly to avoid bloat while maintaining throughput.
    """
    
    @staticmethod
    def calculate_optimal_buffer(bandwidth_mbps: float,
                                  rtt_ms: float,
                                  num_flows: int = 1) -> BufferRecommendation:
        """
        Calculate optimal buffer size.
        
        Rule of thumb (for N flows):
            Buffer = BDP / sqrt(N)
        
        Where BDP = Bandwidth × RTT
        """
        # Bandwidth-Delay Product
        bandwidth_bytes_per_sec = bandwidth_mbps * 1_000_000 / 8
        rtt_sec = rtt_ms / 1000
        bdp_bytes = bandwidth_bytes_per_sec * rtt_sec
        
        # Buffer size
        if num_flows == 1:
            buffer_bytes = bdp_bytes
        else:
            # Appenzeller et al: BDP / sqrt(N)
            buffer_bytes = bdp_bytes / (num_flows ** 0.5)
        
        # Convert to practical units
        buffer_packets = int(buffer_bytes / 1500)  # MTU
        buffer_ms = rtt_ms / (num_flows ** 0.5) if num_flows > 1 else rtt_ms
        
        return BufferRecommendation(
            bdp_bytes=int(bdp_bytes),
            recommended_bytes=int(buffer_bytes),
            recommended_packets=buffer_packets,
            recommended_ms=buffer_ms,
            explanation=(
                f"For {bandwidth_mbps} Mbps link with {rtt_ms}ms RTT "
                f"and ~ flows, buffer  packets "
                
            )
        )
    

     () -> :
        
         {
            : buffer_bytes,
            : buffer_bytes,
            : ,
            : ,
            : ,  
        }
    

     () -> :
        
         

Mental Model

Gettys approaches network performance by asking:

1. What's the RTT under load? That's the true latency
2. How deep are the buffers? Seconds of buffering = seconds of lag
3. Is there flow isolation? One flow shouldn't ruin others
4. Is AQM enabled? fq_codel should be everywhere
5. Would I notice lag? User experience is the metric

The Bufferbloat Checklist

□ Measure RTT under load, not idle
□ Compare loaded RTT to baseline (>10x = severe bloat)
□ Enable fq_codel on all bottleneck queues
□ Size buffers based on BDP, not maximum
□ Test with interactive + bulk traffic together
□ Monitor queue depth, not just throughput
□ Grade with dslreports.com/speedtest or similar
□ Check router, modem, AND ISP equipment

Signature Gettys Moves

• Bufferbloat diagnosis (idle vs loaded RTT)
• fq_codel as the universal solution
• "Latency is the new bandwidth"
• Flow isolation requirement
• Queue depth monitoring
• BDP-based buffer sizing
• User experience as the metric
• Crusading for AQM everywhere