protocol-state-exploitation

name	protocol-state-exploitation
description	Protocol state exploitation targets vulnerabilities in network protocol state machines including SSH/TLS/HTTP2/DNS, covering illegal state transitions, stateful fuzzing, and protocol-level race conditions.
origin	openclaw
version	0.2.0.2
compatibility	["openclaw","claude-code","cursor","windsurf"]
allowed-tools	["Bash","Read","Write","Edit","WebSearch","WebFetch"]
metadata	{"domain":"exploitation","tool_count":12,"guide_count":4,"mitre":"TA0011-Command and Control","last_reviewed":"2026-07-26"}

Skill: Protocol State Exploitation

Supplementary Files:

• payloads.md -- Protocol state exploitation payloads: SSH illegal state transitions, TLS handshake reordering, HTTP/2 Rapid Reset, DNS KeyTrap, stateful fuzzing frameworks, Scapy packet crafting, state machine inference, timing attacks, CyberGym templates
• test-cases.md -- 6 structured test cases covering SSH KEX/USERAUTH exploitation, TLS handshake reordering, HTTP/2 stream violations, DNS KeyTrap, stateful fuzzing, protocol downgrade attacks

Summary

Protocol State Exploitation skill domain covering network protocol state machine vulnerabilities.

Tools: Wireshark, tshark, Scapy, Boofuzz, Sulley, AFLNet, StateAFL, pwntools, openssl s_client, nmap, tcpdump, hping3

Domain: exploitation

MITRE ATT&CK: TA0011-Command and Control

Description

Protocol state exploitation targets vulnerabilities in the state machines that govern network protocol implementations. Unlike traditional memory corruption or logic bugs, protocol state vulnerabilities arise from improper handling of state transitions, illegal message sequences, or race conditions in stateful protocol logic.

Network protocols like SSH, TLS, HTTP/2, and DNS rely on complex finite state machines (FSMs) to manage connection lifecycle, authentication sequences, and data transfer. Implementations often fail to properly validate state transitions, allowing attackers to inject messages at illegal states, trigger race conditions between state updates, or exhaust resources through rapid state transitions.

Key Vulnerability Classes:

• Illegal State Transitions: Sending protocol messages out of order or at invalid states (e.g., SSH USERAUTH before KEXINIT completion, TLS Finished before ClientKeyExchange)
• State Machine Confusion: Exploiting ambiguous or underspecified state definitions to bypass authentication or access control (e.g., HTTP/2 stream state violations)
• Stateful Resource Exhaustion: Rapidly creating and destroying protocol state objects to exhaust memory or CPU (e.g., HTTP/2 Rapid Reset CVE-2023-44487, DNS KeyTrap CVE-2023-50387)
• Protocol Downgrade Attacks: Forcing fallback to weaker protocol versions by manipulating negotiation state
• Race Conditions in State Updates: Exploiting timing windows between state checks and state transitions in multi-threaded implementations
• State Inference Attacks: Using timing side-channels to infer internal protocol state and predict behavior

High-Impact Examples:

• CVE-2024-6387 (regreSSHion): OpenSSH signal handler race during LoginGraceTime state transition enabling RCE
• CVE-2023-44487 (HTTP/2 Rapid Reset): Stream state exhaustion through rapid RST_STREAM causing DoS across all major HTTP/2 implementations
• CVE-2023-50387 (KeyTrap): DNSSEC state machine complexity attack causing CPU exhaustion
• CVE-2014-0224 (OpenSSL CCS Injection): TLS ChangeCipherSpec message injected before key exchange completion bypassing encryption

---

Use Cases

1. SSH Protocol State Exploitation -- Exploit illegal state transitions in SSH KEX/USERAUTH sequences to bypass authentication or achieve RCE (CVE-2024-6387 pattern)
2. TLS/SSL Handshake Reordering -- Inject ChangeCipherSpec or Finished messages at illegal states to bypass encryption or authentication (CVE-2014-0224 pattern)
3. HTTP/2 Rapid Reset DoS -- Exhaust server resources through rapid stream creation and RST_STREAM state transitions (CVE-2023-44487)
4. DNS KeyTrap Exploitation -- Trigger DNSSEC validation complexity attacks through crafted DNSKEY/RRSIG records (CVE-2023-50387)
5. Protocol Downgrade Attacks -- Force TLS version downgrade or cipher suite weakening through state machine manipulation
6. Stateful Fuzzing for 0-days -- Use AFLNet/StateAFL to discover state-dependent bugs in custom protocol implementations
7. HPACK Header Compression Attacks -- Exploit HTTP/2 dynamic table state to cause memory exhaustion or desync attacks
8. State Machine Inference -- Reverse-engineer protocol state machines to identify illegal transitions for targeted exploitation
9. Timing Attacks on State Transitions -- Use precision timing measurement to infer secret state (e.g., session keys, authentication status)
10. Multi-Connection State Confusion -- Race multiple connections against shared state to trigger synchronization bugs

---

Core Tools

Tool	Purpose	Command Example
Wireshark	Packet capture and protocol dissection with state analysis	wireshark -k -i eth0 -Y "ssh or tls or http2" -w capture.pcap
tshark	Command-line packet analysis for automated state extraction	tshark -r capture.pcap -Y "ssh.protocol" -T fields -e ssh.message_code
Scapy	Python packet manipulation framework for crafting illegal state transitions	scapy then send(IP(dst="target")/TCP(dport=22)/SSH_KEX())
Boofuzz	Stateful network protocol fuzzer with session management	boofuzz-gui or programmatic via Python API
Sulley	Legacy stateful fuzzer with protocol state tracking	python sulley/process_monitor.py -c target -p 80
AFLNet	State-guided greybox fuzzer for network protocols	afl-network-server -i in -o out -N tcp://127.0.0.1/8022 -P SSH -D 10000 -q 3 -s 3 -E -K ./sshd
StateAFL	AFL++ extension with state machine awareness	stateafl-fuzz -i in -o out -S state1 -- ./target @@
pwntools	Exploit framework with protocol interaction helpers	python3 -c "from pwn import *; r = remote('target', 22); r.send(b'SSH-2.0-Evil')"
openssl s_client	TLS client for handshake manipulation and testing	openssl s_client -connect target:443 -state -debug
nmap	Network scanner with protocol state probing scripts	nmap --script ssh2-enum-algos,tls-nextprotoneg target
tcpdump	Low-level packet capture for state transition timing analysis	tcpdump -i eth0 -nn -X 'tcp port 22'
hping3	Custom TCP/IP packet crafting with timing control	hping3 -S target -p 443 -c 10 --fast

---

Methodology

Attack Chain

[1] Reconnaissance         [2] State Mapping          [3] Vulnerability ID
  - Capture protocol         - Build FSM model           - Test illegal transitions
    traffic with Wireshark     with Scapy/tshark         - Inject out-of-order msgs
  - Identify protocol        - Identify state            - Fuzz state boundaries
    version and features       variables and guards        with Boofuzz/AFLNet
  - Enumerate supported      - Document legal            - Time state transitions
    algorithms/ciphers         transition sequences        for race conditions
                              
[4] Exploit Development    [5] Amplification          [6] Persistence/Impact
  - Craft PoC with           - Optimize timing with      - Chain with other vulns
    Scapy/pwntools             hping3/parallel conns     - Establish persistence
  - Trigger state            - Amplify resource          - Pivot to internal nets
    confusion or resource      exhaustion (HTTP/2         - Exfiltrate data via
    exhaustion                 Rapid Reset pattern)        covert channels

Practical Steps

1. Protocol Traffic Capture and Analysis

Start by capturing legitimate protocol interactions to understand normal state flow:

# Capture SSH handshake
tcpdump -i eth0 -nn -X 'tcp port 22' -w ssh_capture.pcap

# Analyze with tshark
tshark -r ssh_capture.pcap -Y "ssh" -T fields -e ssh.message_code -e ssh.protocol

# Dissect TLS handshake
tshark -r tls_capture.pcap -Y "ssl.handshake" -V | grep "Handshake Protocol"

# Extract HTTP/2 frames and stream IDs
tshark -r http2_capture.pcap -Y "http2" -T fields -e http2.type -e http2.streamid

2. State Machine Reconstruction

Build a state transition diagram from captured traffic:

# Scapy script to extract SSH state transitions
from scapy.all import *

def analyze_ssh_states(pcap_file):
    packets = rdpcap(pcap_file)
    states = []
    
    for pkt in packets:
        if TCP in pkt and pkt[TCP].dport == 22:
            if Raw in pkt:
                payload = pkt[Raw].load
                # SSH message codes: 20=KEXINIT, 21=NEWKEYS, 50=USERAUTH_REQUEST
                if len(payload) > 5:
                    msg_code = payload[5]
                    states.append(f"Client -> Server: {msg_code}")
        elif TCP in pkt and pkt[TCP].sport == 22:
            if Raw in pkt:
                payload = pkt[Raw].load
                if len(payload) > 5:
                    msg_code = payload[5]
                    states.append(f"Server -> Client: {msg_code}")
    
    return states

# Run analysis
transitions = analyze_ssh_states("ssh_capture.pcap")
for t in transitions:
    print(t)

3. Illegal State Transition Testing

Test protocol implementation by sending messages out of order. This section describes general methodology rather than executable code.

SSH State Transition Violations:

Test illegal state sequences such as:

• USERAUTH before NEWKEYS completion (skipping key exchange)
• NEWKEYS message sent twice (double-send state confusion)
• Message codes sent out of documented order

Use tools like:

• openssl s_client with -state -debug flags to monitor state transitions
• Wireshark with SSH filters to capture and analyze state machine progression
• Scapy for packet crafting (high-level framework, not specific payload code)

Expected indicators of vulnerable implementations:

• Server crashes or disconnects unexpectedly
• Encryption context errors
• State machine confusion responses

TLS Handshake Reordering (CVE-2014-0224 pattern):

Test premature state transitions in TLS, such as:

• ChangeCipherSpec (CCS) before ClientKeyExchange completion
• Finished message without key exchange
• Certificate message out of sequence

Tools for testing:

• openssl s_client -connect target:443 -state -msg to monitor state changes
• Monitor response codes for protocol violations
• Check for encryption bypass indicators

Use Scapy or similar frameworks to craft custom protocol sequences, but focus on testing methodologies rather than specific payload implementations.

4. Resource Exhaustion via State Transitions

HTTP/2 Rapid Reset (CVE-2023-44487):

HTTP/2 implements rapid stream creation and reset cycles that can exhaust server resources. Test this vulnerability by:

• Opening multiple concurrent HTTP/2 streams on a single connection
• Sending RST_STREAM frames immediately after stream creation
• Monitoring server CPU and memory usage

Tools:

• h2load - HTTP/2 load testing tool (use with appropriate rate limiting)
• Wireshark to monitor RST_STREAM frame frequency
• System monitoring tools to track resource exhaustion patterns

Expected indicators:

• Server CPU spike under stream creation load
• Memory growth proportional to stream count
• GOAWAY frames sent by server to terminate connection

DNS KeyTrap (CVE-2023-50387):

DNS DNSSEC validation complexity attack. Test by:

• Querying DNSSEC-enabled domain
• Observing resolver behavior with multiple DNSKEY/RRSIG records
• Measuring resolver CPU usage during validation

Tools:

• dig +dnssec to trigger DNSSEC validation
• tcpdump to capture DNS responses
• DNSSEC validation frameworks to understand complexity

Conceptual attack: Craft DNS responses with excessive DNSKEY records to trigger O(n^2) validation complexity in resolvers.

5. Stateful Fuzzing

Boofuzz for custom protocol fuzzing:

Boofuzz is a state-aware network protocol fuzzer. Use it to:

• Define protocol state machines declaratively
• Fuzz individual message fields at each state
• Test state transitions by connecting states in various sequences
• Monitor for crashes and protocol violations

Key features:

• Message definition with type/length fields
• State linking to test transitions
• Crash detection and logging
• Network target management

For SSH fuzzing specifically, define states for:

• Version exchange
• KEXINIT message with algorithm lists
• Key exchange messages
• NEWKEYS
• USERAUTH

Test invalid transitions by connecting states out of order (e.g., USERAUTH directly after VERSION).

AFLNet for network protocol fuzzing:

AFLNet is a state-guided fuzzer for network protocols:

• Requires protocol specification file (format specification)
• Takes network server as target
• Instruments target with AFL coverage feedback
• Generates mutated inputs based on protocol structure

Workflow:

• Compile target server with AFL instrumentation
• Create valid seed inputs (protocol handshakes)
• Run AFLNet fuzzer with target service
• Monitor output directory for crashes
• Analyze crash traces for state machine bugs

See tool documentation for specific usage patterns.

6. Timing Analysis for State Inference

Measure timing differences to infer internal state behavior:

SSH Authentication Timing:

Test timing differences between:

• Valid vs. invalid usernames (authentication state check)
• Valid vs. invalid passwords (password validation timing)
• Different cipher negotiation states

Use tools:

• time command to measure connection latency
• strace to observe system calls during authentication
• Custom measurement scripts with time.perf_counter()

TLS Cipher Negotiation Timing:

Measure handshake duration for different cipher suites:

• Test supported vs. unsupported ciphers
• Measure negotiation delay per algorithm
• Identify timing patterns that reveal server state

Tools:

• openssl s_client with timing measurement
• Custom Python scripts using ssl module
• System-level timing via tcpdump and packet timestamps

Note: Timing attacks reveal internal behavior but require careful measurement in controlled network conditions.

---

Defense Perspective

Detection Strategies

1. Protocol State Anomaly Detection

Monitor for illegal state transitions in network traffic:

# Wireshark/tshark filters for state violations
# SSH: Detect USERAUTH before NEWKEYS
tshark -r capture.pcap -Y "ssh.message_code == 50" -T fields -e frame.number -e ssh.message_code | \
    while read frame code; do
        # Check if NEWKEYS (21) appeared before this USERAUTH
        tshark -r capture.pcap -Y "frame.number < $frame && ssh.message_code == 21" | grep -q . || \
            echo "Illegal USERAUTH at frame $frame (no prior NEWKEYS)"
    done

# HTTP/2: Detect rapid RST_STREAM (Rapid Reset signature)
tshark -r capture.pcap -Y "http2.type == 3" -T fields -e frame.time_relative -e http2.streamid | \
    awk '{stream[$2]++; if(stream[$2] > 10) print "Rapid Reset on stream", $2}'

2. Resource Exhaustion Monitoring

Track protocol state object counts:

# Monitor HTTP/2 stream count per connection
netstat -tnp | grep :443 | wc -l

# Track DNS query rate (KeyTrap indicator)
tcpdump -i any -nn port 53 -c 1000 | awk '/query/{print $1}' | uniq -c | sort -rn

# Monitor SSH LoginGraceTime connections (regreSSHion pattern)
ss -tn state time-wait | grep :22 | wc -l

3. State Machine Hardening

Implementation best practices:

// Explicit state validation before every transition
enum ssh_state {
    SSH_STATE_INIT,
    SSH_STATE_VERSION_EXCHANGED,
    SSH_STATE_KEXINIT_SENT,
    SSH_STATE_NEWKEYS_RECEIVED,
    SSH_STATE_AUTHENTICATED
};

int handle_userauth(struct ssh_session *session, const uint8_t *packet) {
    // ENFORCE: USERAUTH only allowed after NEWKEYS
    if (session->state != SSH_STATE_NEWKEYS_RECEIVED) {
        log_error("Illegal USERAUTH at state %d", session->state);
        return SSH_ERR_PROTOCOL_ERROR;
    }
    
    // Process USERAUTH
    // ...
    
    session->state = SSH_STATE_AUTHENTICATED;
    return SSH_OK;
}

Mitigation Strategies

1. State Transition Validation

Always validate current state before processing messages:

• Maintain explicit FSM state variable
• Use switch/case with exhaustive state coverage
• Log and abort on illegal transitions
• Never assume previous message was valid

2. Resource Limits

Cap resources consumed by state objects:

// HTTP/2: Limit concurrent streams per connection
#define MAX_CONCURRENT_STREAMS 100

if (conn->active_streams >= MAX_CONCURRENT_STREAMS) {
    send_goaway(conn, HTTP2_ERR_ENHANCE_YOUR_CALM);
    return;
}

// DNS: Limit DNSSEC validation complexity
#define MAX_DNSKEY_COUNT 10
#define MAX_VALIDATION_TIME_MS 100

if (response->ar_count > MAX_DNSKEY_COUNT) {
    log_warn("Excessive DNSKEY count: %d", response->ar_count);
    return SERVFAIL;
}

3. Timeout and Grace Period Hardening

Minimize attack windows in timed state transitions:

// SSH: Reduce LoginGraceTime to minimize regreSSHion window
LoginGraceTime 30  // Default 120s, reduce to 30s

// HTTP/2: Aggressive stream timeout
stream_timeout = 10;  // 10 seconds max per stream

// TLS: Enforce handshake timeout
SSL_CTX_set_timeout(ctx, 30);  // 30 second handshake limit

4. Atomic State Updates

Use atomic operations for state changes in multi-threaded environments:

#include <stdatomic.h>

atomic_int connection_state = ATOMIC_VAR_INIT(STATE_INIT);

// Atomic compare-and-swap for state transition
int expected = STATE_KEXINIT;
if (!atomic_compare_exchange_strong(&connection_state, &expected, STATE_NEWKEYS)) {
    // State was not KEXINIT, transition failed
    return ERR_INVALID_STATE;
}

5. Protocol Downgrade Prevention

Enforce minimum protocol versions:

# SSH: Disable SSH protocol 1
Protocol 2

# TLS: Enforce TLS 1.3 minimum
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1 -TLSv1.2

# HTTP: Enforce HTTP/2 only, disable fallback
Protocols h2

# DNS: Disable legacy recursion patterns
recursion no;

---

Detection Methods

Protocol State Machine Anomalies

• HTTP/2 rapid reset: HTTP/2 RST_STREAM flood; CVE-2023-44487 signature.
• SSH protocol manipulation: Out-of-order SSH messages; unexpected channel requests.
• TCP state manipulation: SYN flood; SYN+ACK anomalies; TCP state exhaustion.
• QUIC anomalies: Connection ID spoofing; migration abuse.

SIEM Detection Rules

• Splunk SPL: index=http2 sourcetype=h2 | where frame_type="RST_STREAM" | stats count by src | where count > 100
• Falco / Zeek: Custom HTTP/2 protocol parsers.
• F5 / Cloudflare DDoS: HTTP/2 rapid reset detection (CVE-2023-44487).

Defense Evasion Techniques

State Machine Fuzzing Stealth

• Single-shot exploitation: One malformed packet per session; below rate detection.
• Distributed sources: Spread malformed packets across many source IPs.
• Use legitimate-looking state transitions: Mimic normal protocol behavior; trigger flaw only in final state.

HTTP/2-Specific Stealth

• Slow RST_STREAM: Pace RST_STREAM below per-stream rate limit.
• Connection reuse: Reuse legitimate connection for attack; bypasses connection-counting.
• QUIC migration abuse: Use connection migration to appear as new client.

Advanced Techniques

State Machine Inference from Black Box

Reverse-engineer protocol FSM when source is unavailable using systematic testing:

Methodology:

1. Capture legitimate protocol interactions with Wireshark
2. Extract message sequences and responses
3. Test permutations of messages systematically
4. Classify responses as ACCEPTED/REJECTED/TIMEOUT
5. Build state transition map from accepted sequences

Tools:

• Wireshark for traffic capture and analysis
• tcpdump for low-level packet capture
• Scapy for manual sequence testing
• Custom scripts to organize results

Limitations: Black-box inference requires significant network interaction and may trigger rate limiting or intrusion detection. Use responsibly in authorized testing only.

Multi-Connection Race Conditions

Some protocol vulnerabilities emerge from race conditions between multiple concurrent connections sharing server state:

Concepts:

• Exploit signal handler races (e.g., SIGALRM during authentication)
• Test shared resource exhaustion (connection limits, memory pools)
• Trigger synchronization bugs in state management

Tools:

• GNU parallel or similar for concurrent connection launching
• stress-ng for resource load generation
• Monitoring tools to observe race window effects

Example Pattern: Launch many concurrent connections while varying timing of protocol messages to hit race windows between state validation and state transitions.

HPACK Dynamic Table Manipulation

HTTP/2 header compression state can be tested for side effects:

Concepts:

• Dynamic table entries persist across requests on same connection
• Large headers can fill dynamic table, affecting subsequent requests
• Test for memory exhaustion or state confusion

Tools:

• h2load with custom header patterns
• h2 Python library for controlled header injection
• Wireshark HPACK dissector for dynamic table analysis

Testing: Send progressively larger headers to monitor memory usage and server behavior under table saturation.

---

Tool-Specific Commands

Wireshark Display Filters

# SSH state transitions
ssh.message_code == 20  # KEXINIT
ssh.message_code == 21  # NEWKEYS
ssh.message_code == 50  # USERAUTH_REQUEST

# TLS handshake state
ssl.handshake.type == 1   # ClientHello
ssl.handshake.type == 2   # ServerHello
ssl.handshake.type == 11  # Certificate
ssl.handshake.type == 20  # Finished
ssl.record.content_type == 20  # ChangeCipherSpec

# HTTP/2 frame types
http2.type == 0   # DATA
http2.type == 1   # HEADERS
http2.type == 3   # RST_STREAM
http2.type == 4   # SETTINGS
http2.type == 5   # PUSH_PROMISE
http2.type == 7   # GOAWAY

# Detect rapid state changes
frame.time_delta < 0.001 && (http2.type == 3 || ssh.message_code)

Scapy Protocol Layers

from scapy.all import *
from scapy.layers.tls.all import *

# Load protocol-specific layers
load_layer("tls")
load_layer("http2")

# Craft custom SSH packet
ssh_version = Raw(b"SSH-2.0-Scapy\r\n")
ssh_kexinit = Raw(b"\x00\x00\x01\x14\x0a\x14" + b"\x00" * 270)

pkt = IP(dst="target")/TCP(dport=22)/ssh_version
send(pkt)

# Craft TLS with illegal state
tls_hello = TLSClientHello(version=0x0303, ciphers=[0xc02f, 0xc030])
tls_ccs = TLSChangeCipherSpec()  # Send prematurely
send(IP(dst="target")/TCP(dport=443)/TLS(msg=[tls_hello, tls_ccs]))

nmap NSE Scripts for State Probing

# SSH algorithm enumeration (reveals state machine capabilities)
nmap --script ssh2-enum-algos target -p 22

# TLS protocol version probing
nmap --script ssl-enum-ciphers target -p 443

# HTTP/2 capability detection
nmap --script http2-detect target -p 443

# Custom NSE for state transition testing
nmap --script custom-ssh-state-test.nse target -p 22

---

CyberGym Protocol Bug Templates

PR1: SSH KEXINIT Ordering Violation

Concept: Send SSH messages out of order to trigger state confusion.

Attack Pattern:

• Normal sequence: VERSION → KEXINIT → KEX_DH_INIT → NEWKEYS → USERAUTH
• Violation: Skip key exchange phases and send USERAUTH directly after VERSION

Expected Outcomes:

• Vulnerable servers may process unauthenticated requests
• Crash or state machine error
• Information disclosure about internal state

Detection:

• Monitor for SSH_MSG_USERAUTH before SSH_MSG_NEWKEYS
• Check for unexpected responses to incomplete handshakes
• Use protocol analyzers to verify state machine progression

PR2: TLS ChangeCipherSpec Injection

Concept: Inject CCS at illegal state (before key exchange completion).

Attack Pattern:

• Normal: ClientHello → ServerHello → Certificate → ClientKeyExchange → CCS → Finished
• Violation: ClientHello → ServerHello → CCS (skip key exchange) → Finished

Expected Outcomes:

• Premature encryption with zero-length master secret
• Plaintext data flows despite "encrypted" state
• Potential authentication bypass

Detection:

• Verify ClientKeyExchange precedes ChangeCipherSpec
• Check master secret derivation
• Monitor for handshake message sequence violations

PR3: HTTP/2 Stream State Violation

Concept: Rapid stream creation/reset cycles exhaust server resources.

Attack Pattern:

• Open HTTP/2 stream with HEADERS
• Immediately send RST_STREAM
• Repeat with different stream IDs rapidly
• Monitor for resource exhaustion (CPU/memory spike)

Expected Outcomes:

• Server CPU usage increases significantly
• Memory grows proportional to stream count
• GOAWAY frame sent when threshold exceeded
• Service denial from resource exhaustion

Detection:

• Monitor RST_STREAM frame frequency
• Track active stream count
• Alert on rapid state transitions per connection

PR4: DNS DNSSEC Complexity Attack

Concept: Craft DNS responses with excessive DNSKEY/RRSIG records.

Attack Pattern:

• Query DNSSEC-enabled domain
• Receive response with many DNSKEY records (>40)
• Each DNSKEY requires RRSIG validation
• O(n²) validation complexity triggers CPU exhaustion

Expected Outcomes:

• Resolver CPU spike during validation
• Slow response times for DNSSEC queries
• Potential recursive query amplification

Detection:

• Monitor DNSKEY record count in responses
• Track resolver CPU during DNSSEC validation
• Alert on unusual validation times

---

Real-World Case Studies

CVE-2024-6387: OpenSSH regreSSHion

Vulnerability: Race condition between LoginGraceTime alarm signal handler and main thread

State Machine Bug:

Normal flow:
  1. Client connects
  2. Server sets LoginGraceTime alarm (120s default)
  3. Client authenticates within 120s
  4. Alarm canceled, session proceeds

Attack flow:
  1. Client connects but never authenticates
  2. LoginGraceTime expires at 120s mark
  3. SIGALRM handler calls async-signal-unsafe functions (malloc/free)
  4. Race condition with main thread causes heap corruption
  5. Heap corruption leads to RCE

Exploitation:

• Open 10,000+ parallel connections
• Keep all connections alive for 118-120s
• At 120s mark, server triggers SIGALRM for all connections
• Signal handler race window amplified by parallel connections
• 1-5% success rate after 10,000 attempts

Fix: Use async-signal-safe functions only in signal handlers

CVE-2023-44487: HTTP/2 Rapid Reset

Vulnerability: Resource exhaustion via rapid stream state transitions

State Machine Bug:

HTTP/2 stream lifecycle:
  IDLE -> OPEN -> HALF_CLOSED -> CLOSED

Attack exploits rapid OPEN -> RST_STREAM -> CLOSED transitions:
  for stream_id in 1..infinity:
      send HEADERS (IDLE -> OPEN)
      send RST_STREAM immediately (OPEN -> CLOSED)
  
Server allocates resources for each OPEN but cleanup is async.
Rapid creation outpaces cleanup, exhausting memory/CPU.

Impact: All major HTTP/2 implementations affected (nginx, Apache, AWS, Cloudflare)

Fix: Rate limit stream creation per connection, enforce GOAWAY after threshold

CVE-2014-0224: OpenSSL CCS Injection

Vulnerability: Accepting ChangeCipherSpec at illegal state

State Machine Bug:

Normal TLS handshake state sequence:
  ClientHello -> ServerHello -> Certificate -> ServerKeyExchange -> 
  ServerHelloDone -> ClientKeyExchange -> ChangeCipherSpec -> Finished

Attack injects CCS before ClientKeyExchange:
  ClientHello -> ServerHello -> ... -> ChangeCipherSpec (ILLEGAL) -> Finished
  
Server prematurely switches to encrypted mode with zero-length master secret,
allowing plaintext injection.

Exploitation: MITM attacker injects CCS, downgrades encryption to NULL

Fix: Explicit state checks before processing CCS message

---

References and Further Reading

Standards and RFCs:

• RFC 4251-4254: SSH Protocol Architecture
• RFC 5246: TLS 1.2, RFC 8446: TLS 1.3
• RFC 7540: HTTP/2 Protocol
• RFC 4033-4035: DNSSEC

Research Papers:

• "StateAFL: Greybox Fuzzing for Stateful Network Protocols" (NDSS 2022)
• "ProFuzzBench: A Benchmark for Stateful Protocol Fuzzing" (ISSTA 2021)
• "Protocol State Fuzzing of TLS Implementations" (USENIX Security 2015)

CVE Analysis:

• CVE-2024-6387 Analysis: https://www.qualys.com/2024/07/01/cve-2024-6387/regresshion.txt
• CVE-2023-44487 (Rapid Reset): https://cloud.google.com/blog/products/identity-security/http2-rapid-reset-attack
• CVE-2023-50387 (KeyTrap): https://www.athene-center.de/keytrap

Tools:

• Boofuzz: https://github.com/jtpereyda/boofuzz
• AFLNet: https://github.com/aflnet/aflnet
• Scapy: https://scapy.net/
• Wireshark: https://www.wireshark.org/