Protocol DDoS Attacks | NOC.org

What Are Protocol DDoS Attacks?

Protocol DDoS attacks are a category of distributed denial of service attack that exploits weaknesses in network protocols — primarily TCP, IP, and ICMP — to exhaust the resources of servers, firewalls, load balancers, and other network infrastructure. Unlike volumetric attacks that aim to saturate bandwidth, protocol attacks target the processing capacity of network equipment by abusing the mechanisms these protocols use to establish and manage connections.

Protocol attacks operate at Layers 3 and 4 of the OSI model and are measured in packets per second (pps) rather than bits per second. A protocol attack does not need massive bandwidth to be effective. A relatively modest stream of carefully crafted packets can exhaust the connection state tables of firewalls, deplete server memory, or crash network devices — making these attacks highly efficient from the attacker's perspective.

How Protocol Attacks Exploit TCP/IP

The TCP/IP protocol suite was designed in the 1970s and 1980s with a focus on reliability and interoperability, not security. Several of its core mechanisms have inherent weaknesses that protocol attacks exploit:

• TCP's stateful handshake: TCP requires a three-way handshake (SYN, SYN-ACK, ACK) before a connection is established. The server must allocate memory to track each half-open connection while waiting for the handshake to complete.
• IP fragmentation: IP allows large packets to be fragmented into smaller pieces for transmission and reassembled at the destination. The reassembly process requires memory and processing, and malformed fragments can crash vulnerable implementations.
• ICMP trust model: ICMP messages (ping, unreachable, redirect) are trusted by default. There is no authentication mechanism, so any device can send ICMP messages with spoofed source addresses.
• Broadcast addressing: IP broadcast addresses allow a single packet to be delivered to every host on a subnet, creating an amplification effect when combined with source address spoofing.

Common Protocol Attack Types

SYN Floods

SYN floods are the most common and well-known protocol attack. They exploit the TCP three-way handshake by sending a massive number of SYN (synchronize) packets to the target without ever completing the handshake.

Here is how a normal TCP handshake works:

1. The client sends a SYN packet to the server, requesting a connection.
2. The server responds with a SYN-ACK packet, acknowledging the request and reserving resources for the connection.
3. The client sends an ACK packet, completing the handshake. The connection is now established.

In a SYN flood, the attacker sends thousands of SYN packets per second, often with spoofed source IP addresses. The server responds to each with a SYN-ACK and allocates memory in its connection state table to track the half-open connection. The ACK never arrives (because the source IP is spoofed or the attacker deliberately ignores the SYN-ACK), so the connection sits in a half-open state until it times out — typically 60 to 75 seconds.

With enough SYN packets, the server's connection table fills completely. Once the table is full, the server cannot accept any new connections — including those from legitimate users. The server itself may still be running with low CPU and memory usage, but it is completely unreachable because it has no capacity for new TCP connections.

Ping of Death

The Ping of Death attack sends ICMP Echo Request packets that are larger than the maximum allowed IP packet size of 65,535 bytes. Because IP supports fragmentation, the oversized packet is broken into fragments that individually comply with size limits. However, when the target system attempts to reassemble the fragments, the resulting packet exceeds the maximum buffer size, causing a buffer overflow that can crash the system, freeze it, or force a reboot.

The original Ping of Death vulnerability was patched in most operating systems by the late 1990s. However, variants that exploit IP fragmentation handling continue to appear. In 2013, a Ping of Death variant affected certain versions of Windows. Modern systems are generally protected, but legacy devices, IoT equipment, and embedded systems may remain vulnerable.

Smurf Attacks

A Smurf attack combines IP address spoofing with ICMP broadcast addressing to create an amplification effect:

1. The attacker sends an ICMP Echo Request (ping) to a network's broadcast address with the source IP spoofed to be the victim's IP address.
2. Every host on the target network receives the broadcast ping and responds with an ICMP Echo Reply — directed at the victim's IP address.
3. If the broadcast network has 200 hosts, a single ping packet generates 200 reply packets directed at the victim.

The amplification factor equals the number of hosts on the broadcast network. Smurf attacks were devastating in the late 1990s and early 2000s but have largely been mitigated by modern network configurations that disable directed broadcast by default (Cisco routers have used no ip directed-broadcast since IOS 12.0). However, misconfigured networks still exist, and the Smurf attack concept remains relevant as a model for understanding amplification-based attacks.

ACK Floods

An ACK flood sends massive numbers of TCP ACK packets to a target. Stateful firewalls and load balancers must check each ACK against their connection state tables to determine whether it belongs to an established connection. When millions of ACK packets arrive that match no known connection, the firewall's processing capacity is consumed checking and discarding each one. This can cause the firewall itself to become a bottleneck, denying service to all traffic passing through it.

RST/FIN Floods

Similar to ACK floods, RST (reset) and FIN (finish) floods send TCP control packets that force stateful devices to perform connection table lookups for connections that do not exist. The processing overhead of handling millions of these packets per second can overwhelm firewalls and intrusion detection systems.

Protocol Attacks vs. Other DDoS Categories

Mitigation Strategies

SYN Cookies

SYN cookies are the most important defense against SYN floods. Instead of allocating memory for each incoming SYN packet, the server encodes the connection state information (source IP, port, timestamp, MSS) into the sequence number of the SYN-ACK response. No state is stored on the server. When the client responds with a valid ACK, the server can reconstruct the connection state from the acknowledgment number. This eliminates the half-open connection table entirely and makes SYN floods ineffective. SYN cookies are supported by Linux (enabled via net.ipv4.tcp_syncookies = 1), Windows, and most modern operating systems.

Stateful vs. Stateless Filtering

The choice between stateful and stateless filtering has significant implications for protocol attack resilience:

• Stateful firewalls track every connection and can make intelligent decisions about which packets to allow based on connection state. However, the state table itself becomes a target — a SYN flood that fills the state table effectively disables the firewall. Under DDoS conditions, stateful firewalls can become the weakest link in the infrastructure.
• Stateless filtering (ACLs on routers, iptables rules without connection tracking) evaluates each packet independently based on header fields alone. Stateless filters do not maintain connection tables, making them immune to state exhaustion attacks. However, they cannot make connection-aware decisions (such as allowing only established TCP connections).

The optimal approach is to use stateless filtering at the network edge for volumetric and protocol attack mitigation, with stateful inspection applied only after the traffic has been scrubbed.

Rate Limiting and Connection Limits

• Limit the number of SYN packets per second from any single IP address.
• Set maximum half-open connection limits per source IP.
• Reduce the SYN-ACK retry count and timeout to free resources faster when handshakes are not completed.
• Limit ICMP traffic rates at the network edge to mitigate ping floods and Smurf-style attacks.

Infrastructure Hardening

• Disable IP directed broadcast on all routers to prevent Smurf attacks.
• Deploy BCP38 ingress filtering to prevent spoofed packets from leaving your network.
• Increase connection table sizes on firewalls and load balancers to provide a larger buffer against state exhaustion.
• Use dedicated DDoS mitigation appliances in front of stateful firewalls to absorb protocol attacks before they reach the firewall's connection table.

Protect Your Network from Protocol Attacks

Protocol DDoS attacks exploit decades-old weaknesses in TCP/IP that cannot be fully patched without breaking backward compatibility. Defense requires a combination of SYN cookies, stateless edge filtering, properly sized infrastructure, and upstream DDoS mitigation. NOC.org's CDN and WAF services handle protocol-layer threats at the network edge, ensuring that attack traffic is neutralized before it reaches your servers. View NOC.org's pricing plans to find the right level of protection for your infrastructure.