**The HollowByte OpenSSL DoS Vulnerability: A Critical Memory Management Flaw**
In a recent disclosure that has caught the attention of the security community, a dangerous denial-of-service (DoS) vulnerability has been identified in the widely-used OpenSSL cryptographic library. Dubbed “HollowByte,” this flaw allows an attacker to force a server to consume massive amounts of memory, leading to resource exhaustion and system instability, all without triggering standard security defenses.
The core of the vulnerability lies in a subtle misstep in memory handling during the TLS handshake process. Every TLS message includes a header that specifies the size of the incoming data. Prior to the fix, OpenSSL would immediately attempt to allocate a buffer of the size declared in this header, before any of the actual message body had even arrived or been validated. For a ClientHello message, this allocated space could be as large as 131 KB.
The critical failure occurs when the attacker then drops the connection. In a standard, non-fragmented attack like the classic Slowloris, this memory would be freed and returned to the system. However, the story ends differently on systems using the glibc memory allocator. Glibc is designed to retain small and medium-sized memory chunks in a pool for future reuse rather than returning them to the kernel.
The HollowByte attack exploits this by constantly varying the declared size of the fake message. This prevents glibc’s allocator from ever recognizing a reusable pattern, leading to severe heap fragmentation. Consequently, the server’s resident set size—the amount of memory actively in use—climbs steadily and, crucially, **never recovers**, even after the attack itself has ceased. In internal testing, a 1GB server was pushed to the point of being OOM (Out-Of-Memory) killed, while a 16GB server had 25% of its total memory locked away in useless fragments.
Compounding the issue is the manner in which the fix was delivered. OpenSSL’s security team chose to classify the patch as a “bug or hardening” fix, a category that falls outside of their standard four-tiered severity system (Critical, High, Moderate, Low). Because of this classification, the fix received no Common Vulnerabilities and Exposures (CVE) identifier, no public advisory, and no entry in the official changelog for version 4.0.1.
This lack of a CVE has downstream consequences. Security tools and package managers, which often rely on CVE numbers to identify and apply patches, cannot automatically detect or remediate the vulnerability. Users may find that their systems report an “up-to-date” version of OpenSSL while remaining vulnerable. The onus is now on individual administrators to manually verify they are running one of the patched versions (OpenSSL 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21, all released on June 9) and restart their services to clear the fragmented memory.
### FAQ
**Q: What is the “HollowByte” vulnerability?**
A: HollowByte is a denial-of-service vulnerability in OpenSSL. It allows an attacker to send a specially crafted handshake message that tricks the server into allocating up to 131 KB of memory per connection. The attacker then immediately drops the connection. On systems using the glibc memory allocator, this memory is never returned to the system, leading to severe memory fragmentation and eventual server crash or freeze.
**Q: How does this attack differ from other denial-of-service attacks like Slowloris?**
A: While Slowloris also aims to exhaust server connections, its attack is largely negated by modern allocators that promptly return freed memory. HollowByte is more insidious because it specifically targets the memory fragmentation behavior of glibc, causing the “lost” memory to stay lost, rapidly consuming all available RAM.
**Q: Why hasn’t this been assigned a CVE number?**
A: The OpenSSL security team classified the fix as a “bug or hardening” change, which is not part of their standard severity rating system. As a result, no CVE was issued, making it difficult for automated security scanners and package managers to identify and patch the flaw.
**Q: Which versions of OpenSSL are vulnerable?**
A: Any version of OpenSSL prior to the following releases is vulnerable:
– OpenSSL 4.0.x before 4.0.1
– OpenSSL 3.6.x before 3.6.3
– OpenSSL 3.5.x before 3.5.7
– OpenSSL 3.4.x before 3.4.6
– OpenSSL 3.0.x before 3.0.21
**Q: Is DTLS (Datagram TLS) also affected?**
A: No. The patch explicitly did not address DTLS. The same vulnerable code path likely exists in DTLS, and the OpenSSL team has not committed to fixing it at this time.
**Q: What can I do to protect my server?**
A: If you are a user of OpenSSL, you must manually upgrade to one of the patched versions listed above. It is also essential to restart your web server or application service to clear any memory that may have already been fragmented by the attack. Manual verification is key, as package managers may not automatically apply the fix without a CVE identifier.
### Conclusion
The HollowByte vulnerability serves as a powerful reminder that security is not just about preventing malicious access, but also about ensuring system resilience. A flaw that does not cause a crash or reveal data can still be devastating by crippling a service through resource starvation. The unique interaction between OpenSSL’s memory allocation and glibc’s fragmentation behavior created a “perfect storm” that evaded traditional security monitoring. While a patch is now available, its unconventional delivery mechanism highlights the need for heightened vigilance and proactive manual checks, especially for long-term support (LTS) versions of critical infrastructure software that may not receive timely updates.
