Understanding the HollowByte Vulnerability in OpenSSL Servers
The HollowByte vulnerability poses a significant threat to OpenSSL servers, allowing potential Denial-of-Service (DoS) attacks with just eleven bytes of data. This flaw primarily affects the memory management capabilities of OpenSSL, raising concerns about the integrity of secure communications over the internet.
What is the HollowByte Vulnerability?
Documented by Okta’s Red Team, the HollowByte vulnerability enables unauthorized attackers to trigger DoS states on OpenSSL servers. In a conventional TLS handshake, a four-byte header within a three-digit length field specifies the size of the subsequent message. Unfortunately, vulnerable versions of OpenSSL reserve this memory space prematurely, even before the actual data is received or its size is verified. Okta’s researchers elucidate this flaw with a striking observation:
“The worker thread blocks indefinitely, waiting for data that will never arrive.”
— Okta
In practical terms, an attacker can exploit this vulnerability by sending an eleven-byte message with a header that falsely indicates a large volume of data. When executed across multiple connections, this minimal data transmission can lead to excessive memory allocation on the server.
Memory Fragmentation Due to Connection Waves
Although OpenSSL attempts to free up allocated buffers after a connection is terminated, issues arise due to the GNU C Library’s (glibc) memory management practices. This library does not immediately return small to medium memory releases to the operating system; instead, it retains them for potential reuse. Attackers strategically exploit this behavior through targeted patterns.
According to Okta, “By triggering connection waves with randomly specified sizes, an attacker prevents the allocator from reusing these freed blocks.” This exploitation leads to significant fragmentation of the heap, steadily increasing the server’s Resident Set Size (RSS). Even after an attacker disconnects, the server remains bloated, and the blocked memory can only be released by performing a complete process restart.
Affected Applications and Software Updates
The OpenSSL library serves as the backbone for secure internet communication. It is integrated into various software projects, including popular web servers such as NGINX and Apache, as well as runtime environments like Node.js, Python, Ruby, PHP, and databases like MySQL and PostgreSQL.
In test environments using NGINX, the HollowByte attack led to complete memory exhaustion on lower-capacity systems. Even high-performance servers experienced a decline in memory capacity by up to 25%, while the attack’s bandwidth usage typically remained beneath standard security monitoring thresholds.
Following the identification of this vulnerability, the OpenSSL team released a patch that addresses the issue without assigning a specific CVE identification number. The flaw has been resolved in OpenSSL version 4.0.1 and backported to earlier versions such as 3.6.3, 3.5.7, 3.4.6, and 3.0.21. The updated versions ignore header size specifications and expand the buffer only when actual data arrives.
Recommendations for OpenSSL Operators
In light of the HollowByte vulnerability, it is crucial for administrators and operators to promptly update their OpenSSL packages to the latest versions. Okta emphasizes the importance of this action to mitigate potential risks associated with DoS attacks.
Conclusion
The HollowByte vulnerability in OpenSSL servers exemplifies a critical issue within memory management, potentially leading to severe disruptions in service. With the severity of this threat, staying informed and ensuring timely updates are essential practices for maintaining a secure digital environment.
