# Container Storage Vulnerabilities: Understanding Residual Data Risks in Multi-Tenant Environments
## Introduction
A significant security finding was recently disclosed involving container-based serverless computing platforms. A researcher identified a flaw that could theoretically allow one customer to access residual data left behind by previous container workloads sharing the same physical infrastructure. The vulnerability was responsibly reported and has since been fully addressed by the platform provider, with no evidence of malicious exploitation or customer data compromise.
This article explores the technical details of how the vulnerability worked, why it matters for multi-tenant computing environments, and what steps were taken to resolve it.
## How Thin Provisioning Works in Container Storage
Modern container platforms often rely on a storage technique called thin provisioning to efficiently manage disk space. Rather than allocating a fixed amount of physical storage upfront for every container, thin provisioning assigns physical blocks only when data is actually written to a new region of the virtual disk.
In the affected environment, each container ran inside its own lightweight virtual machine. The platform used a device-mapper thin provisioning module with a block size of 64 KiB. When a container’s virtual disk was deleted, the physical blocks it had occupied were returned to a shared storage pool serving workloads from multiple customer accounts.
Under normal circumstances, when a thin volume is deleted, the underlying storage system zeros out the blocks before returning them to the pool. However, a specific configuration option in the storage pool was set to skip this zeroing step. This meant that when blocks were reassigned to a new container, they could still contain remnants of data written by the previous owner.
## The Core Issue: Unzeroed Block Reallocation
The problem surfaced from how small writes interacted with the unzeroed block allocation process. When a new container wrote a small amount of data — say, 4 KiB — into a previously unmapped region, the storage system would allocate an entire 64 KiB physical block. The 4 KiB write would overwrite only the portion where the new data was placed, while the remaining 60 KiB of the block retained whatever data had been there before.
A subsequent read of the raw disk device could therefore reveal bytes that the new container had never intentionally written. This created a window where residual data from a previous container’s workload could be observed by a different customer’s container running on the same physical host.
It is important to note that this was not a targeted exploit. An attacker could not choose which host to run on, which previous customer’s data to recover, or guarantee that any specific residual data would be present. The vulnerability was more a systemic issue of storage hygiene in shared infrastructure.
## The Proof of Concept and Validation
The researcher developed a controlled proof of concept to demonstrate the issue. The process involved creating a container, opening its writable root disk, and systematically writing small 4 KiB blocks into 64 KiB-aligned regions corresponding to free space in the container’s filesystem. By reading back those regions and examining the portions not overwritten by the new write, the researcher could recover data remnants from prior container occupants.
To validate that recovered blocks actually belonged to other customers’ filesystems, the researcher used filesystem-level checksums. The ext4 filesystem supports a metadata checksum feature that embeds filesystem- and inode-specific values into directory block checksums. By comparing these checksums against known values from the researcher’s own test filesystem, the team could reliably distinguish their own blocks from those belonging to other customers.
Across multiple production placements spanning several continents, the validation confirmed that residual material was present and recoverable. The recovered block types included directory structures, database pages, and even structurally complete SQLite databases. Importantly, the researcher reported only aggregate statistics and format validation results — no actual file contents, filenames, credentials, or other identifying information was included in the disclosure.
## What Was at Stake
If exploited maliciously, the vulnerability could have enabled cross-tenant data leakage in shared serverless infrastructure. A customer could potentially recover filesystem metadata, directory structures, database pages, and application data belonging to other customers whose containers had previously occupied the same physical storage blocks.
However, several constraints limited the practical risk. The exploitation could not target a specific victim or access an actively attached disk. It depended entirely on the platform’s workload scheduling and which blocks had been previously released and reused. Additionally, the technique only enabled reading residual data — it did not allow modification of another customer’s active workload or disruption of service availability.
## Mitigation and Remediation
The platform provider acted quickly once the vulnerability was confirmed. The remediation involved multiple coordinated steps:
**1. Removing the skip-zeroing configuration** — The `skip_block_zeroing` option was removed from the storage pool configuration across the entire fleet, restoring the default behavior of clearing blocks before making them available to new containers.
**2. Recreating all container disks and cached layers** — Simply zeroing new allocations was not sufficient, because blocks that had already been mapped into running container disks and cached image layers retained their existing mappings. New containers could inherit these mappings from cached layers without triggering fresh block allocation, leaving residual data accessible in unused regions.
**3. Draining and rebuilding affected hosts** — The provider drained hosts during off-peak hours, restarted all virtual machines, and cleared each host’s image cache. This ensured that all disks and cached layers were recreated using properly zeroed block allocations.
The researcher independently confirmed that the proof of concept no longer worked after the initial configuration change was applied, validating the effectiveness of the fix.
## Investigation Into Potential Exploitation
As part of the response, the platform provider examined historical disk I/O telemetry from its container infrastructure to determine whether any other activity matched the pattern of the reported exploit. The proof of concept produced a distinctive signature: a small 4 KiB write followed by a read that recovered substantially more data than what was written, reflecting the unzeroed remainder of a 64 KiB block.
Using these characteristics as detection signatures, the provider analyzed its retained telemetry. The investigation found activity attributable to the reporting researcher and internal engineers conducting authorized validation, but no additional activity consistent with the reported technique. There was no evidence of any malicious actor exploiting this vulnerability.
## Frequently Asked Questions
### What types of platforms are affected by this kind of vulnerability?
Any multi-tenant container or serverless computing platform that uses thin-provisioned shared storage pools without proper block sanitization during reallocation can be susceptible to this class of vulnerability. It is not limited to a single provider or technology stack.
### Could an attacker specifically target another customer’s data?
No. The vulnerability did not allow targeted retrieval. An attacker could not choose which host to run on or which previous customer’s data to recover. The exposure depended on random workload scheduling and which blocks happened to be reused.
### Was any customer data actually compromised?
No. The investigation found no evidence of malicious exploitation. The only activity consistent with the technique came from the authorized researcher and internal validation teams.
### Do customers need to take any action after the fix?
No. The platform provider applied the fix across the entire infrastructure. No customer-side configuration changes or actions were required.
### What kind of data could have been recovered?
If exploited, recoverable data could have included filesystem metadata, directory structures, database pages, and application data remnants — particularly from filesystems that had not been fully overwritten before the container was terminated.
### What lessons can other container platforms learn from this?
The key takeaway is that block-level sanitization is critical when storage is shared across tenants. Skipping zeroing operations for performance reasons introduces a real risk of cross-tenant data leakage. Platforms should also audit cached image layers and snapshot mappings to ensure inherited block allocations do not bypass sanitization.
## Conclusion
This vulnerability highlights an important tension in cloud infrastructure design: the balance between performance optimization and security isolation. Thin provisioning and deferred block zeroing are common techniques for improving storage efficiency in multi-tenant environments, but they introduce risks when not handled carefully. The responsible disclosure process, rapid remediation, and thorough investigation demonstrated by the platform provider serve as a strong model for how the industry should handle such findings.
The incident also underscores the value of bug bounty programs in identifying risks before they can be exploited. Researchers like Oren Yomtov play a crucial role in improving the security posture of widely used infrastructure platforms. As container-based computing continues to grow, continuous security scrutiny and transparent remediation processes will remain essential to maintaining trust in shared cloud environments.
Thank you for reading
