Skip to main content

Leaky Vessels: Docker and runc container breakout vulnerabilities (January 2024)

Written by:
feature-leaky-vessels

January 31, 2024

0 mins read

Notes on Leaky Vessels

We will continue to update this blog with any key updates, including updates on the disclosure of any new related vulnerabilities. This blog includes links to detailed blogs on each of the disclosed vulnerabilities, as well as two open source tools to aid in exploit detection.

Snyk security researcher Rory McNamara, with the Snyk Security Labs team, identified four vulnerabilities — dubbed "Leaky Vessels" — in core container infrastructure components that allow container escapes. An attacker could use these container escapes to gain unauthorized access to the underlying host operating system from within the container. Once an attacker gains access to the underlying host operating system, they could potentially access whatever data was on the system, including sensitive data (credentials, customer info, etc.), and launch further attacks. Upon discovery and verification, the Security Labs team initiated the process for responsible disclosure of the vulnerabilities, starting by notifying Docker, who, after review, forwarded one of the vulnerabilities to the open source runc security group. The disclosure timeline is below

Because these vulnerabilities affect widely used low-level container engine components and container build tools, Snyk strongly recommends that users check for updates from container build and runtime vendors, including Docker, Kubernetes vendors, cloud container services, and open source communities. You should upgrade systems running container engines and container build tools as soon as fixes are released by your providers.

About the vulnerabilities

On January 31, 2024, the maintainers of runc, a CLI tool for spawning and running containers on Linux, announced a vulnerability (CVE-2024-21626) that allows for an order-of-operations container breakout centered around the WORKDIR command. Exploitation of this vulnerability can result in container escape to the underlying host operating system. This could occur by running a malicious image or by building a container image using a malicious Dockerfile or upstream image (i.e. when using FROM). The patched version, runc 1.1.12, was released on January 31, 2024, at around 3:00 PM EST, per the maintainers.

You can read more details about the vulnerability in this high-level blog, which outlines the runc vulnerability itself. In addition, Rory and the Snyk Labs team identified three other container escape vulnerabilities for a total of four vulnerabilities, listed below, with links to the corresponding CVEs and overview blogs:

Tools available from Snyk to help detect these vulnerabilities

These vulnerabilities affect underlying container infrastructure and build tools rather than container images. Snyk Container is designed to help developers eliminate vulnerabilities in their container images, and so these vulnerabilities are outside the scope of what Snyk's products are currently designed to evaluate. However, Snyk developed two open source tools that serve as reference implementations for detecting exploit attempts. Please note that these tools are not covered under Snyk Support, but rather as examples for the community. 

Runtime exploit detection (leaky-vessels-runtime-detector)

The new Helios team at Snyk has built a runtime detection tool for this vulnerability, which can be found at leaky-vessels-runtime-detector, released under the Apache-2.0 license. This standalone tool, released under the Apache-2.0 license, provides a reference implementation for detecting the vulnerabilities as they are executed. The tool ties eBPF hooks to kernel- and user-level functions and to a package detector. This allows them to report invocations of container build and running containers if they match any patterns that indicate a possible exploitation attempt. Note that not all Linux distributions or versions support eBPF, and it's unlikely that customers would be able to leverage it on cloud service providers.

Static container command detection (leaky-vessels-static-detector)

The second tool is a static analysis program, leaky-vessels-static-detector, also released under the Apache-2.0 license, that scans Dockerfiles and image layers to detect commands that appear to be trying to exploit the vulnerabilities. The tool provides JSON-format output that indicates if it has detected any questionable commands. It's important to note that each hit will need to be manually inspected to determine if they are indeed exploits as opposed to legitimate usage of container build commands.

We are releasing these two tools as open source to provide the community with reference implementations for detecting potential exploit attempts. The runtime tool is likely to provide a higher level of confidence in findings than the static tool. However, given the nature of the exploits and the build commands, both tools will likely have some false negative and false positive results. The community can use the tools as examples to create their own tools, or run the tools in their environments. It's important to note that these tools neither fix the vulnerabilities nor block their exploitation, however, the tools will help to identify risk areas. The most prudent path is for customers to update impacted container orchestration platforms as patches become available. 

Are there any active exploits?

The Snyk team has performed ad hoc checks of Dockerfiles from public registries based on the images we see being used most frequently. This is not exhaustive, but in our research, we did not find evidence suggesting that these vulnerabilities have been exploited. Snyk recommends that you continue monitoring your own environment and check your containers until patches are made available and deployed. Given the nature of the issues, Snyk has created two open source tools, outlined above, a dynamic tool to help demonstrate the detection of the actions from the vulnerabilities at runtime, and a static tool to scan images and Dockerfiles to serve as an indicator of potential exploit.

How to prepare for remediation

Remediation should be done at the infrastructure and code tooling level. Look for announcements or releases, if applicable, from the provider or vendor of your container build and orchestration systems.  You will likely need to update your Docker daemons and Kubernetes deployments, as well as any container build tools that you use in CI/CD pipelines, on build servers, and on your developers' workstations. It’s also important to screen existing containers using tools like the ones Snyk built to determine if your orchestration nodes or build infrastructure have already been impacted.

Here are some updates that we've collected from widely used tools and services:

Date

Entity

Information

31-Jan-2024

Maintainers of runc

Released 1.1.12 addressing relevant vulnerabilities

31-Jan-2024

Snyk

Released the reference implementations leaky-vessels-dynamic-detector and leaky-vessels-static-detector to the community to identify potentially questionable containers and images

31-Jan-2024

containerd

Released version 1.6.28

31-Jan-2024

Docker

Docker released buildkit 0.12.5 and moby 25.0.2 and 24.0.9

31-Jan-2024

GCP

Released an update to runc 1.1.12

31-Jan-2024

Ubuntu

Released an update to runc 1.1.12

31-Jan-2024

AWS

Released an update to runc 1.1.12

Anatomy of a disclosure

The timeline for vulnerabilities can be complex, especially when there are multiple entities involved. It's imperative that the disclosures are handled responsibly so bad actors don’t learn of them before fixes are readily available. 

In this case, the vulnerabilities were initially discovered two months ago. At that point, Snyk began the process for responsible disclosure. Here's a timeline of some key milestones in the process.

Timeframe

Item

Week of 20-Nov-2023

Rory McNamara initially discovered the vulnerabilities. He began the internal verification process and additional research to validate findings and build POC exploits.

11-Dec-2023

Initial disclosure sent to Docker with all vulnerabilities, and Docker ACKed the same day.

12-Dec-2023

Snyk received a request from Docker to forward the WORKDIR vulnerability to runc, as it was deemed their responsibility.

13-Dec-2023

Rory was added as a Github Security Advisory (GHSA) collaborator for the Arbitrary Delete and grpc Docker/Buildkit vulns (both initially opened 11-Dec-2023).

19-Dec-2023

Rory was added as GHSA collaborator to WORKDIR by runc (initially opened 11-Dec-2023).

20-Dec-2023

Rory was added as GHSA collaborator for the cache race vulnerability.

02-Jan-2024

runc CVE assigned (Github CNA).

17-Jan-2024

runc sends an announcement to their security mailing list including the patches & embargo date of 31-Jan-2024.

24-Jan-2024

Docker vulnerabilities CVEs assigned (GitHub CNA).

31-Jan-2024

All four "Leaky Vessels" vulnerabilities announced publicly.

31-Jan-2024

Runc released version 1.1.12 which fixes the vulnerabilities.

31-Jan-2024

Snyk released the reference implementations leaky-vessels-dynamic-detector and leaky-vessels-static-detector to the community to identify potentially questionable containers and images.

Each step along the way involves collaboration within and between organizations — not only commercial organizations, but often the teams that maintain the components, which are often made up of a cross-section of the community.

About Snyk's Security Labs team

Snyk's Security Labs team has helped to responsibly disclose over 3,200 vulnerabilities in key packages across a variety of ecosystems. We work closely with open source package maintainers in order to ensure all vulnerabilities are responsibly and efficiently addressed. Our security expertise is one of the reasons that Snyk is trusted by so many big names across the security industry. If you find what you think is a vulnerability and don't know how to proceed to responsibly disclose it, fill out this form and our teams can help.

Was Snyk impacted?

For information about how Snyk addresses vulnerabilities in our own environment, visit the Snyk Trust Portal.

Summary

We will update this blog as we learn more, and we'll be holding a webinar Tuesday, February 6 at 11 AM ET for Leaky Vessels Container Breakout Vulnerabilities - What You Need to Know. Snyk technical experts will provide an in-depth technical review of one of the Leaky Vessels vulnerabilities, what caused it, how it can be exploited, and, most importantly, how it can be mitigated through upgrades and monitoring.

We encourage you to reach out with any questions you have about the vulnerabilities. For the open source tools, create a GitHub ticket on the respective tools (dynamic-detector and static-detector) or reach out on the Snyk community Discord

For more information, about Leaky Vessels, check out:

This article is provided for informational purposes only. Snyk is not responsible for any errors or omissions, or for the results obtained from the use of this information.

Change log

  • January 31, 2024, 3 p.m. ET: Initial release

  • January 31, 2024, 6 p.m. ET: Added CVE URLs; added updates from AWS, containerd, GCP, Docker, and Ubuntu

  • Feb 7, 2024, 1 p.m. ET: YouTube video and deep dive links added to Summary section

  • Feb 8, 2024, 10 a.m. ET: Snyk Learn lesson added to Summary section