Containerd, BuildKit and a Reflection about the Enduring Value of Docker Engine

Jun 29 2018

Two weeks ago was our eighth DockerCon in just four years. Our community of contributors, developers, IT users, enterprises and ecosystem partners has grown exponentially into the millions,  anchored on our founder Solomon Hykes’ simple premise of democratizing the use of the software container. Today as was from the beginning, Docker creates simple tooling and a universal packaging approach that bundles up all application dependencies inside the container.  Docker Engine enables applications to run anywhere consistently on any infrastructure, solving “dependency hell” for developers and operations teams, and eliminating the “it works on my laptop!” problem.

In the past 2 years, Docker Engine’s codebase has been refactored into several reusable components, the most important being containerd, the core container runtime, and BuildKit, the part of Docker Engine used to build images. In the contribute and collaborate track at DockerCon, Michael Crosby and Tonis Tiigli gave an update on these two projects (slides)

Docker platform internals from Docker, Inc.

containerd, the core container runtime in Docker Engine has been leveraged by millions of users and is run in production by tens of thousands of organizations. Eighteen months ago, Docker spun out containerd from Docker Engine; donated it to the Cloud Native Computing Foundation (CNCF) as a top-level project because of its strong alignment with Kubernetes, gRPC and Prometheus. Docker made its donation in collaboration and in concert with the five largest cloud providers: Alibaba, AWS, Google, IBM and Microsoft with the target of having a solution that would work on premises, on all clouds and across Linux, Windows and mainframes.

One goal of Docker’s donation was to foster further innovation in the container ecosystem by providing a core container runtime that could be leveraged by other container system vendors and orchestration projects such as Kubernetes, DC/OS etc.  An important design principle for containerd was to have first-class support for Kubernetes without being exclusively tethered to it, because anchoring a core container runtime to Kubernetes limits its use case to Kubernetes only, leaving aside many use cases for containers such as developer desktop, CI/CD, single node deployments, Edge and IoT.

containerd 1.1 announced in April implements Kubernetes Container Runtime Interface (CRI), so it can be used directly by Kubernetes, as well as Docker Engine. It implements namespaces so that clients from different container systems (e.g. Docker Engine and DC/OS) can leverage a single containerd instance on one system while being logically separated. This release allows  you to plug in any OCI-compliant runtime, such as Kata containers, gVisor etc.  And includes  many performance improvements such as significantly better pod start latency and cpu/memory usage of the CRI plugin. And for additional performance benefits it replaces graphdrivers with more efficient snapshotters, that BuildKit leverages for faster build. The Kubernetes community members working on CRI announced in late May the general availability of containerd 1.1 with Kubernetes.

Michael’s talk outlines the features in containerd 1.1 smart client: I/O redirection from the client side, containerd namespaces to leverage a single runtime instance with a logical isolation from multiple clients (Kubernetes, Docker Engine, other systems), and containers as types in Golang when using containerd Go client library (reminiscent of Brendan Burn’s Metaparticle project).

Tonis’ talk explains all the performance improvements brought by BuildKit, and the capabilities that it opens up because of it’s modular architecture, enabling open source developers who create new build systems using BuildKit directly to create new frontends (beyond Dockerfiles) and backends (beyond Docker images) for it.

BuildKit will be integrated in Docker 18.06 as an experimental feature. This new experimental feature will bring extensibility of the Dockerfile: developers can create their own extensions for Dockerfile parsing by using the new #syntax directive:  # syntax = registry/user/repo:tag

The container image they provide will know how to interpret a new syntax in a Dockerfile. One example Tonis showed is an extension that understands RUN –mount and allows you to mount a volume at build time, such as a maven repository cache, yielding up to 33x performance improvements for your builds. This was one of the most popular feature requests for Docker build.

containerd and BuildKit are both pushing the envelope in terms of what is possible with containers, at runtime and build time. The fact that they are now integrated in Docker Engine makes these innovations in component projects available to mainstream developers without having to change their workflow; whether they use them from a laptop in Docker Desktop, a production Kubernetes cluster, a mainframe where traditional applications are modernized with containers, or a Raspberry Pi device for IoT scenario. Regardless of which system they are using, developers and operators benefit from the application workflow portability that Docker Engine provides, being able to build and run containers using the same trusted codebase everywhere.

Of course there are many other open source projects in these two areas of running and building containers, catering to a specific niche: runtimes like Kata to provide VM type isolation for containers, or CRIO to run OCI containers in Kubernetes, notably in OpenShift on RHEL, builders like Kaniko to build images in Kubernetes, jib to build images of Java applications, or buildah. But with containerd and BuildKit integrated in Docker Engine, you get the next generation of runtime and builder components, with more performance and configurability, integrated in a portable application workflow devs and ops know well, usable for any type of use case (single server, orchestrated runtime, CI/CD, IoT,…).

In summary if you want to make sure that you have one core container runtime that works across all orchestrators (Kubernetes, Swarm, AWS ECS etc.), Linux distributions, Windows and mainframes; on-premises and  across any cloud and on x86, ARM and GPUs then the only solution is containerd. And if you want to realize further value with an optimized image building solution for containerd, BuildKit integrated in Docker Engine is the best solution.

Stay connected with Docker by taking advantage of the on-demand sessions available for viewing from DockerCon San Francisco 2018:

Learn More


0 thoughts on "Containerd, BuildKit and a Reflection about the Enduring Value of Docker Engine"

DockerCon 2022

Registration is now open for DockerCon 2022! Join us for this free, immersive online experience complete with product demos, breakout learning tracks, panel discussions, hacks & tips, deep dive technical sessions, and much more.

Register Now