The latest release series of BuildKit, v0.11, introduces support for build-time attestations and SBOMs, allowing publishers to create images with records of how the image was built. This makes it easier for you to answer common questions, like which packages are in the image, where the image was built from, and whether you can reproduce the same results locally.

This new data helps you make informed decisions about the security of the images you consume — without needing to do all the manual work yourself.

In this blog post, we’ll discuss what attestations and SBOMs are, how to build images that contain SBOMs, and how to start analyzing the resulting data!

What are attestations?

An attestation is a declaration that a statement is true. With software, an attestation is a record that specifies a statement about a software artifact. For example, it could include who built it and when, what inputs it was built with, what outputs it produced, etc.

By writing these attestations, and distributing them alongside the artifacts themselves, you can see these details that might otherwise be tricky to find. To get this kind of information without attestations, you’d have to try and reverse-engineer how the image was built by trying to locate the source code and even attempting to reproduce the build yourself.

To provide this valuable information to the end-users of your images, BuildKit v0.11 lets you build these attestations as part of your normal build process. All it takes is adding a few options to your build step.

BuildKit supports attestations in the in-toto format (from the in-toto framework). Currently, the Dockerfile frontend produces two types of attestations that answer two different questions:

• SBOM (Software Bill of Materials) – An SBOM contains a list of software components inside an image. This will include the names of various packages installed, their version numbers, and any other associated metadata. You can use this to see, at a glance, if an image contains a specific package or determine if an image is vulnerable to specific CVEs.
• SLSA Provenance – The Provenance of the image describes details of the build process, such as what materials (like, images, URLs, files, etc.) were consumed, what build parameters were set, as well as source maps that allow mapping the resulting image back to the Dockerfile that created it. You can use this to analyze how an image was built, determine whether the sources consumed all appear legitimate, and even attempt to rebuild the image yourself.

Users can also define their own custom attestation types via a custom BuildKit frontend. In this post, we’ll focus on SBOMs and how to use them with Dockerfiles.

Getting the latest release

Building attestations into your images requires the latest releases of both Buildx and BuildKit – you can get the latest versions by updating Docker Desktop to the most recent version.

You can check your version number, and ensure it matches the buildx v0.10 release series:

$ docker buildx version
github.com/docker/buildx 0.10.0 ...

To use the latest release of BuildKit, create a docker-container builder using buildx:

$ docker buildx create --use --name=buildkit-container --driver=docker-container

You can check that the new builder is configured correctly, and ensure it matches the buildkit v0.11 release series:

$ docker buildx inspect | grep -i buildkit
Buildkit:  v0.11.1

If you’re using the docker/setup-buildx-action in GitHub Actions, then you’ll get this all automatically without needing to update.

With that out of the way, you can move on to building an image containing SBOMs!

Adding SBOMs to your images

You’re now ready to generate an SBOM for your image!

Let’s start with the following Dockerfile to create an nginx web server:

# syntax=docker/dockerfile:1.5

FROM nginx:latest
COPY ./static /usr/share/nginx/html

You can build and push this image, along with its SBOM, in one step:

$ docker buildx build --sbom=true -t <myorg>/<myimage> --push .

That’s all you need! In your build output, you should spot a message about generating the SBOM:

...
=> [linux/amd64] generating sbom using docker.io/docker/buildkit-syft-scanner:stable-1                           	0.2s
...

BuildKit generates SBOMs using scanner plugins. By default, it uses buildkit-syft-scanner, a scanner built on top of Anchore’s Syft open-source project, to do the heavy lifting. If you like, you can use another scanner by specifying the generator= option. 

Here’s how you view the generated SBOM using buildx imagetools:

$ docker buildx imagetools inspect <myorg>/<myimage> --format "{{ json .SBOM.SPDX }}"
{
	"spdxVersion": "SPDX-2.3",
	"dataLicense": "CC0-1.0",
	"SPDXID": "SPDXRef-DOCUMENT",
	"name": "/run/src/core/sbom",
	"documentNamespace": "https://anchore.com/syft/dir/run/src/core/sbom-a589a536-b5fb-49e8-9120-6a12ce988b67",
	"creationInfo": {
	"licenseListVersion": "3.18",
	"creators": [
	"Organization: Anchore, Inc",
	"Tool: syft-v0.65.0",
	"Tool: buildkit-v0.11.0"
	],
	"created": "2023-01-05T16:13:17.47415867Z"
	},
	...

SBOMs also work with the local and tar exporters. When you export with these exporters, instead of attaching the attestations directly to the output image, the attestations are exported as separate files into the output filesystem:

$ docker buildx build --sbom=true -o ./image .
$ ls -lh ./image
-rw-------  1 user user 6.5M Jan 17 14:36 sbom.spdx.json
...

Viewing the SBOM in this case is as simple as cat-ing the result:

$ cat ./image/sbom.spdx.json | jq .predicate
{
	"spdxVersion": "SPDX-2.3",
	"dataLicense": "CC0-1.0",
	"SPDXID": "SPDXRef-DOCUMENT",
	…

Supplementing SBOMs

Generating SBOMs using a scanner is a good first start! But some packages won’t be correctly detected because they’ve been installed in a slightly unconventional way.

If that’s the case, you can still get this information into your SBOMs with a bit of manual interaction.

Let’s suppose you’ve installed foo v1.2.3 into your image by downloading it using curl:

RUN curl https://example.com/releases/foo-v1.2.3-amd64.tar.gz | tar xzf - && \
    mv foo /usr/local/bin/

Software installed this way likely won’t appear in your SBOM unless the SBOM generator you’re using has special support for this binary (for example, Syft has support for detecting certain known binaries).

You can manually generate an SBOM for this piece of software by writing an SPDX snippet to a location of your choice on the image filesystem using a Dockerfile heredoc:

COPY /usr/local/share/sbom/foo.spdx.json <<"EOT"
{
	"spdxVersion": "SPDX-2.3",
	"SPDXID": "SPDXRef-DOCUMENT",
	"name": "foo-v1.2.3",
	...
}
EOT

This SBOM should then be picked up by your SBOM generator and included in the final SBOM for the whole image. This behavior is included out-of-the-box in buildkit-syft-scanner, but may not be part of every generator’s toolkit.

Even more SBOMs!

While the above section is good for scanning a basic image, it might struggle to provide more detailed package and file information. BuildKit can help you scan additional components of your build, including intermediate stages and your build context using the BUILDKIT_SBOM_SCAN_STAGE and BUILDKIT_SBOM_SCAN_CONTEXT arguments respectively.

In the case of multi-stage builds, this allows you to track dependencies from previous stages, even though that software might not appear in your final image.

For example, for a demo C/C++ program, you might have the following Dockerfile:

# syntax=docker/dockerfile:1.5

FROM ubuntu:22.04 AS build
ARG BUILDKIT_SBOM_SCAN_STAGE=true
RUN apt-get update && apt-get install -y git build-essential
WORKDIR /src
RUN git clone https://example.com/myorg/myrepo.git .
RUN make build

FROM scratch
COPY --from=build /src/build/ /

If you just scanned the resulting image, it wouldn’t reveal that the build tools, like Git or GCC (included in the build-essential package), were ever used in the build process! By integrating SBOM scanning into your build using the BUILDKIT_SBOM_SCAN_STAGE build argument, you can get much richer information that would otherwise have been completely lost.

You can access these additionally generated SBOM documents in imagetools as well:

$ docker buildx imagetools inspect <myorg>/<myimage> --format "{{ range .SBOM.AdditionalSPDXs }}{{ json . }}{{ end }}"
{
	"spdxVersion": "SPDX-2.3",
	"SPDXID": "SPDXRef-DOCUMENT",
	...
}
{
	"spdxVersion": "SPDX-2.3",
	"SPDXID": "SPDXRef-DOCUMENT",
	...
}
...

For the local and tar exporters, these will appear as separate files in your output directory:

$ docker buildx build --sbom=true -o ./image .
$ ls -lh ./image
-rw------- 1 user user 4.3M Jan 17 14:40 sbom-build.spdx.json
-rw------- 1 user user  877 Jan 17 14:40 sbom.spdx.json
...

Analyzing images

Now that you’re publishing images containing SBOMs, it’s important to find a way to analyze them to take advantage of this additional data.

As mentioned above, you can extract the attached SBOM attestation using the imagetools subcommand:

$ docker buildx imagetools inspect <myorg>/<myimage> --format "{{json .SBOM.SPDX}}"
{
	"spdxVersion": "SPDX-2.3",
	"dataLicense": "CC0-1.0",
	"SPDXID": "SPDXRef-DOCUMENT",
	...

If your target image is built for multiple architectures using the --platform flag, then you’ll need a slightly different syntax to extract the SBOM attestation:

$ docker buildx imagetools inspect <myorg>/<myimage> --format "{{ json (index .SBOM "linux/amd64").SPDX}}"
{
	"spdxVersion": "SPDX-2.3",
	"dataLicense": "CC0-1.0",
	"SPDXID": "SPDXRef-DOCUMENT",
	...

Now suppose you want to list all of the packages, and their versions, inside an image. You can modify the value passed to the --format flag to be a go template that lists the packages:

$ docker buildx imagetools inspect <myorg>/<myimage> --format '{{ range .SBOM.SPDX.packages }}{{ println .name .versionInfo }}{{ end }}' | sort
adduser 3.118
apt 2.2.4
base-files 11.1+deb11u6
base-passwd 3.5.51
bash 5.1-2+deb11u1
bsdutils 1:2.36.1-8+deb11u1
ca-certificates 20210119
coreutils 8.32-4+b1
curl 7.74.0-1.3+deb11u3
...

Alternatively, you might want to get the version information for a piece of software that you know is installed:

$ docker buildx imagetools inspect <myorg>/<myimage> --format '{{ range .SBOM.SPDX.packages }}{{ if eq .name "nginx" }}{{ println .versionInfo }}{{ end }}{{ end }}'
1.23.3-1~bullseye

You can even take the whole SBOM and use it to scan for CVEs using a tool that can use SBOMs to search for CVEs (like Anchore’s Grype):

$ docker buildx imagetools inspect <myorg>/<myimage> --format '{{ json .SBOM.SPDX }}' | grype
NAME          	INSTALLED            	FIXED-IN 	TYPE  VULNERABILITY 	SEVERITY   
apt           	2.2.4                             	deb   CVE-2011-3374 	Negligible  
bash          	5.1-2+deb11u1        	(won't fix) deb   CVE-2022-3715 	 
...

These operations should complete super quickly! Because the SBOM was already generated at build, you’re just querying already-existing data from Docker Hub instead of needing to generate it from scratch every time.

Going further

In this post, we’ve only covered the absolute basics to getting started with BuildKit and SBOMs — you can find out more about the things we’ve talked about on docs.docker.com:

• Read more about build-time attestations
• Learn about how to use buildx to create SBOMs
• Implement your own SBOM scanner using the BuildKit SBOM protocol
• Dive into how attestations are stored in the registry

And you can find out more about other features released in the latest BuildKit v0.11 release here.

BuildKit v0.11 is now available, along with Buildx v0.10 and v1.5 of the Dockerfile syntax. We’ve released new features, bug fixes, performance improvements, and improved documentation for all of the Docker Build tools.
Let’s dive into what’s new! We’ll cover the highlights, but you can get the whole story in the full changelogs.

1. SLSA Provenance

BuildKit can now create SLSA Provenance attestation to trace the build back to source and make it easier to understand how a build was created. Images built with new versions of Buildx and BuildKit include metadata like links to source code, build timestamps, and the materials used during the build. To attach the new provenance, BuildKit now defaults to creating OCI-compliant images.

Although docker buildx will add a provenance attestation to all new images by default, you can also opt into more detail. These additional details include your Dockerfile source, source maps, and the intermediate representations used by BuildKit. You can enable all of these new provenance records using the new --provenance flag in Buildx:

$ docker buildx build --provenance=true -t <myorg>/<myimage> --push .

Or manually set the provenance generation mode to either min or max (read more about the different modes):

$ docker buildx build --provenance=mode=max -t <myorg>/<myimage> --push .

You can inspect the provenance of an image using the imagetools subcommand. For example, here’s what it looks like on the moby/buildkit image itself:

$ docker buildx imagetools inspect moby/buildkit:latest --format '{{ json .Provenance }}'
{
  "linux/amd64": {
    "SLSA": {
      "buildConfig": {

You can use this provenance to find key information about the build environment, such as the git repository it was built from:

$ docker buildx imagetools inspect moby/buildkit:latest --format '{{ json (index .Provenance "linux/amd64").SLSA.invocation.configSource }}'
{
  "digest": {
	"sha1": "830288a71f447b46ad44ad5f7bd45148ec450d44"
  },
  "entryPoint": "Dockerfile",
  "uri": "https://github.com/moby/buildkit.git#refs/tags/v0.11.0"
}

Or even the CI job that built it in GitHub actions:

$ docker buildx imagetools inspect moby/buildkit:latest --format '{{ (index .Provenance "linux/amd64").SLSA.builder.id }}'
https://github.com/moby/buildkit/actions/runs/3878249653

Read the documentation to learn more about SLSA Provenance attestations or to explore BuildKit’s SLSA fields.

2. Software Bill of Materials

While provenance attestations help to record how a build was completed, Software Bill of Materials (SBOMs) record what components are used. This is similar to tools like docker sbom, but, instead of requiring you to perform your own scans, the author of the image can build the results into the image.

You can enable built-in SBOMs with the new --sbom flag in Buildx:

$ docker buildx build --sbom=true -t <myorg>/<myimage> --push .

By default, BuildKit uses docker/buildkit-syft-scanner (powered by Anchore’s Syft project) to build an SBOM from the resulting image. But any scanner that follows the BuildKit SBOM scanning protocol can be used here:

$ docker buildx build --sbom=generator=<custom-scanner> -t <myorg>/<myimage> --push .

Similar to SLSA provenance, you can use imagetools to query SBOMs attached to images. For example, if you list all of the discovered dependencies used in moby/buildkit, you get this:

$ docker buildx imagetools inspect moby/buildkit:latest --format '{{ range (index .SBOM "linux/amd64").SPDX.packages }}{{ println .name }}{{ end }}'
github.com/Azure/azure-sdk-for-go/sdk/azcore
github.com/Azure/azure-sdk-for-go/sdk/azidentity
github.com/Azure/azure-sdk-for-go/sdk/internal
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
...

Read the SBOM attestations documentation to learn more.

3. SOURCE_DATE_EPOCH

Getting reproducible builds out of Dockerfiles has historically been quite tricky — a full reproducible build requires bit-for-bit accuracy that produces the exact same result each time. Even builds that are fully deterministic would get different timestamps between runs.

The new SOURCE_DATE_EPOCH build argument helps resolve this, following the standardized environment variable from the Reproducible Builds project. If the build argument is set or detected in the environment by Buildx, then BuildKit will set timestamps in the image config and layers to be the specified Unix timestamp. This helps you get perfect bit-for-bit reproducibility in your builds.

SOURCE_DATE_EPOCH is automatically detected by Buildx from the environment. To force all the timestamps in the image to the Unix epoch:

$ SOURCE_DATE_EPOCH=0 docker buildx build -t <myorg>/<myimage> .

Alternatively, to set it to the timestamp of the most recent commit:

$ SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) docker buildx build -t <myorg>/<myimage> .

Read the documentation to find out more about how BuildKit handles SOURCE_DATE_EPOCH. 

4. OCI image layouts as named contexts

BuildKit has been able to export OCI image layouts for a while now. As of v0.11, BuildKit can import those results again using named contexts. This makes it easier to build contexts entirely locally — without needing to push intermediate results to a registry.

For example, suppose you want to build your own custom intermediate image based on Alpine that contains some development tools:

$ docker buildx build . -f intermediate.Dockerfile --output type=oci,dest=./intermediate,tar=false

This builds the contents of intermediate.Dockerfile and exports it into an OCI image layout into the intermediate/ directory (using the new tar=false option for OCI exports). To use this intermediate result in a Dockerfile, refer to it using any name you like in the FROM statement in your main Dockerfile:

FROM base
RUN ... # use the development tools in the intermediate image

You can then connect this Dockerfile to your OCI layout using the new oci-layout:// URI schema for the --build-context flag:

$ docker buildx build . -t <myorg>/<myimage> --build-context base=oci-layout://intermediate

Instead of resolving the image base to Docker Hub, BuildKit will instead read it from oci-layout://intermediate in the current directory, so you don’t need to push the intermediate image to a remote registry to be able to use it.

Refer to the documentation to find out more about using oci-layout:// with the --build-context flag.

5. Cloud cache backends

To get good build performance when building in ephemeral environments, such as CI pipelines, you need to store the cache in a remote backend. The newest release of BuildKit supports two new storage backends: Amazon S3 and Azure Blob Storage.

When you build images, you can provide the details of your S3 bucket or Azure Blob store to automatically store your build cache to pull into future builds. This build cache means that even though your CI or local runners might be destroyed and recreated, you can still access your remote cache to get quick builds when nothing has changed.

To use the new backends, you can specify them using the --cache-to and --cache-from flags:

$ docker buildx build --push -t <user>/<image> \
  --cache-to type=s3,region=<region>,bucket=<bucket>,name=<cache-image>[,parameters...] \
  --cache-from type=s3,region=<region>,bucket=<bucket>,name=<cache-image> .

$ docker buildx build --push -t <registry>/<image> \
  --cache-to type=azblob,name=<cache-image>[,parameters...] \
  --cache-from type=azblob,name=<cache-image>[,parameters...] .

You also don’t have to choose between one cache backend or the other. BuildKit v0.11 supports multiple cache exports at a time so you can use as many as you’d like.

Find more information about the new S3 backend in the Amazon S3 cache and the Azure Blob Storage cache backend documentation. 

6. OCI Image annotations

OCI image annotations allow attaching metadata to container images at the manifest level. They’re an alternative to labels that are more generic, and they can be more easily attached to multi-platform images.

All BuildKit image exporters now allow setting annotations to the image exporters. To set the annotations of your choice, use the --output flag:

$ docker buildx build ... \
    --output "type=image,name=foo,annotation.org.opencontainers.image.title=Foo"

You can set annotations at any level of the output, for example, on the image index:

$ docker buildx build ... \
    --output "type=image,name=foo,annotation-index.org.opencontainers.image.title=Foo"

Or even set different annotations for each platform:

$ docker buildx build ... \
    --output "type=image,name=foo,annotation[linux/amd64].org.opencontainers.image.title=Foo,annotation[linux/arm64].org.opencontainers.image.title=Bar"

You can find out more about creating OCI annotations on BuildKit images in the documentation.

7. Build inspection with --print

If you are starting in a codebase with Dockerfiles, understanding how to use them can be tricky. Buildx supports the new --print flag to print details about a build. This flag can be used to get quick and easy information about required build arguments and secrets, and targets that you can build. 

For example, here’s how you get an outline of BuildKit’s Dockerfile:

$ BUILDX_EXPERIMENTAL=1 docker buildx build --print=outline https://github.com/moby/buildkit.git
TARGET:  	buildkit
DESCRIPTION: builds the buildkit container image

BUILD ARG              	   VALUE   DESCRIPTION
RUNC_VERSION           	   v1.1.4   
ALPINE_VERSION         	   3.17	 
BUILDKITD_TAGS                  	defines additional Go build tags for compiling buildkitd
BUILDKIT_SBOM_SCAN_STAGE   true 

We can also list all the different targets to build:

$ BUILDX_EXPERIMENTAL=1 docker buildx build --print=targets https://github.com/moby/buildkit.git
TARGET             	DESCRIPTION
alpine-amd64      	 
alpine-arm        	 
alpine-arm64      	 
alpine-s390x      	 

Any frontend that implements the BuildKit subrequests interface can be used with the buildx --print flag. They can even define their own print functions, and aren’t just limited to outline or targets.

The --print feature is still experimental, so the interface may change, and we may add new functionality over time. If you have feedback, please open an issue or discussion on the docker/buildx GitHub, we’d love to hear your thoughts!

8. Bake features

The Bake file format for orchestrating builds has also been improved.

Bake now supports more powerful variable interpolation, allowing you to use fields from the same or other blocks. This can reduce duplication and make your bake files easier to read:

target "foo" {
  dockerfile = target.foo.name + ".Dockerfile"
  tags       = [target.foo.name]
}

Bake also supports null values for build arguments and allows labels to use the defaults set in your Dockerfile so your bake definition doesn’t override those:

variable "GO_VERSION" {
  default = null
}
target "default" {
  args = {
    GO_VERSION = GO_VERSION
  }
}

Read the Bake documentation to learn more. 

More improvements and bug fixes 

In this post, we’ve only scratched the surface of the new features in the latest release. Along with all the above features, the latest releases include quality-of-life improvements and bug fixes. Read the full changelogs to learn more:

• BuildKit v0.11 changelog
• Buildx v0.10 changelog
• Dockerfile v1.5 changelog

We welcome bug reports and contributions, so if you find an issue in the releases, let us know by opening a GitHub issue or pull request, or get in contact in the #buildkit channel in the Docker Community Slack.

In this Community All-Hands session, Ákos Takács demonstrated how to solve many of these pesky problems and gain the superpower of fixing containers.

Each issue can impact your image builds and final applications. Some bugs may not trigger clear error messages. To further complicate things, source-code inspection isn’t always helpful. 

But, common container issues don’t have to be your kryptonite! We’ll share Ákos’ favorite tips and show you how to conquer these development challenges.

In this tutorial:

• Finding and fixing common container mistakes
  • Using the CLI for extra container visibility
    • Change your CLI output formatting for visibility and readability
  • Remember to leverage your logs
  • Tackle issues with ENTRYPOINT
  • Access and inspect container content
    • Dive deeply into files and folders
  • Solve Docker Build errors
  • Solve Docker Compose errors
  • Optional: Make direct file edits within running containers
  • Investigate less and develop more

Finding and fixing common container mistakes

Everyone is prone to the occasional silly mistake. You code when you’re tired, suffer from the occasional keyboard slip, or sometimes fail to copy text correctly between steps. These missteps can carry forward from one command to the next. And because easy-to-miss things like spelling errors or character omissions can fly under the radar, you’re left doing plenty of digging to solve basic problems. Nobody wants that, so what tools are at your disposal? 

Using the CLI for extra container visibility

Say we have an image downloaded from Docker Hub — any image at all — and use some variation of the docker run command to run it. The resulting container will be running the default command. If you want to surface that command, entering docker container ls --all will grab a list of containers with their respective commands. 

Users often copy these commands and reuse them within other longer CLI commands. As you’d expect, it’s incredibly easy to highlight incorrectly, copy an incomplete phrase, and run a faulty command that uses it.

While spinning up a new container, you’ll hit a snag. The runtime in this instance will fail since Docker cannot find the executable. It’s not located in the PATH, which indicates a problem:

Running the docker container ls --all command also offers some hints. Note the httpd-foregroun container command paired with its created (but not running) container. Conversely, the v0 container that’s running successfully leverages a valid, complete command:

How do we investigate further? Use the docker run --rm -it --name MYCONTAINER [IMAGE] bash command to open an interactive terminal within your container. Take the container’s default command and attempt to run it again. A “command not found” error message will appear.

This is much more succinct and shows that you’ve likely entered the wrong command — in this case by forgetting a character. While Ákos’ example uses httpd, it’s applicable to almost any container image. 

Change your CLI output formatting for visibility and readability

Container commands are clipped once they exceed a certain length in the terminal output. That prevents you from inspecting the command in its entirety. 

Luckily, Ákos showed how the --format ‘{{ json . }}’ | jq -C flag can improve how your terminal displays outputs. Instead of cutting off portions of text, here’s how your docker container ls --all result will look:

You can read and compare any parameters in full. Nothing is hidden. If you don’t have jq installed, you could instead enter the following command to display outputs similarly minus syntax highlighting. This beats the default tabular layout for troubleshooting:

docker container ls --all --format ‘{{ json . }}’ | python3 -m json.tool --json-lines

Lastly, why not just expand the original table view while only displaying relevant information? Run the following command with the --no-trunc flag to expand those table rows and completely reveal each cell’s contents:

docker container ls --all --format ‘table {{ .Names }}/t{{ .Status }}/t{{ .Command }}’ --no-trunc

These examples highlight the importance of visibility and transparency in troubleshooting. When you can uncover and easily digest the information you need, making corrections is much easier.      

Remember to leverage your logs

By following best practices, any active application running within a Docker container will produce log outputs. While you might view logging as a problem-catching mechanism, many running containers don’t experience issues.

Ákos believes it’s important to understand how normal log entries look. As a result, identifying abnormal log entries becomes that much easier. The docker logs command enables this:

The process of tuning your logs differs between tools and languages. For example, Ákos drew from methods involving httpd — like trace for detailed trace-level messages or LogLevel for filtering error messages — but these practices are widely applicable. You’ll probably want to zero in on startup and runtime errors to diagnose most issues. 

Log handling is configurable. Here are some common commands to help you drill down into container issues (and reduce noise):

Grab your container’s last 100 logs:

docker logs --tail 100 [container ID]

Grab all logs for a specific container:

docker logs [container ID]

View all active processes within a running container, should its logs be inaccessible:

docker top [container ID]

Log inspection enables easier remediation. Alongside Ákos, we agree that you should confirm any container changes or fixes after making them. This means you’ve taken the right steps and can move ahead.

Want to view all your logs together within Docker Desktop? Download our Logs Explorer extension, which lets you browse through your logs using filters and advanced search capabilities. You can even view new logs as they populate.

Tackle issues with ENTRYPOINT

When running applications, you’ll need to run executable files within your container. The ENTRYPOINT portion of your Dockerfile sets the main command within a container and basically assigns it a task. These ENTRYPOINT instructions rely on executable files being in the container. 

In Ákos’ example, he tackles a scenario where improper permissions can prevent Docker from successfully mounting and running an entrypoint.sh executable. You can copy his approach by doing the following: 

1. Use the ls -l $PWD/examples/v6/entrypoint.sh command to view your file’s permissions, which may be inadequate.
2. Confirm that permissions are incorrect. 
3. Run a chmod 774 command to let this file read, write, and execute for all users.
4. Use docker run to spin up a container v7 from the original entrypoint, which may work briefly but soon stop running. 
5. Inspect the entrypoint.sh file to confirm our desired command exists. 

We can confirm this again by entering docker container inspect v7-exiting to view our container definition and parameters. While the Entrypoint is specified, its Cmd definition is null. That’s what’s causing the issue:

Why does this happen? Many don’t know that by setting --entrypoint, any image with a default command will empty that command automatically. You’ll need to redefine your command for your container to work properly. Here’s how that CLI command might look:

docker run -d -v $PWD/examples/v7/entrypoint.sh:/entrypoint.sh --entrypoint /entrypoint.sh --name v7-running httpd:2.4 httpd-foreground

This works for any container image but we’re just drawing from an earlier example. If you run this and list your containers again, v7 will be active. Confirm within your logs that everything looks good. 

Access and inspect container content

Carefully managing files and system resources is critical during local development. That’s doubly true while working with multiple images, containers, or resource constraints. There are scenarios where your containers bloat as their contents accumulate over time. 

Keeping your files tidy is one thing. However, you may also want to copy your files from your container and move them into a temporary folder — using the docker cp command with a specified directory. Using a variation of ls -la ./var/v8, borrowing from Ákos’ example, then produces a list containing every file. 

This is great for visibility and confirming your container’s contents. And we can diagnose any issues one step further with docker container diff v8 to view which files have been changed, appended, or even deleted. If you’re experiencing strange container behavior, digging into these files might be useful. 

Note: You can also leverage our Resource Usage extension to monitor disk space consumption, network activity, CPU usage, and memory usage in real time!

Dive deeply into files and folders

Close inspection is where hexdump comes in handy. The hexdump function converts your file into hexadecimal code, which is much more readable than binary. Ákos used the following commands:

docker cp v8:/usr/local/apache2/bin/httpd ./var/v8-httpd`
`hexdump -C -n 100 ./var/v8-httpd

You can adjust this -n number to read additional or fewer initial bytes. If your file contains text, this content will stand out and reveal the file’s main purpose. But, say you want to access a folder. While changing your directory and running docker container inspect … is standard, this method doesn’t work for Docker Desktop users. Since Desktop runs things in a VM, the host cannot access the folders within. 

Ákos showcased CTO Justin Cormack’s own nsenter1 image on GitHub, which lets us tap into those containers running with Docker Desktop environments. Docker Captain Bret Fisher has since expanded upon nsenter1’s documentation while adding useful commands. With these pieces in place, run the following command:

docker run --rm --privileged --pid=host alpine:3.16.2 nsenter -t 1 -m -u -i -n -p -- sh -c “ cd \”$(docker container inspect v8 --format ‘{{ .GraphDriver.Data.UpperDir }}’}\” \&& find .”

This command’s output mirrors that from our earlier docker container diff command. You can also run a hexdump using that same image above, which gives you the same troubleshooting abilities regardless of your environment. You can also inspect your entrypoint.sh to make important changes.  

Solve Docker Build errors 

While Docker BuildKit is quick and resilient, you can encounter errors that prevent image build completion. To learn why, run the following command to view each sequential build stage:

docker build $PWD/[MY SOURCE] --tag “MY TAG” --progress plain

BuildKit will provide readable context for each step and display any errors that occur:

If you see a missing file or directory error like the one above, don’t worry! You can use the cat $PWD/[MY SOURCE]/[MY DOCKERFILE] command to view the contents of your Dockerfile. Not only can you see where you misstepped more clearly, but you can also add a new instruction before the failing command to list your folder’s contents. 

Maybe those contents need updating. Maybe your folder is empty! In that case, you need to update everything so docker build has something to leverage. 

Next, run the build command again with the --no-cache flag added. This flag tells Docker to cleanly build from scratch each time without relying on caching:

You can progressively build updated versions of your Dockerfile and test those changes, given the cascading nature of instructions. Writing new instructions after the last working instruction — or making changes earlier on in your file — can eliminate those pesky build issues. Mechanisms like unlink or cp are helpful. The first behaves like rm while accepting only one argument, while cp copies critical files and folders into your image from a source.  

Solve Docker Compose errors

We use Docker Compose to spin up multiple services simultaneously using the docker compose --project-directory $PWD/[MY SOURCE] up -d command. 

However, one or more of those containers might unexpectedly exit. By running docker compose --project-directory $PWD/[MY SOURCE] ps to list out our services, you can see which containers are running or exited.

To pinpoint the problem, you’d usually grab logs via the docker compose logs command. You won’t need to specify a project directory in most cases. However, your container produces no logs since it isn’t running. 

Next, run the cat $PWD/[MY SOURCE]/docker-compose.yml command to view your Docker Compose file’s contents. It’s likely that your services definitions need fixing, so digging line by line within the CLI is helpful. Enter the following command to make this output even clearer:

docker compose --project-directory $PWD/[MY SOURCE] config

Your container exits when the commands contained within are invalid — just like we saw earlier. You’ll be able to see if you’ve entered a command incorrectly or if that command is empty. From there, you can update your Compose file and re-run docker compose --project-directory $PWD/[MY SOURCE] up -d. You can now confirm that everything is working by listing your services again. Your terminal will also output logs! 

Optional: Make direct file edits within running containers

Finally, it’s possible (and tempting) to directly edit your files within your container. This is viable while testing new changes and inspecting your containers. However, it’s usually considered best practice to create a new image and container instead. 

If you want to make edits within running containers, an editor like VS Code allows this, while IntelliJ doesn’t by comparison. Install the Docker extension for VS Code. You can then browse through your containers in the left sidebar, expand your collection of resources, and directly access important files. For example, web developers can directly edit their index.html files to change how user content is structured. 

Investigate less and develop more

Overall, the process of fixing a container, on the surface, may seem daunting to newer Docker users. The methods we’ve highlighted above can dramatically reduce that troubleshooting complexity — saving you time and effort. You can spend less time investigating issues and more time creating the applications users love. And we think those skills are pretty heroic. 

For more information, you can view Ákos Takács’ full presentation on YouTube to carefully follow each step. Want to dive deeper? Check out these additional resources to become a Docker expert: 

• Learn how to view logs for a container or service
• View our Docker Run reference documentation
• Learn about best practices for writing Dockerfiles

With over 90,500 stars and 24,400 forks, Node’s developer community is highly active. With more devs creating Node.js apps than ever before, finding efficient ways to build and deploy and cross platform is key. Let’s discuss how containerization can help before jumping into the meat of our guide. 

Why is containerizing a Node application important?

Containerizing your Node application has numerous benefits. First, Docker’s friendly, CLI-based workflow lets any developer build, share, and run containerized Node applications. Second, developers can install their app from a single package and get it up and running in minutes. Third, Node developers can code and test locally while ensuring consistency from development to production.

We’ll show you how to quickly package your Node.js app into a container. We’ll also tackle key concerns that are easy to forget — like image vulnerabilities, image bloat, missing image tags, and poor build performance. Let’s explore a simple todo list app and discuss how our nine tips might apply.

Analyzing a simple todo list application

Let’s first consider a simple todo list application. This is a basic React application with a Node.js backend and a MongoDB database. The source code of the complete project is available within our GitHub samples repo.

Building the application

Luckily, we can build our sample application in just a few steps. First, you’ll want to clone the appropriate awesome-compose sample to use it with your project:

git clone https://github.com/dockersamples/awesome-compose/
cd awesome-compose/react-express-mongodb
docker compose -f docker-compose.yaml up -d

Second, enter the docker compose ps command to list out your services in the terminal. This confirms that everything is accounted for and working properly:

docker compose ps
NAME                COMMAND                  SERVICE             STATUS              PORTS
backend             "docker-entrypoint.s…"   backend             running             3000/tcp
frontend            "docker-entrypoint.s…"   frontend            running             0.0.0.0:3000->3000/tcp
mongo               "docker-entrypoint.s…"   mongo               running             27017/tcp

Third, open your browser and navigate to https://localhost:3000 to view your application in action. You’ll see your todo list UI and be able to directly interact with your application:

This is a great way to spin up a functional application in a short amount of time. However, remember that these samples are foundations you can build upon. They’re customizable to better suit your needs. And this can be important from a performance standpoint — since our above example isn’t fully optimized. Next, we’ll share some general optimization tips and more to help you build the best app possible. 

Our top nine tips for containerizing and optimizing Node applications

1) Use a specific base image tag instead of “version:latest”

When building Docker images, we always recommended specifying useful tags which codify version information, intended destination (prod or test, for instance), stability, or other useful information for deploying your application across environments.

Don’t rely on the latest tag that Docker automatically pulls, outside of local development. Using latest is unpredictable and may cause unexpected behavior. Each time you pull a latest image version, it could contain a new build or untested code that may break your application. 

Consider the following Dockerfile that uses the specific node:lts-buster Docker image as a base image instead of node:latest. This approach may be preferable since lts-buster is a stable image:

# Create image based on the official Node image from dockerhub
FROM node:lts-buster

# Create app directory
WORKDIR /usr/src/app

# Copy dependency definitions
COPY package.json ./package.json
COPY package-lock.json ./package-lock.json

# Install dependencies
#RUN npm set progress=false \
#    && npm config set depth 0 \
#    && npm i install
RUN npm ci

# Get all the code needed to run the app
COPY . .

# Expose the port the app runs in
EXPOSE 3000

# Serve the app
CMD ["npm", "start"]

Overall, it’s often best to avoid using FROM node:latest in your Dockerfile.

2) Use a multi-stage build

With multi-stage builds, a Docker build can use one base image for compilation, packaging, and unit testing. A separate image holds the application’s runtime. This makes the final image more secure and shrinks its footprint (since it doesn’t contain development or debugging tools). Multi-stage Docker builds help ensure your builds are 100% reproducible and lean. You can create multiple stages within a Dockerfile to control how you build that image.

You can containerize your Node application using a multi-layer approach. Each layer may contain different app components like source code, resources, and even snapshot dependencies. What if we want to package our application into its own image like we mentioned earlier? Check out the following Dockerfile to see how it’s done:

FROM node:lts-buster-slim AS development

WORKDIR /usr/src/app

COPY package.json ./package.json
COPY package-lock.json ./package-lock.json
RUN npm ci

COPY . .

EXPOSE 3000

CMD [ "npm", "run", "dev" ]

FROM development as dev-envs
RUN <<EOF
apt-get update
apt-get install -y --no-install-recommends git
EOF


# install Docker tools (cli, buildx, compose)
COPY --from=gloursdocker/docker / /
CMD [ "npm", "run", "dev" ]

We first add an AS development label to the node:lts-buster-slim statement. This lets us refer to this build stage in other build stages. Next, we add a new development stage labeled dev-envs. We’ll use this stage to run our development.

Now, let’s rebuild our image and run our development. We’ll use the same docker build command as above — while adding the --target development flag to specifically run the development build stage:

docker build -t node-docker --target dev-envs .

3) Fix security vulnerabilities in your Node image

Today’s developers rely on third-party code and apps while building their services. External software can introduce unwanted vulnerabilities into your code if you’re not careful. Leveraging trusted images and continually monitoring your containers helps protect you.

Whenever you build a node:lts-buster-slim Docker image, Docker Desktop prompts you to run security scans of the image to detect any known vulnerabilities.

Let’s use the the Snyk Extension for Docker Desktop to inspect our Node.js application. To begin, install Docker Desktop 4.8.0+ on your Mac, Windows, or Linux machine. Next, check the box within Settings > Extensions to Enable Docker Extensions.

You can then browse the Extensions Marketplace by clicking the “Add Extensions” button in the left sidebar, then searching for Snyk.

Snyk’s extension lets you rapidly scan both local and remote Docker images to detect vulnerabilities.

Install the Snyk and enter the node:lts-buster-slim Node Docker Official Image into the “Select image name” field. You’ll have to log into Docker Hub to start scanning. Don’t worry if you don’t have an account — it’s free and takes just a minute to create.

When running a scan, you’ll see this result within Docker Desktop:

Snyk uncovered 70 vulnerabilities of varying severity during this scan. Once you’re aware of these, you can begin remediation to fortify your image.

That’s not all. In order to perform a vulnerability check, you can use  the docker scan command directly against your Dockerfile:

docker scan -f Dockerfile node:lts-buster-slim

4) Leverage HEALTHCHECK

The HEALTHCHECK instruction tells Docker how to test a container and confirm that it’s still working. For example, this can detect when a web server is stuck in an infinite loop and cannot handle new connections — even though the server process is still running.

When an application reaches production, an orchestrator like Kubernetes or a service fabric will most likely manage it. By using HEALTHCHECK, you’re sharing the status of your containers with the orchestrator to enable configuration-based management tasks. Here’s an example:

# syntax=docker/dockerfile:1.4

FROM node:lts-buster-slim AS development

# Create app directory
WORKDIR /usr/src/app

COPY package.json ./package.json
COPY package-lock.json ./package-lock.json
RUN npm ci

COPY . .

EXPOSE 3000

CMD [ "npm", "run", "dev" ]

FROM development as dev-envs
RUN <<EOF
apt-get update
apt-get install -y --no-install-recommends git
EOF

RUN <<EOF
useradd -s /bin/bash -m vscode
groupadd docker
usermod -aG docker vscode
EOF

HEALTHCHECK CMD curl --fail http://localhost:3000 || exit 1  


# install Docker tools (cli, buildx, compose)
COPY --from=gloursdocker/docker / /
CMD [ "npm", "run", "dev" ]
```

When HEALTHCHECK is present in a Dockerfile, you’ll see the container’s health in the STATUS column after running the docker ps command. A container that passes this check is healthy. The CLI will label unhealthy containers as unhealthy:

docker ps
CONTAINER ID   IMAGE                            COMMAND                  CREATED          STATUS                             PORTS                    NAMES
1d0c5e3e7d6a   react-express-mongodb-frontend   "docker-entrypoint.s…"   23 seconds ago   Up 21 seconds (health: starting)   0.0.0.0:3000->3000/tcp   frontend
a89721d3c42d   react-express-mongodb-backend    "docker-entrypoint.s…"   23 seconds ago   Up 21 seconds (health: starting)   3000/tcp                 backend
194c953f5653   mongo:4.2.0                      "docker-entrypoint.s…"   3 minutes ago    Up 3 minutes                       27017/tcp                mongo

You can also define a healthcheck (note the case difference) within Docker Compose! This can be pretty useful when you’re not using a Dockerfile. Instead of writing a plain text instruction, you’ll write this configuration in YAML format. 

Here’s a sample configuration that lets you define healthcheck within your docker-compose.yml file:

backend:
    container_name: backend
    restart: always
    build: backend
    volumes:
      - ./backend:/usr/src/app
      - /usr/src/app/node_modules
    depends_on:
      - mongo
    networks:
      - express-mongo
      - react-express
    expose:
      - 3000
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000"]
      interval: 1m30s
      timeout: 10s
      retries: 3
      start_period: 40s

5) Use .dockerignore

To increase build performance, we recommend creating a .dockerignore file in the same directory as your Dockerfile. For this tutorial, your .dockerignore file should contain just one line:

node_modules

This line excludes the node_modules directory — which contains output from Maven — from the Docker build context. There are many good reasons to carefully structure a .dockerignore file, but this simple file is good enough for now.

Let’s now explain the build context and why it’s essential . The docker build command builds Docker images from a Dockerfile and a “context.” This context is the set of files located in your specified PATH or URL. The build process can reference any of these files. 

Meanwhile, the compilation context is where the developer works. It could be a folder on Mac, Windows, or a Linux directory. This directory contains all necessary application components like source code, configuration files, libraries, and plugins. With a .dockerignore file, we can determine which of the following elements like source code, configuration files, libraries, plugins, etc. to exclude while building your new image. 

Here’s how your .dockerignore file might look if you choose to exclude the node_modules directory from your build:

Backend:

Frontend:

6) Run as a non-root user for security purpose

Running applications with user privileges is safer since it helps mitigate risks. The same applies to Docker containers. By default, Docker containers and their running apps have root privileges. It’s therefore best to run Docker containers as non-root users. 

You can do this by adding USER instructions within your Dockerfile. The USER instruction sets the preferred user name (or UID) and optionally the user group (or GID) while running the image — and for any subsequent RUN, CMD, or ENTRYPOINT instructions:

# syntax=docker/dockerfile:1.4
FROM node:lts-buster AS development

WORKDIR /usr/src/app

COPY package.json ./package.json
COPY package-lock.json ./package-lock.json

RUN npm ci

COPY . .

EXPOSE 3000


CMD ["npm", "start"]

FROM development as dev-envs


RUN <<EOF
   apt-get update
   apt-get install -y --no-install-recommends git
EOF

RUN <<EOF
   useradd -s /bin/bash -m vscode
   groupadd docker
   usermod -aG docker vscode
EOF

USER vscode

# install Docker tools (cli, buildx, compose)
COPY --from=gloursdocker/docker / /
CMD [ "npm", "start" ]

7) Favor multi-architecture Docker images

Your CPU can only run binaries for its native architecture. For example, Docker images built for an x86 system can’t run on an Arm-based system. With Apple fully transitioning to their custom Arm-based silicon, it’s possible that your x86 (Intel or AMD) container image won’t work with Apple’s M-series chips. 

Consequently, we always recommended building multi-arch container images. Below is the mplatform/mquery Docker image that lets you query the multi-platform status of any public image in any public registry:

docker run --rm mplatform/mquery node:lts-buster
Unable to find image 'mplatform/mquery:latest' locally
d0989420b6f0: Download complete
af74e063fc6e: Download complete
3441ed415baf: Download complete
a0c6ee298a93: Download complete
894bcacb16df: Downloading [=============================================>     ]  3.146MB/3.452MB
Image: node:lts-buster (digest: sha256:a5d9200d3b8c17f0f3d7717034a9c215015b7aae70cb2a9d5e5dae7ff8aa6ca8)
 * Manifest List: Yes (Image type: application/vnd.docker.distribution.manifest.list.v2+json)
 * Supported platforms:
   - linux/amd64
   - linux/arm/v7
   - linux/arm64/v8

We introduced the docker buildx command to help you build multi-architecture images. Buildx is a Docker component that enables many powerful build features with a familiar Docker user experience. 
All Buildx builds run using the Moby BuildKit engine.

BuildKit is designed to excel at multi-platform builds, or those not just targeting the user’s local platform. When you invoke a build, you can set the --platform flag to specify the build output’s target platform (like linux/amd64, linux/arm/v7, linux/arm64/v8, etc.):

docker buildx build --platform linux/amd64,linux/arm/v7 -t node-docker .

8) Explore graceful shutdown options for Node

Docker containers are ephemeral in nature. They can be stopped and destroyed, then either rebuilt or replaced with minimal effort. You can terminate containers by sending a SIGTERM notice signal to the process. This little grace period requires you to ensure that your app is handling ongoing requests and cleaning up resources in a timely fashion. 

On the other hand, Node.js accepts and forwards signals like SIGINT and SIGTERM from the OS, which is key to properly shutting down your app. Node.js lets your app decide how to handle those signals. If you don’t write code or use a module to handle them, your app won’t shut down gracefully. It’ll ignore those signals until Docker or Kubernetes kills it after a timeout period. 

Using certain init options like docker run --init or tini within your Dockerfile is viable when you can’t change your app code. However, we recommend writing code to handle proper signal handling for graceful shutdowns.

Check out this video from Docker Captain Bret Fisher (12:57) where he covers all three available Node shutdown options in detail.

9) Use the OpenTelemetry API to measure NodeJS performance

How do Node developers make their apps faster and more performant? Generally, developers rely on third-party observability tools to measure application performance. This performance monitoring is essential for creating multi-functional Node applications with top notch user experiences.

Observability extends beyond application performance. Metrics, traces, and logs are now front and center. Metrics help developers to understand what’s wrong with the system, while traces help you discover how it’s wrong. Logs tell you why it’s wrong. Developers can dig into particular metrics and traces to holistically understand system behavior.

Observing Node applications means tracking your Node metrics, requests rates, request error rate, and request durations. OpenTelemetry is one popular collection of tools and APIs that help you instrument your Node.js application.

You can also use an open-source tool like SigNoz to analyze your app’s performance. Since SigNoz offers a full-stack observability tool, you don’t need to rely on multiple tools.

Conclusion

In this guide, we explored many ways to optimize your Docker images — from carefully crafting your Dockerfile to securing your image via Snyk scanning. Building better Node.js apps doesn’t have to be complex. By nailing some core fundamentals, you’ll be in great shape. 

If you’d like to dig deeper, check out these additional recommendations and best practices for building secure, production-grade Docker images:

• Docker development best practices
• Dockerfile best practices
• Building images with BuildKit
• Best practices for scanning images 
• Getting started with the Snyk Extension

Tons of developers use Docker containers to package their Spring Boot applications. According to VMWare’s State of Spring 2021 report, the number of organizations running containerized Spring apps spiked to 57% — compared to 44% in 2020.

What’s driving this significant growth? The ever-increasing demand to reduce startup times of web applications and optimize resource usage, which greatly boosts developer productivity.

Why is containerizing a Spring Boot app important?

Running your Spring Boot application in a Docker container has numerous benefits. First, Docker’s friendly, CLI-based workflow lets developers build, share, and run containerized Spring applications for other developers of all skill levels. Second, developers can install their app from a single package and get it up and running in minutes. Third, Spring developers can code and test locally while ensuring consistency between development and production.

Containerizing a Spring Boot application is easy. You can do this by copying the .jar or .war file right into a JDK base image and then packaging it as a Docker image. There are numerous articles online that can help you effectively package your apps. However, many important concerns like Docker image vulnerabilities, image bloat, missing image tags, and poor build performance aren’t addressed. We’ll tackle those common concerns while sharing nine tips for containerizing your Spring Boot code.

A Simple “Hello World” Spring Boot application

To better understand the unattended concern, let’s build a sample “Hello World” application. In our last blog post, you saw how easy it is to build the “Hello World!” application by downloading this pre-initialized project and generating a ZIP file. You’d then unzip it and complete the following steps to run the app.

Under the src/main/java/com/example/dockerapp/ directory, you can modify your DockerappApplication.java file with the following content:

package com.example.dockerapp;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@SpringBootApplication
public class DockerappApplication {

@RequestMapping("/")
public String home() {
return "Hello World!";
}

public static void main(String[] args) {
SpringApplication.run(DockerappApplication.class, args);
}

}

The following command takes your compiled code and packages it into a distributable format, like a JAR:

./mvnw package
java -jar target/*.jar

By now, you should be able to access “Hello World” via http://localhost:8080.

In order to Dockerize this app, you’d use a Dockerfile.  A Dockerfile is a text document that contains every instruction a user could call on the command line to assemble a Docker image. A Docker image is composed of a stack of layers, each representing an instruction in our Dockerfile. Each subsequent layer contains changes to its underlying layer.

Typically, developers use the following Dockerfile template to build a Docker image.

FROM eclipse-temurin
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "/app.jar"]

The first line defines the base image which is around 457 MB. The  ARG instruction specifies variables that are available to the COPY instruction. The COPY copies the  JAR file from the target/ folder to your Docker image’s root. The EXPOSE instruction informs Docker that the container listens on the specified network ports at runtime. Lastly, an ENTRYPOINT lets you configure a container that runs as an executable. It corresponds to your java -jar target/*.jar  command.

You’d build your image using the docker build command, which looks like this:

$ docker build -t spring-boot-docker .
Sending build context to Docker daemon  15.98MB
Step 1/5 : FROM eclipse-temurin
---a3562aa0b991
Step 2/5 : ARG JAR_FILE=target/*.jar
---Running in a8c13e294a66
Removing intermediate container a8c13e294a66
---aa039166d524
Step 3/5 : COPY ${JAR_FILE} app.jar
COPY failed: no source files were specified

One key drawback of our above example is that it isn’t fully containerized. You must first create a JAR file by running the ./mvnw package command on the host system. This requires you to manually install Java, set up the  JAVA_HOME environment variable, and install Maven. In a nutshell, your JDK must reside outside of your Docker container — adding even more complexity into your build environment. There has to be a better way.

1) Automate all the manual steps

We recommend building up the JAR during the build process within your Dockerfile itself. The following RUN instructions trigger a goal that resolves all project dependencies, including plugins, reports, and their dependencies:

FROM eclipse-temurin
WORKDIR /app

COPY .mvn/ .mvn
COPY mvnw pom.xml ./
RUN ./mvnw dependency:go-offline

COPY src ./src

CMD ["./mvnw", "spring-boot:run"]

💡 Avoid copying the JAR file manually while writing a Dockerfile

2) Use a specific base image tag, instead of latest

When building Docker images, it’s always recommended to specify useful tags which codify version information, intended destination (prod or test, for instance), stability, or other useful information for deploying your application in different environments. Don’t rely on the automatically-created latest tag. Using latest is unpredictable and may cause unexpected behavior. Every time you pull the latest image, it might contain a new build or untested release that could break your application.

For example, using the eclipse-temurin:latest Docker image as a base image isn’t ideal. Instead, you should use specific tags like eclipse-temurin:17-jdk-jammy , eclipse-temurin:8u332-b09-jre-alpin etc.

💡 Avoid using FROM eclipse-temurin:latest in your Dockerfile

3) Use Eclipse Temurin instead of JDK, if possible

On the OpenJDK Docker Hub page, you’ll find a list of recommended Docker Official Images that you should use while building Java applications. The upstream OpenJDK image no longer provides a JRE, so no official JRE images are produced. The official OpenJDK images just contain “vanilla” builds of the OpenJDK provided by Oracle or the relevant project lead.

One of the most popular official images with a build-worthy JDK is Eclipse Temurin. The Eclipse Temurin project provides code and processes that support the building of runtime binaries and associated technologies. These are high performance, enterprise-caliber, and cross-platform.

FROM eclipse-temurin:17-jdk-jammy

WORKDIR /app

COPY .mvn/ .mvn
COPY mvnw pom.xml ./
RUN ./mvnw dependency:go-offline

COPY src ./src

CMD ["./mvnw", "spring-boot:run"]

4) Use a Multi-Stage Build

With multi-stage builds, a Docker build can use one base image for compilation, packaging, and unit tests. Another image holds the runtime of the application. This makes the final image more secure and smaller in size (as it does not contain any development or debugging tools). Multi-stage Docker builds are a great way to ensure your builds are 100% reproducible and as lean as possible. You can create multiple stages within a Dockerfile and control how you build that image.

You can containerize your Spring Boot applications using a multi-layer approach. Each layer may contain different parts of the application such as dependencies, source code, resources, and even snapshot dependencies. Alternatively, you can build any application as a separate image from the final image that contains the runnable application. To better understand this, let’s consider the following Dockerfile:

FROM eclipse-temurin:17-jdk-jammy
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "/app.jar"]

Spring Boot uses a “fat JAR” as its default packaging format. When we inspect the fat JAR, we see that the application forms a very small part of the entire JAR. This portion changes most frequently. The remaining portion contains the Spring Framework dependencies. Optimization typically involves isolating the application into a separate layer from the Spring Framework dependencies. You only have to download the dependencies layer — which forms the bulk of the fat JAR — once, plus it’s cached in the host system.

The above Dockerfile assumes that the fat JAR was already built on the command line. You can also do this in Docker using a multi-stage build and copying the results from one image to another. Instead of using the Maven or Gradle plugin, we can also create a layered JAR Docker image with a Dockerfile. While using Docker, we must follow two more steps to extract the layers and copy those into the final image.

In the first stage, we’ll extract the dependencies. In the second stage, we’ll copy the extracted dependencies to the final image:

FROM eclipse-temurin:17-jdk-jammy as builder
WORKDIR /opt/app
COPY .mvn/ .mvn
COPY mvnw pom.xml ./
RUN ./mvnw dependency:go-offline
COPY ./src ./src
RUN ./mvnw clean install

FROM eclipse-temurin:17-jre-jammy
WORKDIR /opt/app
EXPOSE 8080
COPY --from=builder /opt/app/target/*.jar /opt/app/*.jar
ENTRYPOINT ["java", "-jar", "/opt/app/*.jar" ]

The first image is labeled builder. We use it to run eclipse-temurin:17-jdk-jammy, build the fat JAR, and unpack it.

Notice that this Dockerfile has been split into two stages. The later layers contain the build configuration and the source code for the application, and the earlier layers contain the complete Eclipse JDK image itself. This small optimization also saves us from copying the target directory to a Docker image — even a temporary one used for the build. Our final image is just 277 MB, compared to the first stage build’s 450MB size.

5) Use .dockerignore

To increase build performance, we recommend creating a .dockerignore file in the same directory as your Dockerfile. For this tutorial, your .dockerignore file should contain just one line:

target

This line excludes the target directory, which contains output from Maven, from the Docker build context. There are many good reasons to carefully structure a .dockerignore file, but this simple file is good enough for now. Let’s now explain the build context and why it’s essential . The docker build command builds Docker images from a Dockerfile and a “context.” This context is the set of files located in your specified PATH or URL. The build process can reference any of these files.

Meanwhile, the compilation context is where the developer works. It could be a folder on Mac, Windows or a Linux directory. This directory contains all necessary application components like source code, configuration files, libraries, and plugins. With the .dockerignore file, we can determine which of the following elements like source code, configuration files, libraries, plugins, etc. to exclude while building your new image.

Here’s how your .dockerignore file might look if you choose to exclude the conf, libraries, and plugins directory from your build:

6) Favor Multi-Architecture Docker Images

Your CPU can only run binaries for its native architecture. For example, Docker images built for an x86 system can’t run on an Arm-based system. With Apple fully transitioning to their custom Arm-based silicon, it’s possible that your x86 (Intel or AMD) Docker Image won’t work with Apple’s recent M-series chips. Consequently, we always recommended building multi-arch container images. Below is the mplatform/mquery Docker image that lets you query the multi-platform status of any public image, in any public registry:

docker run --rm mplatform/mquery eclipse-temurin:17-jre-alpine
Image: eclipse-temurin:17-jre-alpine (digest: sha256:ac423a0315c490d3bc1444901d96eea7013e838bcf7cc09978cf84332d7afc76)
* Manifest List: Yes (Image type: application/vnd.docker.distribution.manifest.list.v2+json)
* Supported platforms:
- linux/amd64

We introduced the docker buildx command to help you build multi-architecture images. Buildx is a Docker component that enables many powerful build features with a familiar Docker user experience. All builds executed via Buildx run via the Moby BuildKit builder engine. BuildKit is designed to excel at multi-platform builds, or those not just targeting the user’s local platform. When you invoke a build, you can set the --platform flag to specify the build output’s target platform, (like linux/amd64, linux/arm64, or darwin/amd64):

docker buildx build --platform linux/amd64, linux/arm64 -t spring-helloworld .

7) Run as non-root user for security purposes

Running applications with user privileges is safer, since it helps mitigate risks. The same applies to Docker containers. By default, Docker containers and their running apps have root privileges. It’s therefore best to run Docker containers as non-root users. You can do this by adding USER instructions within your Dockerfile. The USER instruction sets the preferred user name (or UID) and optionally the user group (or GID) while running the image — and for any subsequent RUN, CMD, or ENTRYPOINT instructions:

FROM eclipse-temurin:17-jdk-alpine
RUN addgroup demogroup; adduser  --ingroup demogroup --disabled-password demo
USER demo

WORKDIR /app

COPY .mvn/ .mvn
COPY mvnw pom.xml ./
RUN ./mvnw dependency:go-offline

COPY src ./src

CMD ["./mvnw", "spring-boot:run"]

8) Fix security vulnerabilities in your Java image

Today’s developers rely on third-party code and applications while building their services. By using external software without care, your code may be more vulnerable. Leveraging trusted images and continually monitoring your containers is essential to combatting this. Whenever you build a “Hello World” Docker image, Docker Desktop prompts you to run security scans of the image to detect any known vulnerabilities, like Log4Shell:

exporting to image                                                      0.0s
== exporting layers                                                    0.0s
== writing image sha256:cf6d952a1ece4eddcb80c8d29e0c5dd4d3531c1268291  0.0s
== naming to docker.io/library/spring-boot1                            0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

Let’s use the the Snyk Extension for Docker Desktop to inspect our Spring Boot application. To begin, install Docker Desktop 4.8.0+ on your Mac, Windows, or Linux machine and Enable Extension Marketplace.

Snyk’s extension lets you rapidly scan both local and remote Docker images to detect vulnerabilities.

Install the Snyk extension and supply the “Hello World” Docker Image.

Snyk’s tool uncovers 70 vulnerabilities of varying severity. Once you’re aware of these, you can begin remediation to galvanize your image.

💡 In order to perform a vulnerability check, you can use the following command directly against the Dockerfile: docker scan -f Dockerfile spring-helloworld

9) Use the OpenTelemetry API to measure Java performance

How do Spring Boot developers ensure that their apps are faster and performant? Generally, developers rely on third-party observability tools to measure the performance of their Java applications. Application performance monitoring is essential for all kinds of Java applications, and developers must create top notch user experiences.

Observability isn’t just limited to application performance. With the rise of microservices architectures, the three pillars of observability — metrics, traces, and logs — are front and center. Metrics help developers to understand what’s wrong with the system, while traces help you discover how it’s wrong. Logs tells you why it’s wrong, letting developers dig into particular metrics or traces to holistically understand system behavior.

Observing Java applications requires monitoring your Java VM metrics via JMX, underlying host metrics, and Java app traces. Java developers should monitor, analyze, and diagnose application performance using the Java OpenTelemetry API. OpenTelemetry provides a single set of APIs, libraries, agents, and collector services to capture distributed traces and metrics from your application. Check out this video to learn more.

Conclusion

In this blog post, you saw some of the many ways to optimize your Docker images by carefully crafting your Dockerfile and securing your image by using Snyk Docker Extension Marketplace. If you’d like to go further, check out these bonus resources that cover recommendations and best practices for building secure, production-grade Docker images.

• Docker Development Best Practices
• Dockerfile Best Practices
• Build Images with BuildKit
• Best Practices for Scanning Images
• Getting Started with Docker Extensions

Multi-architecture (multi-arch) images typically contain variants for different architectures and OSes. These images may also support CPU architectures like arm32v5+, arm64v8, s390x, and others. The magic of multi-arch images is that Docker automatically grabs the variant matching your OS and CPU pairing.

While a regular container image has a manifest, a multi-architecture image has a manifest list. The list combines the manifests that show information about each variant’s size, architecture, and operating system.

Multi-architecture images are beneficial when you want to run your container locally on your x86-64 Linux machine, and remotely atop AWS Elastic Compute Cloud (EC2) Graviton2 CPUs. Additionally, it’s possible to build language-specific, multi-arch images — as we’ve done with Rust.

Follow along as we learn about each component behind multi-arch image builds, then quickly create our image using Buildx and Docker Desktop.

Building Multi-Architecture Images with Buildx and Docker Desktop

You can build a multi-arch image by creating the individual images for each architecture, pushing them to Docker Hub, and entering docker manifest to combine them within a tagged manifest list. You can then push the manifest list to Docker Hub. This method is valid in some situations, but it can become tedious and relatively time consuming.

Note: However, you should only use the docker manifest command in testing — not production. This command is experimental. We’re continually tweaking functionality and any associated UX while making docker manifest production ready.

However, two tools make it much easier to create multi-architectural builds: Docker Desktop and Docker Buildx. Docker Buildx enables you to complete every multi-architecture build step with one command via Docker Desktop.

Before diving into the nitty gritty, let’s briefly examine some core Docker technologies.

Dockerfiles

The Dockerfile is a text file containing all necessary instructions needed to assemble and deploy a container image with Docker. We’ll summarize the most common types of instructions, while our documentation contains information about others:

• The FROM instruction headlines each Dockerfile, initializing the build stage and setting a base image which can receive subsequent instructions.
• RUN defines important executables and forms additional image layers as a result. RUN also has a shell form for running commands.
• WORKDIR sets a working directory for any following instructions. While you can explicitly set this, Docker will automatically assign a directory in its absence.
• COPY, as it sounds, copies new files from a specified source and adds them into your container’s filesystem at a given relative path.
• CMD comes in three forms, letting you define executables, parameters, or shell commands. Each Dockerfile only has one CMD, and only the latest CMD instance is respected when multiple exist.

Dockerfiles facilitate automated, multi-layer image builds based on your unique configurations. They’re relatively easy to create, and can grow to support images that require complex instructions. Dockerfiles are crucial inputs for image builds.

Buildx

Buildx leverages the docker build command to build images from a Dockerfile and sets of files located at a specified PATH or URL. Buildx comes packaged within Docker Desktop, and is a CLI plugin at its core. We consider it a plugin because it extends this base command with complete support for BuildKit’s feature set.

We offer Buildx as a CLI command called docker buildx, which you can use with Docker Desktop. In Linux environments, the buildx command also works with the build command on the terminal. Check out our Docker Buildx documentation to learn more.

BuildKit Engine

BuildKit is one core component within our Moby Project framework, which is also open source. It’s an efficient build system that improves upon the original Docker Engine. For example, BuildKit lets you connect with remote repositories like Docker Hub, and offers better performance via caching. You don’t have to rebuild every image layer after making changes.

While building a multi-arch image, BuildKit detects your specified architectures and triggers Docker Desktop to build and simulate those architectures. The docker buildx command helps you tap into BuildKit.

Docker Desktop

Docker Desktop is an application — built atop Docker Engine — that bundles together the Docker CLI, Docker Compose, Kubernetes, and related tools. You can use it to build, share, and manage containerized applications. Through the baked-in Docker Dashboard UI, Docker Desktop lets you tackle tasks with quick button clicks instead of manually entering intricate commands (though this is still possible).

Docker Desktop’s QEMU emulation support lets you build and simulate multiple architectures in a single environment. It also enables building and testing on your macOS, Windows, and Linux machines.

Now that you have working knowledge of each component, let’s hop into our walkthrough.

Prerequisites

Our tutorial requires the following:

• The correct Go binary for your OS, which you can download here
• The latest version of Docker Desktop
• A basic understanding of how Docker works. You can follow our getting started guide to familiarize yourself with Docker Desktop.

Building a Sample Go Application

Let’s begin by building a basic Go application which prints text to your terminal. First, create a new folder called multi_arch_sample and move to it:

mkdir multi_arch_sample && cd multi_arch_sample

Second, run the following command to track code changes in the application dependencies:

go mod init multi_arch_sample

Your terminal will output a similar response to the following:

go: creating new go.mod: module multi_arch_sample
go: to add module requirements and sums:
  	go mod tidy

Third, create a new main.go file and add the following code to it:

package main
 
import (
  	"fmt"
  	"net/http"
)
 
 
func readyToLearn(w http.ResponseWriter, req *http.Request) {
  	w.Write([]byte("<h1>Ready to learn!</h1>"))
	fmt.Println("Server running...")
}
 
func main() {
     
	http.HandleFunc("/", readyToLearn)
  	http.ListenAndServe(":8000", nil)
}

This code created the function readyToLearn, which prints “Ready to learn!” at the 127.0.0.1:8000 web address. It also outputs the phrase Server running… to the terminal.

Next, enter the go run main.go command to run your application code in the terminal, which will produce the Ready to learn! response.

Since your app is ready, you can prepare a Dockerfile to handle the multi-architecture deployment of your Go application.

Creating a Dockerfile for Multi-arch Deployments

Create a new file in the working directory and name it Dockerfile. Next, open that file and add in the following lines:

# syntax=docker/dockerfile:1
 
# specify the base image to  be used for the application
FROM golang:1.17-alpine
 
# create the working directory in the image
WORKDIR /app
 
# copy Go modules and dependencies to image
COPY go.mod ./
 
# download Go modules and dependencies
RUN go mod download
 
# copy all the Go files ending with .go extension
COPY *.go ./
 
# compile application
RUN go build -o /multi_arch_sample
 
# network port at runtime
EXPOSE 8000
 
# execute when the container starts
CMD [ "/multi_arch_sample" ]

Building with Buildx

Next, you’ll need to build your multi-arch image. This image is compatible with both the amd64 and arm32 server architectures. Since you’re using Buildx, BuildKit is also enabled by default. You won’t have to switch on this setting or enter any extra commands to leverage its functionality.

The builder builds and provisions a container. It also packages the container for reuse. Additionally, Buildx supports multiple builder instances — which is pretty handy for creating scoped, isolated, and switchable environments for your image builds.

Enter the following command to create a new builder, which we’ll call mybuilder:

docker buildx create --name mybuilder --use --bootstrap

You should get a terminal response that says mybuilder. You can also view a list of builders using the docker buildx ls command. You can even inspect a new builder by entering docker buildx inspect <name>.

Triggering the Build

Now, you’ll jumpstart your multi-architecture build with the single docker buildx command shown below:

docker buildx build --push \
--platform linux/amd64,linux/arm64 \
--tag your_docker_username/multi_arch_sample:buildx-latest .

This does several things:

• Combines the build command to start a build
• Shares the image with Docker Hub using the push operation
• Uses the --platform flag to specify the target architectures you want to build for. BuildKit then assembles the image manifest for the architectures
• Uses the --tag flag to set the image name as multi_arch_sample

Once your build is finished, your terminal will display the following:

[+] Building 123.0s (23/23) FINISHED

Next, navigate to the Docker Desktop and go to Images > REMOTE REPOSITORIES. You’ll see your newly-created image via the Dashboard!

Conclusion

Congratulations! You’ve successfully explored multi-architecture builds, step by step. You’ve seen how Docker Desktop, Buildx, BuildKit, and other tooling enable you to create and deploy multi-architecture images. While we’ve used a sample Go web application, you can apply these processes to other images and applications.

To tackle your own projects, learn how to get started with Docker to build more multi-architecture images with Docker Desktop and Buildx. We’ve also outlined how to create a custom registry configuration using Buildx.

The Big Picture

• MergeOp and DiffOp are two new features released in BuildKit v0.10. These operations let you assemble container images by composing filesystems (MergeOp) and splitting them apart (DiffOp), all while minimizing the creation of duplicated data both locally on disk and in remote registries.
• Minimizing data duplication enhances cache reusability, which can offer a significant performance boost to many workflows. One early adopter, Netflix, reported that highly complex builds — which previously took over an hour — now take only three minutes after updating their internal tooling to use MergeOp instead of copies.
• Additionally, these new operations make the workflows you’d find in a package manager, where software dependencies are structured in a complex DAG, much more performant and easily expressible as BuildKit builds.
• MergeOp and DiffOp, much like other BuildKit features, are low-level primitives meant for higher-level tooling and interfaces. Though they’re quite new, support for them already exists in:

• • The new `COPY –link` flag supported by Dockerfile, as described in this previous blog post, which uses MergeOp underneath the hood
  • HLB and Dagger, which both now support Merge+Diff operations

This post showcases how MergeOp and DiffOp help encode knowledge of software dependency relationships, which enables BuildKit to maximize cache reusability and unlocks some interesting new use cases.

BuildKit Crash Course

Before diving into new features, let’s take a quick crash course on BuildKit and LLB.

Filesystem DAG Solver

BuildKit is fundamentally a Directed Acyclic Graph (DAG) solver where each vertex in the DAG represents an operation performed on a set of one or more filesystems. Each vertex operation can accept as input the filesystems of other vertices, and then output one or more filesystems of its own. These inputs and outputs create the edges of the DAG.

When solving a DAG, BuildKit uses caching, parallelization, and other optimizations to calculate the filesystems output by the “target” vertex of the solve as efficiently as possible. You can optionally export your target vertex’s result in different ways: such as a container image pushed to a registry, or as a locally-created directory on the client’s machine.

LLB DAGs

DAGs are provided to BuildKit in the form of LLB, which is a low-level language designed as something that higher-level build specifications can be “compiled” to. This mirrors how many different programming languages are compilable to a single target like assembly language. A few examples of tools and languages that compile to LLB are Dockerfile, HLB, Earthfile, and Dagger (which converts Cue to LLB).

Crucially, this lets BuildKit focus on generic features or optimizations like caching and exporting.  Higher-level tools can leverage these mechanisms instead of reinventing the wheel.

Example: Dockerfile to LLB DAG

First, consider the following Dockerfile:

FROM golang:1.17-alpine3.15 as gopls
RUN go install golang.org/x/tools/gopls@latest

FROM golang:1.17-alpine3.15 as goimports
RUN go install golang.org/x/tools/cmd/goimports@latest

FROM alpine:3.15
RUN apk add --update --no-cache neovim
COPY --from=gopls /go/bin/gopls /usr/local/bin/gopls
COPY --from=goimports /go/bin/goimports /usr/local/bin/goimports

We can take this and easily compile it into an LLB DAG:

Here, you can see a few different types of vertices, with each type representing a different kind of operation to create or modify a filesystem:

• SourceOp vertices import a new filesystem from external sources like container images, Git repositories, or others. They have no inputs and usually form the “roots” of an LLB graph.
• ExecOp vertices run a command atop an input filesystem and output the filesystem plus any changes made. They may also accept multiple input filesystems mounted at different locations, also letting them output each modified filesystem separately.
• CopyOp vertices output a filesystem where specified sets of files and directories — from one input vertex (the source) — have been copied atop another input vertex (the destination). Technically, copying is a subtype of general FileOp vertices. It supports a number of different basic file operations, but we refer to it as CopyOp here for simplicity.

Key Takeaways

Here’s what we’ve learned so far:

• BuildKit solves DAGs of filesystem operations
• BuildKit supports exporting the results of those solves in a variety of formats, including container images and more
• Through LLB, BuildKit can be a foundation for a wide variety of current higher level build-specification languages

The Problem: Linear Copy Chains

While a number of build systems and languages have been compiled to LLB with great success, one class of systems was — until Merge+Diff — difficult to support as efficiently as needed.

Surprisingly, this class of systems makes heavy use of DAGs to represent software dependencies. This includes users who are building software and libraries with many overlapping, transitive dependencies. Additional examples include package managers used by Linux distros and programming languages.

Superficially, it’s odd that BuildKit had any trouble modeling software-dependency DAGs since it uses DAGs to represent builds. Let’s explore an example to see what those issues were.

The example revolves around building software from source with the following dependency DAG:

Note that this DAG is not LLB yet. This dependency DAG just shows what you need to build each vertex’s software, where each edge is a build-time dependency. For example, the bar binary only needs base and the bar libs to build, but the baz binary needs base, bar, and foo libs.

We want to convert this into an LLB DAG that builds the foo, bar, and baz binaries — and then exports that build as a container image. We want our container image to only contain those binaries — not build time-only dependencies like the static library files.

We’re taking a few shortcuts for simplicity’s sake. Accordingly, we’re bundling multiple dependencies like libc, GCC, and source code inside base while combining groups of static libraries into single vertices. In reality, you can divide those dependencies between separate vertices at the expense of extra complexity.

Building DAGs with Copy Chains

We’ll start by converting each target vertex (foo, bar, and baz) into LLB individually before combining them all together. Thankfully, foo and bar have simple dependency structures:

The final CopyOp in each graph isolates /bin/foo and /bin/bar onto scratch, which is just an empty filesystem. Conversely, baz is a bit more involved:

Here, we split the foo and bar libs builds into separate steps, since they share no dependencies. This lets BuildKit parallelize the build steps for each and cache the results separately.

The results of those ExecOps are then combined into a single build dependency closure via CopyOp, which creates a filesystem where the lib/ directory contains both foo and bar libs.

Finally, let’s combine everything into one LLB graph:

Now, say you submit your LLB to BuildKit for a solve and want to export the result as a registry container image. The final chain of CopyOps becomes the layers of the exported image. This is pushed to the registry alongside an image manifest.

If you run the solve again with the same BuildKit instance, you’ll find that — thanks to BuildKit’s caching features — no actual work is required:

• During the solve and before running each vertex, BuildKit checks to see if any vertex inputs have recently changed. If not, BuildKit uses cached results and avoids a rerun.
• When exporting the result as a container image, BuildKit only needs to push those layers to the registry that aren’t already present. Since the registry already contains layers from the previous build, no pushes are needed.

Cascading Cache Invalidation

This is all good news, but there are problems. Where do those come from? Let’s consider what happens if you were to rerun the build again — but with a change to the bar libs’ builds. In this case, we’ll assume that you’re building with a new flag.

Again, BuildKit checks to see if any LLB vertex dependencies have changed before grabbing the cached result. Changes invalidate the cache and force a rerun, which cascades to any descendent vertices:

Here, we mark any cache-invalidated vertices with an ❌. Some invalidations here are perfectly logical:

• The ExecOp building the bar libs can’t be cached because the definition changed, adding BUILDFLAG=something-new to the vertex definition.
• The ExecOp which builds bar must be invalid since its dependency libs are invalid.

Meanwhile, we’ve marked unexpected invalidations in the LLB with ❌⁉️:

• The copy of the foo libs needed to create the dependency closure under the build of /bin/baz is invalidated. This is because that copy’s destination is atop the build for the bar libs — creating a cache dependency. This happens even though our foo and bar libs don’t actually share any dependencies. The CopyOp must re-run, yet it ultimately creates the same layer as before.
• The copy of /bin/foo into the final image layer is also invalid for similar reasons. The final image is constructed as a linear chain of CopyOps, even though no components share dependencies. Since /bin/foo comes after /bin/bar in the chain, those cascading invalidations will impact it.

We could technically fix these niche issues by switching our copying order. However, that’d only shift any bar lib invalidation to your foo libs. These unnecessary invalidations have costs:

• BuildKit must perform the copies locally again, which eats up time and disk space.
• When exporting LLB results as container layers, BuildKit works harder to determine the contents of each layer and recompress those to tarballs. This still happens even though the layers didn’t need to change, and usually end up with the exact same contents as before.
• BuildKit has an optimization that prevents remote registry layers from being pulled locally unless strictly necessary. This minimizes data downloads. It works with image layers referenced by a SourceOp, plus layers referenced via BuildKit’s remote cache import feature. However, its usefulness is hampered when unneeded layers are caught in a cascading cache invalidation. This process can forcefully pull them down as the destination of a CopyOp.

The above issues impact many use cases, yet are often inconsequential since they don’t create a performance bottleneck. Copying small quantities of data isn’t often noticeable. However, our example is fairly basic. Dependency DAGs can get much more complicated than this, and involve many more files larger in both size and quantity. This is where cache invalidation becomes a true bottleneck.

Key Takeaways

We generally use the following approach to build each vertex of a dependency DAG via LLB:

1. 1. Combine dependencies – Topologically sort the build-time dependency graph and create an LLB vertex that outputs a filesystem containing that closure. CopyOps can be used to combine independent filesystems when needed, which you can see while copying foo libs atop bar libs — for baz builds in the example above.
   2. Execute Build – Run the vertex build as an ExecOp atop the vertex from step one.
   3. Package Artifacts – If needed, separate the build artifacts created during step two from the files and directories of the dependency closure. This creates a well-defined, isolated package. Isolation is also possible via a CopyOp, which becomes apparent while copying each binary into the final image in our previous example.

We’ll refer back to these steps later!

The issue with this approach is that too much information is lost in step one when a DAG of software dependencies is linearized into a copy chain. A change to one input invalidates the destination directory for the next input. Each subsequent input then falls victim to that cascading effect. This happens despite the fact that each DAG input should be independent. This creates overly-fragile caches and impairs BuildKit’s ability to maximize filesystem re-use — both during builds and while exporting as container layers.

LLB needs new types of operations that let us compose filesystems while keeping them independent. This is where Merge+Diff comes in handy.

The Solution: Composable Filesystems with Merge+Diff

MergeOp

MergeOp is a new type of LLB vertex that lets you efficiently merge filesystems without creating interdependencies. Each vertex input is a filesystem and the output is a filesystem, where each input is applied atop another. Here’s a basic example:

Here, our MergeOp output is annotated with its contents. These are simply the combined contents of each input to the MergeOp. Any overlapping directories have their contents merged. If any two files are located along the same path, the file from the latter-most input takes precedence (the order of the inputs thus matters). This example shows a merger of SourceOp and CopyOp vertices, but MergeOp accepts the result of any LLB vertex.

When building software dependency DAGs, MergeOp is useful during the “Combine Dependencies” step listed previously. It replaces CopyOp and avoids its common pitfalls in this use case.

MergeOp behaves in a few key ways:

• Input Layer Reuse – When a MergeOp’s result is included in a container image, each input’s layers are conjoined rather than recreated or squashed. This enables maximum re-use of previously-exported layers.
• Laziness – MergeOp is implemented lazily, which means that the on-disk representation of the merged filesystem is only created when strictly necessary. Local creation is necessary when this representation is an ExecOp input. It’s not necessary to create a MergeOp result locally when you directly export it as a container image. The full implications of this feature are explored later under “Container Images as Lazy Package Merges”.
• Hardlink Optimizations – When you must create a  MergeOp result locally, the on-disk filesystem will hard link files to create the merged tree rather than copying them. This prevents large files from becoming a bottleneck during merges — especially when using filesystems like ext4 that don’t support reflink optimizations.

Hardlinking is only available when using the two leading snapshotter backends: overlay and native. Currently, other backends will create merged filesystems by copying files, preferring to use the copy_file_range syscall when available. However, equivalent optimizations for other snapshotter types like estargz will likely appear in the future.

If you’re wondering why hard links are needed at all for the overlay snapshotter when you could instead combine lowerdirs into an overlay mount, your curiosity is valid. Sadly, this approach doesn’t always work due to opaque directories, as described in corner cases 1 and 2 of this tangentially related Github issue. It may be possible to make it work with more effort, but the hard link approach remains easier for now.

DiffOp

DiffOp is another new type of LLB vertex that lets you efficiently separate a filesystem from its dependency base. It accepts two inputs: a “lower” and an “upper” filesystem. It then outputs the filesystem that separates upper from lower.

You can also view DiffOp as a way to “unmerge” filesystems — something inverse to MergeOp. Here’s a basic example:

You can see that the diff is between the SourceOp base and an ExecOp made atop this base. This essentially detaches the ExecOp from its dependency. The DiffOp’s result only contains any changes made during the ExecOp.

While building software-dependency DAGs, DiffOp is useful during the “Package Artifacts” step listed earlier. It helps isolate files and lets them use their own “package” independent of their original dependencies — without making a copy. However, DiffOp use cases are more obscure than those of MergeOp. BuildKit’s user guides cover these scenarios in greater detail.

DiffOp shares behavioral characteristics with MergeOp:

• Input Layer Reuse – When there’s a “known path” from the lower input to the upper input, a container image export that includes the DiffOp result will just reuse the layers that separate lower and upper. When the path is unknown, DiffOp still outputs a consistent result but cannot re-use input layers.
• Laziness – DiffOp is also implemented lazily; the on-disk filesystem representation is only created when needed. Exporting a DiffOp result as a container image does not require “unlazying” its inputs (except where there’s no known path between lower and upper, as mentioned above).
• Hardlink Optimizations – BuildKit creates DiffOp results locally with the same optimizations (and current caveats) as MergeOp.

Example: Two New LLB Operations

Let’s see how these two new LLB operations can solve our problems from the previous section. Here’s a new-and-improved approach for building a vertex in a dependency DAG using LLB:

1. 1. Combine dependencies – Plug dependencies into a MergeOp (in topologically-sorted order, if needed).
   2. Execute Build – Run the build as an ExecOp atop the merged filesystem from step one.
   3. Package Artifacts – Use DiffOp (or equivalent techniques from the user guide) to extract build artifacts into their own independent filesystems. You can then plug DiffOp’s output into other MergeOps or export it directly.

Using those rules, here’s the updated LLB:

Now, let’s consider the same scenario as before, where we perform an initial build of the above LLB and a subsequent build modified bar libs:

In this instance, only expected invalidations occur. BuildKit performs each build step using only vital dependencies. Cache invalidations only cascade to vertices to which they should. There are longer any copies of foo libs or foo that invalidate unnecessarily.

We owe this to the fact that invalidation of one MergeOp input doesn’t invalidate others, avoiding issues with linearized copy chains. While a MergeOp as a whole is invalidated when one input is invalidated, the merged filesystem is only created when strictly necessary.

The MergeOps for each bar and baz dependency closures must be created on-disk since ExecOps is running atop them. Luckily, hardlinks make this process efficient — provided that a supported snapshotter backend is in use.

The final MergeOp will not be created on-disk. This — in combination with DiffOp’s lazy implementation — means that the exported image will only consist of the layers created by the make install ExecOps. You won’t need to create any intermediate data or layers.

Key Takeaways

Merge and Diff enable filesystem operations to be composed and decomposed in ways mirroring the relationships expressed in software-dependency DAGs. By allowing filesystem layers reuse, BuildKit can maximize its cache reuse of those layers. This improves performance and reproducibility.

More Applications

In addition to building your dependency DAGs more efficiently, Merge and Diff unlock plenty of other interesting possibilities. We’ll explore a few hypothetical ones here.

Importing Package DAGs

Many existing container-image builds install packages from a Linux distro package manager through an ExecOp (e.g. executing apt, apk, dnf, etc). LLB for this might resemble the following:

Similar patterns are common to language package managers like npm, pip, and Go modules.

This approach works, but suffers from the same caching problems described earlier. In this case, if one install package changes, it invalidates that install step. You must run the entire process again, even if the package DAG is only partially modified. This is another case where LLB doesn’t encode any knowledge about your actual dependency DAG, which disrupts BuildKit’s ability to cache data effectively.

MergeOp unlocks new possibilities here. We can do the following:

• Take the list of desired packages and expand them into the full dependency DAG, including all transitive dependencies.
• Convert each package into an LLB state by downloading the package contents with an ExecOp. DiffOp may help you separate the artifacts. However, using a separate mount during the ExecOp may be adequate.
• Use MergeOp to combine those individual packages in topological order.

The LLB to implement this could look something like the following:

Note that we’re using separate ExecOp mounts instead of DiffOp, as described in the user guide. DiffOp is a workable solution if need be.

This solves the caching problems common to older approaches.  You can import packages into a container image without duplicating data and work. If you run another build with a slightly different set of packages, BuildKit can re-use its cache for each package from the previous build.

Additionally, now that the package DAG has been compiled into LLB, you can extend it arbitrarily. Should you want to build your own software atop your imported packages, it’s as easy as taking your LLB from the above steps and adding your own ops.

Container Images as Lazy Package Merges

Say that someone used the above techniques to build or import a large package DAG, and then pushed each individual package in the DAG as a single layer image to a container registry.

Next, let’s say that someone has a list of packages that they want to convert into a container image. Using BuildKit, the “build” would consist of topologically sorting the dependency DAG of those packages and combining them with a MergeOp:

“Build” is in quotes because little actual work is required. Image vertices are lazy just like MergeOp and DiffOp, so the entire LLB DAG is lazy. In practice, this build consists of constructing a manifest for the container image, and pushing that to the registry. The input package images never need to be pulled down locally thanks to these lazy implementations. After pushing the manifest, anyone can grab that image from the registry and run it using the container runtime of their choice — just as they would with any other image.

In this scenario, the container image registry becomes a package cache and “builds” are just declarations of which packages must be present in a given image, pushed as a manifest to the registry.

A Few Wrenches

This is all pretty neat! However, there are some caveats:

• Package managers often support hooks that result in the execution of scripts during  package installs. In many cases, you can disable these scripts for container image builds. When they’re strictly required, you can always represent them using ExecOps on top of MergeOps. However, that’ll mitigate some performance optimizations since you must create a merged filesystem locally.
• If a runtime package manager is required after image export, then the LLB will also need a way of merging a layer containing the package database. For example, this is necessary if you want to run apt commands in a Debian-based image — which may be useful within development environments.
• To convert a package DAG to LLB, you’ll need programmatic access to the package manager’s metadata database. This is easier said than done. You do have multiple options, such as parsing the CLI output or using library support, which you should evaluate on a case-by-case basis.

None of these issues are show-stoppers, but anyone creating tools while that utilize Merge and Diff for the above use cases should keep them in mind.

Thanks!

My interest in Merge+Diff stems from my own attempts at building complex package DAGs with BuildKit. However, other BuildKit users were also interested in the features — particularly Netflix — and contracted me to design and implement them. Their support has been greatly appreciated, in particular from engineers Edgar Lee and Aaron Lehmann. Both have provided invaluable feedback from start to finish.

Also, many thanks to the community of maintainers of BuildKit — especially Tõnis Tiigi, who’s also expressed interest in these features for a while. Merge+Diff was fairly complex to design and not straightforward during code-review, but thanks to him and everyone else involved, I think we’ve created some really powerful tools. These should unlock lots of interesting use cases moving forward.

In this guide, we’ll highlight a tentpole feature of BuildKit v0.10: new build information-structure generation, from build metadata. This allows you to see all sources (images, Git repositories, and HTTP URLs) and configurations passed on to your build. This information is also embeddable within the image config.

We’ll discuss how to tackle this process, and share some best practices along the way.

Getting Started

While this feature is automatically activated upon updating to BuildKit v0.10, we also recommend using BuildKit’s Dockerfile v1.4 to reliably capture original image names. You can do so by adding the following syntax to your Dockerfile: # syntax=docker/dockerfile:1.4.

Additionally, we recommend creating a new docker-container builder with Buildx that uses the latest stable version of BuildKit. Enter the following CLI command:

$ docker buildx create --use --bootstrap --name mybuilder

Note: to return to the default builder, enter the $ docker buildx use default command.

Next, let’s create a basic Dockerfile:

 # syntax=docker/dockerfile:1.4

FROM busybox AS base
ARG foo=baz
RUN echo bar > /foo

FROM alpine:3.15 AS build
COPY --from=base /foo /
RUN echo baz > /bar

FROM scratch
COPY --from=build /bar /
ADD https://raw.githubusercontent.com/moby/moby/master/README.md /

We’ll build this image using Buildx v0.8.1 — which comes packaged within the latest version of Docker Desktop (v4.7). The latest Buildx version lets you inspect and use any build information that’s been generated:

$ docker buildx build --build-arg foo=bar --metadata-file metadata.json .

Storing Build Metadata as a File

We’re using the --metadata-file flag, which writes the build result metadata within the metadata.json file. This flag helps retrieve metadata information about your build result — including the digest of your resulting image, and the new containerimage.buildinfo key.

The --metadata-file flag improves upon the previous --iidfile flag, which would only capture the resulting image ID. The following Dockerfile shows the containerimage.buildinfo key in practice:

{
"containerimage.buildinfo": {
"frontend": "dockerfile.v0",
"attrs": {
"build-arg:foo": "bar",
"filename": "Dockerfile",
"source": "docker/dockerfile:1.4"
},
"sources": [
{
"type": "docker-image",
"ref": "docker.io/library/alpine:3.15",
"pin": "sha256:d6d0a0eb4d40ef96f2310ead734848b9c819bb97c9d846385c4aca1767186cd4"
},
{
"type": "docker-image",
"ref": "docker.io/library/busybox:latest",
"pin": "sha256:caa382c432891547782ce7140fb3b7304613d3b0438834dce1cad68896ab110a"
},
{
"type": "http",
"ref": "https://raw.githubusercontent.com/moby/moby/master/README.md",
"pin": "sha256:419455202b0ef97e480d7f8199b26a721a417818bc0e2d106975f74323f25e6c"
}
]
},
"containerimage.config.digest": "sha256:cd82085d327d4b41f86212fc372f75682f131b5ce5c0c918dabaa0fbf04ec53f",
"containerimage.descriptor": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"digest": "sha256:6afb8217371adb4b75bd7767d711da74ba226ed868fa5560e01a8961ab150ccb",
"size": 732
},
"containerimage.digest": "sha256:6afb8217371adb4b75bd7767d711da74ba226ed868fa5560e01a8961ab150ccb",
}

What’s noteworthy about the structure of this result? The new containerimage.buildinfo key now contains your build information. You’ll also see a host of important field names:

• frontend defines the BuildKit frontend responsible for the build (we’re building from a Dockerfile above).
• attrs encompasses the build configuration parameters (e.g. when typing --build-arg).
• sources defines build sources.

Additionally, each sources entry describes an external source that your Dockerfile used while building the result. However, it’s worth highlighting some other JSON data:

• type can reference a docker-image for all container images referenced with FROM, or with git using a Git context. Finally, type can reference HTTP URL contexts or remote URLs used by ADD commands.
• ref is the reference defined in your Dockerfile.
• pin lets you know which dependency version is installed at any time (digest).

Remember that sources are captured for all of your build stages, and not just for the last stage’s base image that was exported.

Storing Build Metadata as Part of Your Image

Your metadata file isn’t the only transport available. BuildKit also embeds build information within the image config as your image is pushed. This makes your build information portable. Here’s what that push command looks like:

$ docker buildx build --build-arg foo=bar --tag crazymax/buildinfo:latest --push .

You can check the build information for any existing image — while on the latest Buildx version — using the imagetools inspect command:

$ docker buildx imagetools inspect crazymax/buildinfo:latest --format "{{json .BuildInfo}}"

{
"frontend": "dockerfile.v0",
"sources": [
{
"type": "docker-image",
"ref": "docker.io/library/alpine:3.15",
"pin": "sha256:d6d0a0eb4d40ef96f2310ead734848b9c819bb97c9d846385c4aca1767186cd4"
},
{
"type": "docker-image",
"ref": "docker.io/library/busybox:latest",
"pin": "sha256:caa382c432891547782ce7140fb3b7304613d3b0438834dce1cad68896ab110a"
},
{
"type": "http",
"ref": "https://raw.githubusercontent.com/moby/moby/master/README.md",
"pin": "sha256:419455202b0ef97e480d7f8199b26a721a417818bc0e2d106975f74323f25e6c"
}
]
}

Unlike with your metadata-file results, build attributes aren’t automatically available within the image config. Attribute inclusion is currently an opt-in configuration to avoid troublesome leaks. Developers occasionally use these attributes as secrets, which isn’t ideal from a security standpoint. We recommend using --mount=type=secret instead.

To add build attributes to your embedded image config, use the image output attribute, buildinfo-attrs:

$ docker buildx build \
--build-arg foo=bar \
--output=type=image,name=crazymax/buildinfo:latest,push=true,buildinfo-attrs=true .

Alternatively, you can use the Buildx BUILDKIT_INLINE_BUILDINFO_ATTRS build argument:

$ docker buildx build \
--build-arg BUILDKIT_INLINE_BUILDINFO_ATTRS=1 \
--build-arg foo=bar \
--tag crazymax/buildinfo:latest \
--push .

That’s it! You may now review any newly-generated build dependencies stemming from your image builds.

What’s next?

Transparency is important. Always aim to make your Docker images more self-descriptive, decipherable, reproducible, and visible. In this case, we’ve made it easier to uncover any inputs used while building an image. Additionally, you might compare images updated with security patches to pinned versions of your build sources. That lets you know if your image is up-to-date or safe to use.

This is one important step in our Secure Software Supply Chain (SSSC) journey, and towards better build reproducibility. More information about reproducibility is also available within our BuildKit repo.

However, we want to go even further. Docker is bringing SBOMs to all container images via BuildKit. We want to get our development community involved in this effort to bolster BuildKit — and take that next major step towards higher image-level transparency.

Are you interested in learning more about BuildKit? Our latest BuildKit release has shipped with other useful features — like those showcased in Tonis Tiigi’s blog post. If you’ve been clamoring for improved remote cache support and rapid image rebase, it’s well worth a read!

With the Dockerfile 1.4 release, the COPY and ADD commands for copying files from the build context or from another stage now accept a new flag `–link`. Using this flag enables much better cache semantics as well as the ability to perform a fast 2nd-day rebase of your builds on top of new base images without rebuilding them.

In order to use this flag, you will need to add a line containing # syntax=docker/dockerfile:1.4 to the top of your Dockerfile. This makes sure that the proper frontend image with support for this flag is loaded. In order to get the correct cache semantics for the flag, BuildKit v0.10 needs to be used as well.

# syntax=docker/dockerfile:1.4
FROM ...
COPY --link foo bar

docker buildx create --use --name mybuilder
docker buildx build .

Before we get into the details of what this new flag does, let’s go over how the Dockerfile commands work at the moment.

Docker images consist of layers that are tarballs in the registry that make up the container filesystem. When you pull an image, these tarballs get extracted on top of each other. The implementation of how this extraction happens and how files actually get stored on the disk depends on the underlying snapshotter type. If you use the overlay snapshotter, your filesystem can create a special mount that combines multiple directories into one. For other snapshotters the process usually involves making (shallow) copies of files.

Every RUN, COPY or ADD command in Dockerfile also creates a new snapshot that is added on top of previously created contents. Once the build is ready and you want to export an image as a build result, we will run a “differ” component that compares all the snapshots and creates new tarballs containing the new files that were added in each snapshot.

An important concept to understand here is that in order for a new layer to be created, the previous layers(also called parent layers) already need to be created before and exist on disk. Whenever you used COPY command to move some files to a directory, all the previous commands on the same stage needed to be completed before. Without it, you wouldn’t have the destination directory where the files would be copied to.

This limitation changes now with the new --link flag that has been added to COPY and ADD commands. When this flag is present, the COPY command works in a different mode where files are instead copied to a completely new snapshot. Then this new snapshot is turned into a new layer tarball on its own, and that tarball is linked into the chain of previous tarball layers. This linking action is usually just a metadata change where a new item is added to the layers array without the need to access or move any files. As shown in the next examples, it can even happen remotely with the layers existing in the remote registry without ever needing to pull or push them.

To summarize:

• COPY --link=false (previous method and default): Files are copied on top of the result of the previous command Layers are created later by comparing snapshots on disk
• COPY --link=true: Files are copied to a new location and turned into an independent layer Layer identifier is added on top of previous layers

By removing the dependency from the destination directory, we don’t need to wait for previous commands to finish before completing the COPY command. We also do not need to invalidate our build cache for the current command when previous commands on the same Dockerfile stage change.

Let’s look at some example use-cases that this enables.

Example: Rebasing an existing image

The previous release of BuildKit v0.9 introduced another new feature: lazy image pulling. What this feature means is that whenever BuildKit needs to access a remote image/cache, it will delay the pulling of its layers until there is a task that actually needs to read files from them. For example, when a layer is just used in another image this pulling is not needed and BuildKit can just create a new image referencing the previous layer by its immutable digest.

FROM ubuntu
ENV MYCONFIG=foo
VOLUME /data

For example, if you build this Dockerfile with docker buildx build -t myuser/myubuntu --push . on a clean system without cache, you will notice that the whole build only takes a couple of seconds before your new image is ready in your repository. This is because the layers of the ubuntu image are never pulled to your local machine and never pushed to hub repository. Instead, BuildKit creates a new image config and manifest containing the Ubuntu layer digests and pushes only them. The layers are linked directly from the Ubuntu repository using the cross-repo mount feature of the registry. This pattern can also be used with a remote cache source where your build would only need to validate that remote cache is still up-to-date and not actually pull down any layers.

This method works well for metadata commands like ENV and VOLUME that only modify the image config. If you used a command that created new layers like COPY or RUN, the base image still needed to be pulled first because local files were needed in order to run these commands.

COPY --link removes this requirement. Let’s look at a common multi-stage build Dockerfile that has been updated to use COPY --link:

#syntax=docker/dockerfile:1.4
FROM golang AS build
....
RUN go build -o /myapp .

FROM alpine:3.14
COPY --from=build --link /out/myapp /bin
ENTRYPOINT ["/bin/myapp"]

When you build this file with BuildKit v0.10, the first thing you will notice is that your build completes without ever pulling the Alpine image. This is because copying myapp to the /bin/ directory does not depend on Alpine files anymore. If you push this image to another Docker Hub repository Alpine layers are linked directly. Only if you export the image in some other way, for example into a local OCI tarball with --output type=oci will the layers be actually pulled.

Now when we have built and pushed this image for the first time, we can look at what happens when we need to update this image in the future. Either in the case a new Alpine 3.14 image with security fixes comes out or when we want to update to 3.15.

To avoid rebuilding everything again we can store remote cache from our earlier build. BuildKit supports many cache backend but the easiest, in this case, is to use “inline cache” that just embeds the build cache information into the image config.

To enable inline cache we either run:

docker buildx build --cache-to type=inline --push -t mysuser/myapp .

or

docker buildx build --build-arg BUILDKIT_INLINE_CACHE=1 --push -t mysuser/myapp .

Now we can use the image itself as a cache source when doing subsequent builds. For example, let’s see what happens when we update our previous Dockerfile to use Alpine 3.15 instead and build using the previous cache.

FROM golang AS build
....
FROM alpine:3.15
COPY --from=build --link /out/myapp /bin/
ENTRYPOINT ["/bin/myapp"]

docker buildx build --cache-from myuser/myapp -t myuser/myapp --push .

Similarily to our initial build, we will see that alpine:3.15 is not actually pulled to the local machine, and instead, the layer blobs were directly moved inside the registry. What might be more interesting is that the golang image was not pulled as well. This is because we can verify that the myapp binary has not changed, and therefore the second layer in our image has not changed as well, and we can just rebase it on top of the new alpine image. This all happens completely remotely without any local layers.

Note that without --link this was not possible before as the COPY operation depended on /bin directory from the base image and its cache was not valid anymore because the base image changed, resulting in pulling both Alpine and Golang image and recompilation of myapp binary.

Example: Better remote cache support

As another example, let’s look at how the cache is handled if you have multiple COPY commands.

#syntax=docker/dockerfile:1.4
FROM golang AS build
....
RUN go build -o /myapp .

FROM ubuntu AS config
...
RUN generate -o /myapp.config

FROM alpine:3.14
COPY --from=config --link /myapp.config /etc/
COPY --from=build --link /myapp /bin/
ENTRYPOINT ["/bin/myapp"]

In this file, we have added a second copy that adds a generated config file from another build stage. It is a very common pattern to use multiple stages for dependencies and then copy them all together in a final stage. This is how you get the best parallelization and cache reuse for your builds.

Let’s say we build and push this Dockerfile with inline cache as before:

docker buildx build --cache-to type=inline -t myuser/myapp2 --push .

Now let’s consider what happens when we need to do a rebuild using our previous inline cache and our config file generation has changed. The stage with our config generation needs to run again, but what happens to the last stage?

Without using --link, if the file myapp.config changed it would mean that Alpine image was pulled and extracted, myapp.config copied over that snapshot, and because that changed the dependencies for the COPY of myapp it would need to be recompiled and copied again as well. Note that the possibility of cache reuse here depended on the order of commands, the cache could be used until the last COPY command that matched cache, and all commands after that would need to run again. If cache for myapp would have been invalidated, we still would have gotten cache for myapp.config because that file was copied before, but not vice versa.

By adding --link, the cache reuse is now much better. All the COPY commands are now independent and none of them depend on the base image. After the new config is generated, it is directly converted into a new layer. Then this layer is replaced inside the previous image. The bottom layers for the base image and the top layer containing myapp are left as is – they never need to be pulled to the local machine at all. Only the new layer is pushed together with the new image manifest.

Differences from --link=false

You might wonder why was the new flag added at all instead of changing all the COPY commands to use new semantics automatically. The reason is that it is not completely backward-compatible in some rare cases. For example, let’s say your copy command is COPY myapp /path/to/myapp. If the destination directory you specified in /path/to/myapp contained a symlink in one of the components, it would have been followed and files copied to the symlink target instead. With --link, all the copies are independent, and they are never allowed to see what files the destination path contained. So instead of following a symlink, COPY --link myapp /path/to/myapp would first always create a new directory /path/to and copy the file inside it.

Another case you might see is with a command like COPY myapp /usr/bin. Notice that the destination path does not end with a slash. Without --link the previous semantics would have checked if /usr/bin is a directory. If it was, then the file would be copied as /usr/bin/myapp. If it was not then the new file would have been copied to /usr as regular file bin. These kinds of checks require extracting files on disk so that their types can be verified and are not allowed with --link. Therefore when using --link, you need to make sure that the destination path does not contain a symlink and not use ambiguous destination directory detection.

The cases listed above should be quite rare and easy to fix by simple Dockerfile modifications. If you don’t rely on symlinks in your COPY commands, the recommendation is to always start using --link. The performance of linked copies should always be either better or equivalent to regular copies, and you get much better cache reuse and optimizations for your builds.

If you are interested more about the internals on COPY --link, it is powered by the new MergeOp feature in BuildKit’s LLB definition. You can read more about MergeOp, as well as the companion DiffOp feature that is conceptually a reverse of MergeOp from BuildKit documentation..

As of a couple weeks ago, Docker’s BuildKit tool for building Dockerfiles now supports heredoc syntax! With these new improvements, we can do all sorts of things that were difficult before, like multiline RUNs without needing all those pesky backslashes at the end of each line, or the creation of small inline configuration files.

In this post, I’ll cover the basics of what these heredocs are, and more importantly what you can use them for, and how to get started with them! 🎉

BuildKit (a quick refresher)

From BuildKit’s own github:

BuildKit is a toolkit for converting source code to build artifacts in an efficient, expressive and repeatable manner.

Essentially, it’s the next generation builder for docker images, neatly separate from the rest of the main docker runtime; you can use it for building docker images, or images for other OCI runtimes.

It comes with a lot of useful (and pretty) features beyond what the basic builder supports, including neater build log output, faster and more cache-efficient builds, concurrent builds, as well as a very flexible architecture to allow easy extensibility (I’m definitely not doing it justice).

You’re either most likely using it already, or you probably want to be! You can enable it locally by setting the environment variable DOCKER_BUILDKIT=1 when performing your docker build, or switch to using the new(ish) docker buildx command.

At a slightly more technical level, buildkit allows easy switching between multiple different “builders”, which can be local or remote, in the docker daemon itself, in docker containers or even in a Kubernetes pod. The builder itself is split up into two main pieces, a frontend and a backend: the frontend produces intermediate Low Level Builder (LLB) code, which is then constructed into an image by the backend.

You can think of LLB to BuildKit as the LLVM IR is to Clang.

Part of what makes buildkit so fantastic is it’s flexibility – these components are completely detached from each other, so you can use any frontend in any image. For example, you could use the default Dockerfile frontend, or compile your own self-contained buildpacks, or even develop your own alternative file format like Mockerfile.

Getting setup

To get started with using heredocs, first make sure you’re setup with buildkit. Switching to buildkit gives you a ton of out-of-the-box improvements to your build setup, and should have complete compatibility with the old builder (and you can always switch back if you don’t like it).

With buildkit properly setup, you can create a new Dockerfile: at the top of this file, we need to include a #syntax= directive. This directive informs the parser to use a specific frontend – in this case, the one located at docker/dockerfile:1.3-labs on Docker Hub.

# syntax=docker/dockerfile:1.3-labs

With this line (which has to be the very first line), buildkit will find and download the right image, and then use it to build the image.

We then specify the base image to build from (just like we normally would):

FROM ubuntu:20.04

With all that out the way, we can use a heredoc, executing two commands in the same RUN!

RUN <<EOF
echo "Hello" >> /hello
echo "World!" >> /hello
EOF

Why?

Now that heredocs are working, you might be wondering – why all the fuss? Well, this feature has kind of, until now, been missing from Dockerfiles.

See moby/moby#34423 for the original issue that proposed heredocs in 2017.

Let’s suppose you want to build an image that requires a lot of commands to setup. For example, a fairly common pattern in Dockerfiles involves wanting to update the system, and then to install some additional dependencies, i.e. apt update, upgrade and install all at once.

Naively, we might put all of these as separate RUNs:

RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get install -y ...

But, sadly like too many intuitive solutions, this doesn’t quite do what we want. It certainly works – but we create a new layer for each RUN, making our image much larger than it needs to be (and making builds take much longer).

So, we can squish this into a single RUN command:

RUN apt-get update && \
    apt-get upgrade -y && \
    apt-get install -y ...

And that’s what most Dockerfiles do today, from the official docker images down to the messy ones I’ve written for myself. It works fine, images are small and fast to build… but it does look a bit ugly. And if you accidentally forget the line continuation symbol \, well, you’ll get a syntax error!

Heredocs are the next step to improve this! Now, we can just write:

RUN <<EOF
apt-get update
apt-get upgrade -y
apt-get install -y ...
EOF

We use the <<EOF to introduce the heredoc (just like in sh/bash/zsh/your shell of choice), and EOF at the end to close it. In between those, we put all our commands as the content of our script to be run by the shell!

More ways to run…

So far, we’ve seen some basic syntax. However, the new heredoc support doesn’t just allow simple examples, there’s lots of other fun things you can do.

For completeness, the hello world example using the same syntax we’ve already seen:

RUN <<EOF
echo "Hello" >> /hello
echo "World!" >> /hello
EOF

But let’s say your setup scripts are getting more complicated, and you want to use another language – say, like Python. Well, no problem, you can connect heredocs to other programs!

RUN python3 <<EOF
with open("/hello", "w") as f:
    print("Hello", file=f)
    print("World", file=f)
EOF

In fact, you can use as complex commands as you like with heredocs, simplifying the above to:

RUN python3 <<EOF > /hello
print("Hello")
print("World")
EOF

If that feels like it’s getting a bit fiddly or complicated, you can also always just use a shebang:

RUN <<EOF
#!/usr/bin/env python3
with open("/hello", "w") as f:
    print("Hello", file=f)
    print("World", file=f)
EOF

There’s lots of different ways to connect heredocs to RUN, and hopefully some more ways and improvements to come in the future!

…and some file fun!

Heredocs in Dockerfiles also let us mess around with inline files! Let’s suppose you’re building an nginx site, and want to create a custom index page:

FROM nginx

COPY index.html /usr/share/nginx/html

And then in a separate file index.html, you put your content. But if your index page is just really simple, it feels frustrating to have to separate everything out: heredocs let you keep everything in the same place if you want!

FROM nginx

COPY <<EOF /usr/share/nginx/html/index.html
(your index page goes here)
EOF

You can even copy multiple files at once, in a single layer:

COPY <<robots.txt <<humans.txt /usr/share/nginx/html/
(robots content)
robots.txt
(humans content)
humans.txt

Finishing up

Hopefully, I’ve managed to convince you to give heredocs a try when you can! For now, they’re still only available in the staging frontend, but they should be making their way into a release very soon – so make sure to take a look and give your feedback!If you’re interested, you can find out more from the official buildkit Dockerfile syntax guide.