Stable Diffusion has gained significant attention for its ability to generate highly detailed images conditioned on text descriptions, thereby opening up new possibilities in areas such as creative design, visual storytelling, and content generation. With its open source nature and accessibility, Stable Diffusion has become a go-to tool for many researchers and developers seeking to harness the power of deep learning. 

In this article, we will explore how to optimize deep learning workflows by leveraging Stable Diffusion alongside Docker on WSL 2, enabling seamless and efficient experimentation with this cutting-edge technology.

In this comprehensive guide, we will walk through the process of setting up the Stable Diffusion WebUI Docker, which includes enabling WSL 2 and installing Docker Desktop. You will learn how to download the required code from GitHub and initialize it using Docker Compose. 

The guide provides instructions on adding additional models and managing the system, covering essential tasks such as reloading the UI and determining the ideal location for saving image output. Troubleshooting steps and tips for monitoring hardware and GPU usage are also included, ensuring a smooth and efficient experience with Stable Diffusion WebUI (Figure 1).

Why use Docker Desktop for Stable Diffusion?

In the realm of image-based generative AI, setting up an effective execution and development environment on a Windows PC can present particular challenges. These challenges arise due to differences in software dependencies, compatibility issues, and the need for specialized tools and frameworks. Docker Desktop emerges as a powerful solution to tackle these challenges by providing a containerization platform that ensures consistency and reproducibility across different systems.

By leveraging Docker Desktop, we can create an isolated environment that encapsulates all the necessary components and dependencies required for image-based generative AI workflows. This approach eliminates the complexities associated with manual software installations, conflicting library versions, and system-specific configurations.

Using Stable Diffusion WebUI

The Stable Diffusion WebUI is a browser interface that is built upon the Gradio library, offering a convenient way to interact with and explore the capabilities of Stable Diffusion. Gradio is a powerful Python library that simplifies the process of creating interactive interfaces for machine learning models.

Setting up the Stable Diffusion WebUI environment can be a tedious and time-consuming process, requiring multiple steps for environment construction. However, a convenient solution is available in the form of Stable Diffusion WebUI Docker project. This Docker image eliminates the need for manual setup by providing a preconfigured environment.

If you’re using Windows and have Docker Desktop installed, you can effortlessly build and run the environment using the docker-compose command. You don’t have to worry about preparing libraries or dependencies beforehand because everything is encapsulated within the container.

You might wonder whether there are any problems because it’s a container. I was anxious before I started using it, but I haven’t had any particular problems so far. The images, models, variational autoencoders (VAEs), and other data that are generated are shared (bind mounted) with my Windows machine, so I can exchange files simply by dragging them in Explorer or in the Files of the target container on Docker Desktop. 

The most trouble I had was when I disabled the extension without backing it up, and in a moment blew away about 50GB of data that I had spent half a day training. (This is a joke!)

Architecture

I’ve compiled a relatively simple procedure to start with Stable Diffusion using Docker Desktop on Windows. 

Prerequisites:

• Windows 10 Pro, 21H2 Build 19044.2846
• 16GB RAM
• NVIDIA GeForce RTX 2060 SUPER
• WSL 2 (Ubuntu)
• Docker Desktop 4.18.0 (104112)

Setup with Docker Compose

We will use the WebUI called AUTOMATIC1111 to utilize Stable Diffusion this time. The environment for these will be constructed using Docker Compose. The main components are shown in Figure 2.

The configuration of Docker Compose is defined in docker-compose.yml. We are using a Compose extension called x-base_service to describe the major components common to each service.

To start, there are settings for bind mount between the host and the container, including /data, which loads modes, and /output, which outputs images. Then, we make the container recognize the GPU by loading the NVIDIA driver.

Furthermore, the service named sd-auto:58 runs AUTOMATIC1111, WebUI for Stable Diffusion, within the container. Because there is a port mapping (TCP:7860), between the host and the container in the aforementioned common service settings, it is possible to access from the browser on the host side to the inside of the container.

Getting Started

Prerequisite

WSL 2 must be activated and Docker Desktop installed.

On the first execution, it downloads 12GB of Stable Diffusion 1.5 models, etc. The Web UI cannot be used until this download is complete. Depending on your connection, it may take a long time until the first startup.

Downloading the code

First, download the Stable Diffusion WebUI Docker code from GitHub. If you download it as a ZIP, click Code > Download ZIP and the stable-diffusion-webui-docker-master.zip file will be downloaded (Figure 3). 

Unzip the file in a convenient location. When you expand it, you will find a folder named stable-diffusion-webui-docker-master. Open the command line or similar and run the docker compose command inside it.

Or, if you have an environment where you can use Git, such as Git for Windows, it’s quicker to download it as follows:

git clone https://github.com/AbdBarho/stable-diffusion-webui-docker.git

In this case, the folder name is stable-diffusion-webui-docker. Move it with cd stable-diffusion-webui-docker.

Supplementary information for those familiar with Docker

If you just want to get started, you can skip this section.

By default, the timezone is UTC. To adjust the time displayed in the log and the date of the directory generated under output/txt2img to Japan time, add TZ=Asia/Tokyo to the environment variables of the auto service. Specifically, add the following description to environment:.

auto: &automatic
    <<: *base_service
    profiles: ["auto"]
    build: ./services/AUTOMATIC1111
    image: sd-auto:51
    environment:
      - CLI_ARGS=--allow-code --medvram --xformers --enable-insecure-extension-access --api
      - TZ=Asia/Tokyo

Tasks at first startup

The rest of the process is as described in the GitHub documentation. Inside the folder where the code is expanded, run the following command:

docker compose --profile download up --build

After the command runs, the log of a container named webui-docker-download-1 will be displayed on the screen. For a while, the download will run as follows, so wait until it is complete:

webui-docker-download-1  | [DL:256KiB][#4561e1 1.4GiB/3.9GiB(36%)][#42c377 1.4GiB/3.9GiB(37%)]

If the process ends successfully, it will be displayed as exited with code 0 and returned to the original prompt:

…(snip)
webui-docker-download-1  | https://github.com/xinntao/Real-ESRGAN/blob/master/LICENSE 
webui-docker-download-1  | https://github.com/xinntao/ESRGAN/blob/master/LICENSE 
webui-docker-download-1  | https://github.com/cszn/SCUNet/blob/main/LICENSE 
webui-docker-download-1 exited with code 0

If a code other than 0 comes out like the following, the download process has failed:

webui-docker-download-1  | 42c377|OK  |   426KiB/s|/data/StableDiffusion/sd-v1-5-inpainting.ckpt 
webui-docker-download-1  | 
webui-docker-download-1  | Status Legend: 
webui-docker-download-1  | (OK):download completed.(ERR):error occurred. 
webui-docker-download-1  | 
webui-docker-download-1  | aria2 will resume download if the transfer is restarted. 
webui-docker-download-1  | If there are any errors, then see the log file. See '-l' option in help/m 
an page for details. 
webui-docker-download-1 exited with code 24

In this case, run the command again and check whether it ends successfully. Once it finishes successfully, run the command to start the WebUI. 

Note: The following is for AUTOMATIC1111’s UI and GPU specification:

docker compose --profile auto up --build

When you run the command, loading the model at the first startup may take a few minutes. It may look like it’s frozen like the following display, but that’s okay:

webui-docker-auto-1  | LatentDiffusion: Running in eps-prediction mode
webui-docker-auto-1  | DiffusionWrapper has 859.52 M params.

If you wait for a while, the log will flow, and the following URL will be displayed:

webui-docker-auto-1  | Running on local URL:  http://0.0.0.0:7860

Now the startup preparation of the Web UI is set. If you open http://127.0.0.1:7860 from the browser, you can see the Web UI. Once open, select an appropriate model from the top left of the screen, write some text in the text field, and select the Generate button to start generating images (Figure 4).

When you click, the button will be reversed. Wait until the process is finished (Figure 5).

At this time, the log of image generation appears on the terminal you are operating, and you can also check the similar display by looking at the log of the container on Docker Desktop (Figure 6).

When the status reaches 100%, the generation of the image is finished, and you can check it on the screen (Figure 7).

The created images are automatically saved in the output/txt2img/date folder directly under the directory where you ran the docker compose command.

To stop the launched WebUI, enter Ctrl+C on the terminal that is still running the docker compose command.

Gracefully stopping... (press Ctrl+C again to force)
Aborting on container exit...
[+] Running 1/1
 ? Container webui-docker-auto-1  Stopped                                                     11.4s
canceled

When the process ends successfully, you will be able to run the command again. To use the WebUI again after restarting, re-run the docker compose command:

docker compose --profile auto up --build

To see the operating hardware status, use the task manager to look at the GPU status (Figure 8).

To check whether the GPU is visible from inside the container and to see whether the information comes out, run the nvidia-smi command from docker exec or the Docker Desktop terminal.

root@e37fcc5a5810:/stable-diffusion-webui# nvidia-smi 
Mon Apr 17 07:42:27 2023 
+---------------------------------------------------------------------------------------+ 
| NVIDIA-SMI 530.41.03              Driver Version: 531.41       CUDA Version: 12.1     | 
|-----------------------------------------+----------------------+----------------------+ 
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC | 
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. | 
|                                         |                      |               MIG M. | 
|=========================================+======================+======================| 
|   0  NVIDIA GeForce RTX 2060 S...    On | 00000000:01:00.0  On |                  N/A | 
| 42%   40C    P8                6W / 175W|   2558MiB /  8192MiB |      2%      Default | 
|                                         |                      |                  N/A | 
+-----------------------------------------+----------------------+----------------------+ 
+---------------------------------------------------------------------------------------+ 
| Processes:                                                                            | 
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory | 
|        ID   ID                                                             Usage      | 
|=======================================================================================| 
|    0   N/A  N/A       149      C   /python3.10                               N/A      | 
+---------------------------------------------------------------------------------------+

Adding models and VAEs

If you download a model that is not included from the beginning, place files with extensions, such as .safetensors in stable-diffusion-webui-docker\data\StableDiffusion. In the case of VAE, place .skpt files in stable-diffusion-webui-docker\data\VAE.

If you’re using Docker Desktop, you can view and operate inside on the Files of the webui-docker-auto-1 container, so you can also drag it into Docker Desktop. 

Figure 9 shows the Docker Desktop screen. It says MOUNT in the Note column, and it shares the information in the folder with the container from the Windows host side.

Now, after placing the file, a link to Reload UI is shown in the footer of the WebUI, so select there (Figure 10).

When you select Reload UI, the system will show a loading screen, and the browser connection will be cut off. When you reload the browser, the model and VAE files are automatically loaded. To remove a model, delete the model file from data\StableDiffusion.

Conclusion

With Docker Desktop, image generation using the latest generative AI environment can be done easier than ever. Typically, a lot of time and effort is required just to set up the environment, but Docker Desktop solves this complexity. If you’re interested, why not take a challenge in the world of generative AI? Enjoy!

Learn more

• Get the latest release of Docker Desktop.
• Have questions? The Docker community is here to help.
• New to Docker? Get started.

• On the OS side, Windows 11 users can now enable their GPU without participating in  the Windows Insider program. Windows 10 users still need to register.
• Nvidia CUDA drivers have been released.
• Last, the GPU support has been merged in Docker Desktop (in fact since version 3.1).

Nvidia used the term near-native to describe the performance to be expected.

Where to find the Docker images

Base Docker images are hosted at https://hub.docker.com/r/nvidia/cuda. The original project is located at https://gitlab.com/nvidia/container-images/cuda.

What they contain

The nvidia-smi utility allows users to query information on the accessible devices.

$ docker run -it --gpus=all --rm nvidia/cuda:11.4.2-base-ubuntu20.04 nvidia-smi
Tue Dec  7 13:25:19 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.00       Driver Version: 510.06       CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:01:00.0 Off |                  N/A |
| N/A    0C    P0    13W /  N/A |    132MiB /  4096MiB |     N/A      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------

The dmon function of nvidia-smi allows monitoring the GPU parameters :

$ docker exec -ti $(docker ps -ql) bash
root@7d3f4cbdeabb:/src# nvidia-smi dmon
# gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
# Idx     W     C     C     %     %     %     %   MHz   MHz
    0    29    69     -     -     -     0     0  4996  1845
    0    30    69     -     -     -     0     0  4995  1844

The nbody utility is a CUDA sample that provides a benchmarking mode.

$ docker run -it --gpus=all --rm nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -benchmark
...
> 1 Devices used for simulation
GPU Device 0: "Turing" with compute capability 7.5

> Compute 7.5 CUDA device: [NVIDIA GeForce GTX 1650 Ti]
16384 bodies, total time for 10 iterations: 25.958 ms
= 103.410 billion interactions per second
= 2068.205 single-precision GFLOP/s at 20 flops per interaction

Quick comparison to a CPU suggest a different order of magnitude of performance. GPU is 2000 times faster:

> Simulation with CPU
4096 bodies, total time for 10 iterations: 3221.642 ms
= 0.052 billion interactions per second
= 1.042 single-precision GFLOP/s at 20 flops per interaction

What can you do with a paravirtualized GPU?

Run cryptographic tools

Using a GPU is of course useful when operations can be heavily parallelized. That’s the case for hash analysis. dizcza hosted its nvidia-docker based images of hashcat on Docker hub. This image magically works on Docker Desktop!

$ docker run -it --gpus=all --rm dizcza/docker-hashcat //bin/bash
root@a6752716788d:~# hashcat -I
hashcat (v6.2.3) starting in backend information mode

clGetPlatformIDs(): CL_PLATFORM_NOT_FOUND_KHR

CUDA Info:
==========

CUDA.Version.: 11.6

Backend Device ID #1
  Name...........: NVIDIA GeForce GTX 1650 Ti
  Processor(s)...: 16
  Clock..........: 1485
  Memory.Total...: 4095 MB
  Memory.Free....: 3325 MB
  PCI.Addr.BDFe..: 0000:01:00.0

From there it is possible to run hashcat benchmark

hashcat -b
...
Hashmode: 0 - MD5
Speed.#1.........: 11800.8 MH/s (90.34ms) @ Accel:64 Loops:1024 Thr:1024 Vec:1
Hashmode: 100 - SHA1
Speed.#1.........:  4021.7 MH/s (66.13ms) @ Accel:32 Loops:512 Thr:1024 Vec:1
Hashmode: 1400 - SHA2-256
Speed.#1.........:  1710.1 MH/s (77.89ms) @ Accel:8 Loops:1024 Thr:1024 Vec:1
...

Draw fractals

The project at https://github.com/jameswmccarty/CUDA-Fractal-Flames uses CUDA for generating fractals. There are two steps to build and run on Linux. Let’s see if we can have it running on Docker Desktop. A simple Dockerfile with nothing fancy helps for that.

# syntax = docker/dockerfile:1.3-labs
FROM nvidia/cuda:11.4.2-base-ubuntu20.04
RUN apt -y update
RUN DEBIAN_FRONTEND=noninteractive apt -yq install git nano libtiff-dev cuda-toolkit-11-4
RUN git clone --depth 1 https://github.com/jameswmccarty/CUDA-Fractal-Flames /src
WORKDIR /src
RUN sed 's/4736/1024/' -i fractal_cuda.cu # Make the generated image smaller
RUN make

And then we can build and run:

$ docker build . -t cudafractal
$ docker run --gpus=all -ti --rm -v ${PWD}:/tmp/ cudafractal ./fractal -n 15 -c test.coeff -m -15 -M 15 -l -15 -L 15

Note that the --gpus=all is only available to the run command. It’s not possible to add GPU intensive steps during the build.

Here’s an example image:

Machine learning

Well really, looking at GPU usage without looking at machine learning would be a miss. The tensorflow:latest-gpu image can take advantage of the GPU in Docker Desktop. I will simply point you to Anca’s blog earlier this year. She described a tensorflow example and deployed it in the cloud: https://www.docker.com/blog/deploy-gpu-accelerated-applications-on-amazon-ecs-with-docker-compose/

Conclusion: What are the benefits for developers? 

At Docker, we want to provide a turn key solution for developers to execute their workflows seamlessly:

• With Docker Desktop, developers can run their code locally and deploy to the infrastructure of their choice.
• We provide support in the issue tracker https://github.com/docker/for-win
• See what’s coming up and recommend feature requests in the Docker public roadmap https://github.com/docker/roadmap
• Download the latest version of Docker Desktop now.

DockerCon2022

Join us for DockerCon2022 on Tuesday, May 10. DockerCon is a free, one day virtual event that is a unique experience for developers and development teams who are building the next generation of modern applications. If you want to learn about how to go from code to cloud fast and how to solve your development challenges, DockerCon 2022 offers engaging live content to help you build, share and run your applications. Register today at https://www.docker.com/dockercon/

Last week, during the Docker Community All Hands, we announced the availability of a developer preview build of Docker Desktop for WSL 2 supporting GPU for our Developer Preview Program. We already have more than 1,000 who have joined us to help test preview builds of Docker Desktop for Windows (and Mac!). If you’re interested in joining the program for future releases you should do it now!

Today we are excited to announce the general preview of Docker Desktop support for GPU with Docker in WSL2. There are over one and a half million users of Docker Desktop for Windows today and we saw in our roadmap how excited you all were for us to provide this support.

Preview of Docker Desktop with GPU support in WSL2

To get started with Docker Desktop with Nvidia GPU support on WSL 2, you will need to download our technical preview build.

Once you have the preview build installed there are still a couple of steps you will need to do to get started using your GPU: 

• You will need access to a PC with an Nvidia GPU (if you don’t have this we would still like feedback on this build, we have changed a fair bit in our VM to get this working!) 
• The latest Windows Insider version from the Dev preview ring 
• Beta drivers from Nvidia supporting WSL 2 GPU paravirtualization: https://developer.nvidia.com/cuda/wsl
• WSL 2 backend enabled in Docker Desktop

Once you have all of this working you can have a go at the command below to check that GPU support is working. 

Keep in mind that this is a technical preview release: it may break, it has not been tested as thoroughly as our normal releases and ‘here be dragons’. 

If you do find issues and want to give us feedback, please raise bugs on our public repo https://github.com/docker/for-win. We use this feedback to improve the product and your support in testing this will help us get this ready for GA sooner 🙂 

Enjoy the tech preview and happy GPU Hacking!

With this announcement, the launch of the Windows 2004 release imminently and Docker Desktop v2.3.02 reaching WSL2 GA , we thought this would be a good time to reflect on how we got to where we are today with WSL 2.

April 2019

Casting our minds back to 2019 (a very different time!), we first discussed WSL 2 with Microsoft in April. We were excited to get started and wanted to find a way to get a build as soon as possible.

May 2019

It turned out the easiest way to do this was to collect a laptop at Kubecon EU (never underestimate the bandwidth of a 747). We brought this back and started work on what would be our first ‘hacky’ version of WSL 2 for Docker Desktop.

June 2019

With some internal demo’s done we decided to announce what we were planning <3

This announcement was a bit like watching a swan on a lake, our blog post was calm and collected, but beneath the water we were kicking madly to take us towards something we could share more widely.

July 2019

We finally got far enough along that we were ready to share something!

And not long after we released our first preview of Docker Desktop using WSL 2

August-September 2019

Once again, with a preview out and things seeming calm we went back to work. We were talking with Microsoft weekly about how we could improve what we had, on fixing bugs and generally on improving the experience. Simon and Ben did have enough time though to head over to the USA to talk to Microsoft about how we were getting on.

October 2019

We released a major rework to how Docker Desktop would integrate with WSL 2:

Along with adding K8s support and providing feature parity with our old Hyper-V based back end. We also made the preview more visible in Docker Desktop and our user numbers started to increase

November 2019 – Feb 2020

This time flew by, we spent a lot of this time chasing down bugs, looking at how we could improve the local experience and also what the best ways of working would be:

March 2020

We had built up a fair bit of confidence in what we had built and finally addressed one of the largest outstanding items we still had in our backlog – we added Windows Home support

This involved us removing the functionality associated with running our old Moby VM in Hyper V and all of the options associated with running Windows containers – as these are not supported on Windows Home. With this we were now able to focus on heading straight to GA…

April 2020

We doubled down how we were getting ready for GA, learning lessons on improving our development practice. We wanted to share how we were preparing and testing WSL 2 ready for the 2.7m people out there running Docker Desktop.

May 2020

We finally reached our GA with Docker Desktop v2.3.02!

Now we are out in the wild, we shared some ideas and best practices to make sure you are getting the best experience out of Docker Desktop when working with WSL 2. 

(And of course for Windows Pro users this still comes with all the same features you know and love including the ability to switch back over to using Windows Containers.)

What’s next?

Next, is that people start to use Docker Desktop with WSL 2! To try out Docker Desktop with WSL 2 today, make sure you are on Windows 2004 or higher and download the latest Docker Desktop to get started.

If you are enjoying  Docker Desktop but have ideas of what we could do to make it better then please give us feedback. You can let us know what features you want to see next via our roadmap, including voting up GPU support for WSL 2. 

Docker Desktop with the WSL 2 backend can be used as before from a Windows terminal. We focused on compatibility to keep you happy with your current development workflow.

But to get the most out of Windows 10 2004 we have some recommendations for you.

Fully embrace WSL 2

The first and most important best practice we want to share, is to fully embrace WSL 2. Your project files should be stored within your WSL 2 distro of choice, you should run the docker CLI from this distro, and you should avoid accessing files stored on the Windows host as much as possible.

For backward compatibility reasons, we kept the possibility to interact with Docker from the Windows CLI, but it is not the preferred option anymore.

Running docker CLI from WSL will bring you…

Awesome mounts performance

Both your own WSL 2 distro and docker-desktop run on the same utility VM. They share the same Kernel, VFS cache etc. They just run in separate namespaces so that they have the illusion of running totally independently. Docker Desktop leverages that to handle bind mounts from a WSL 2 distro without involving any remote file sharing system. This means that when you mount your project files in a container (with docker run -v ~/my-project:/sources <...>), docker will propagate inotify events and share the same cache as your own distro to avoid reading file content from disk repeatedly.

A little warning though: if you mount files that live in the Windows file system (such as with docker run -v /mnt/c/Users/Simon/windows-project:/sources <...>), you won’t get those performance benefits, as /mnt/c is actually a mountpoint exposing Windows files through a Plan9 file share. 

Compatibility with Linux toolchains and build scripts

Most reasonably sized projects involving Linux containers come with a bunch of automation scripts. Those scripts are often developed for Linux first (because most of the time, CI/CD pipelines for those projects run on Linux), and developers running on Windows are often considered second-class citizens. They are often using less polished versions of those scripts, and have to deal with subtle behavioral differences.

By fully embracing WSL 2, Windows developers can use the exact same build and automation scripts as on Linux. This means that Windows-specific scripts don’t need to be maintained anymore. Also, that means that you won’t experience issues with different line endings between Windows and Mac/Linux users!

What about my IDE?

If you want an IDE for editing your files, you can do that even if they are hosted within your WSL 2 distro. There are 3 different ways:

• Use Visual Studio Code Remote to WSL extension

If your IDE is Visual Studio Code, using Remote to WSL is the best way to continue working on your project. Visual Studio Code architecture is based on a client/server approach where pretty much everything except rendering and input processing is done in a server process, while the UI itself runs in a client process. Remote to WSL leverages that to run the whole server process within WSL while the UI runs as a classic win32 process.

That means that you get the same experience as before, but all your language services, terminals etc. run within WSL.

For more information, see Microsoft’s VS Code Remote to WSL documentation.

• Point your IDE to your distro network share

WSL provides a network share for each of your running distros. For example, if I have a project in my Ubuntu distro at `~/wall-e`, I can access it from Windows Explorer (and from any Windows Process) via the special network share `\\wsl$\Ubuntu\home\simon\wall-e`. 

• Run an X11 server on Windows, and run a Linux native IDE

The setup is a bit more complicated, but you always have the possibility to run an X11 server on Windows (VcXsrv, X410,…) and configure your DISPLAY environment variable such that GUI apps on Linux get rendered properly.

Use BuildKit and multi-stage builds

Docker Desktop WSL 2 backend has access to all your CPU cores. To leverage this as much as possible (and also to get access to the latest build features), you should enable BuildKit by default.

The easiest way to do that is to add the following line to your ~/.profile file:

export DOCKER_BUILDKIT=1.

This way, anytime you run docker build, it will run the build with the awesome BuildKit which is capable of running different build stages concurrently.

Use resource limits

Docker Desktop WSL 2 backend can use pretty much all CPU and memory resources on your machine. This is awesome for most cases, but there is a category of workloads where this can cause issues. Indeed, some containers (mainly databases, or caching services) tend to allocate as much memory as they can, and leave other processes (Linux or Win32) starving. Docker provides a way to impose limits in allocatable memory (as well as quotas on CPU usage) by a container. You can find documentation about it here: https://docs.docker.com/config/containers/resource_constraints/.

Reclaim cached memory

WSL 2 automatically reclaims memory when it is freed, to make it available to Windows processes. However, if the kernel decides to keep content in cache (and with Docker, it tends to happen quite a lot), the amount of memory reclaimed might not be sufficient.

To reclaim more memory, after stopping your containers, you can run echo 1 > /proc/sys/vm/drop_caches as root to drop the kernel page cache and make WSL 2 reclaim memory used by its VM.

What next

We are excited for people to use Docker Desktop with WSL 2 and hope that the tips and tricks in this article will help you get the best performance for all of your workloads. 

If you have another tip or idea you want to share with us for using Docker send us a tweet @docker or if you have feedback on our implementation then raise a ticket against our Github Repo.

One of the great benefits is that with the next update of Windows 10 we will also support running Docker Desktop on Windows 10 Home. We worked closely with Microsoft during the last few months to make Docker Desktop and WSL 2 fit together.

In this blog post we look behind the scenes at how we set up new WSL 2 capable test machines to run automated tests in our CI pipeline.

It started with a laptop

Let’s keep in mind that all automation somehow starts with manual steps and you evolve from there to get better and more automated. At the beginning of this project we were given a laptop back at KubeCon 2019 with an early version of WSL 2.

With that single laptop our development team could start getting their hands on that new feature and integrating it into Docker Desktop. But of course, this doesn’t really scale for a whole team and we also needed automated tests.

The Docker Desktop test matrix

In the Docker Desktop team we run several test suites across several Windows and Mac machines with different operating system versions installed. Each code change is tested with a matrix of tests on selected machines.

One of our challenges was to add Windows machines to this matrix with WSL 2 enabled. At that time the Windows Insider program started to ship first releases and we could start automating the process to keep new test machines up to date.

On-demand test runners

The startup time of Docker Desktop is much faster with the WSL 2 backend. This gave us the option to run the end-to-end tests in virtual machines. We enhanced our CI infrastructure to spin up Windows 10 Insider machines in Azure on demand. This gave us more flexibility to keep the test machines at a working version of WSL 2 in our pool and also trying out the latest Insider builds.

Our internal CI dashboard shows all the test machines and the jobs running on them changed every few weeks. We constantly moved from one Insider release to the next. Currently we are concentrating on the final Slow Ring builds 19041.x, but we are also continuing with the next Fast Ring machines to have feedback from upcoming Windows builds.

Automated pipeline to build the test machines

The Azure VM images we use to spin up WSL 2 test machines are created with a separate CI pipeline. We use Packer to create the VM image from an ISO file and run provision scripts to prepare everything we need to run it as a CI runner. The pipeline of how we build and upload the VM image also contains more than just the build step. We first check the source code of the Packer template and the PowerShell and Unix shell scripts to fail fast if a code change broke something. The Packer build itself takes the longest time, it also runs a Windows Update in the VM to get the latest OS version. After the build we added a verification step using InSpec to check if the software we need is installed correctly.

The output of this Packer pipeline is an Azure VM image that can be used to spin up new on-demand runners in other CI pipelines. We normally run some tests in a canary environment to see if the VM image really boots up and attaches to our CI infrastructure. If everything is fine we update the configuration for the Docker Desktop CI for our end-to-end tests.

A new challenge: Windows 10 Home

With that automation for Windows 10 Pro machines at hand we were able to add Windows 10 Home very easily. Of course there were some challenges, for example Windows 10 Home does not provide Remote Desktop support. We added a VNC server to be able to attach to the cloud runners if we want to investigate problems.

Conclusion

In the last 12 months the Docker Desktop team worked hard to bring not only the WSL 2 support to Docker Desktop, but also enabled Windows 10 Home users to easily run Docker on their machines. We really look forward to the official release of Windows 10, version 2004 and love to hear your feedback.

Feedback on this first version of Docker Desktop for Windows Home is welcomed! To get started, you will need to be on Windows Insider Preview build 19040 or higher and install the Docker Desktop Edge 2.2.2.0.

What’s in Docker Desktop for Windows Home?

Docker Desktop for WSL 2 Windows Home is a full version of Docker Desktop for Linux container development. It comes with the same feature set as our existing Docker Desktop WSL 2 backend. This gives you: 

• Latest version of Docker on your Windows machine 
• Install Kubernetes in one click on Windows Home 
• Integrated UI to view/manage your running containers 
• Start Docker Desktop in <5 seconds
• Use Linux Workspaces
• Dynamic resource/memory allocation 
• Networking stack, support for http proxy settings, and trusted CA synchronization 

How do I get started developing with Docker Desktop? 

For the best experience of developing with Docker and WSL 2, we suggest having your code inside a Linux distribution. This improves the file system performance and thanks to products like VSCode mean you can still do all of your work inside the Windows UI and in an IDE you know and love. 

Firstly make sure you are on the Windows insider program, are on 19040 and have installed Docker Desktop Edge.

Next install a WSL distribution of Linux (for this example I will assume something like Ubuntu from the Microsoft store).

You may want to check your distro is set to V2, to check in powershell run

If you see your distro is a version one you will need to run 

Once you have a V2 WSL distro, Docker Desktop will automatically set this up with Docker.

The next step is to start working with your code inside this Ubuntu distro and ideally with your IDE still in Windows. In VSCode this is pretty straightforward.

You will want to open up VSCode and install the Remote WSL extension, this will allow you to work with a remote server in the Linux distro and your IDE client still on Windows. 

 Now we need to get started working in VSCode remotely, the easiest way to do this is to open up your terminal and type:

This will open a new VSCode connected remotely to your default distro which you can check in the bottom corner of the screen. 

(or you can just look for Ubuntu in your start menu, open it and then run  code . )

Once in VSCode there I use the terminal in VSCode to pull my code and start working natively in Linux with Docker from my Windows Home Machine!

Other tips and tricks:

• If you want to get the best out of the file system performance avoid mounting from the windows file system (even from a WSL distro. eg: avoid docker run -v /mnt/c/users:/users)
• If you are worried about the size of the docker-desktop-data VHDX or need to change it you can do this through the WSL tooling built into Windows:
  https://docs.microsoft.com/en-us/windows/wsl/wsl2-ux-changes#understanding-wsl-2-uses-a-vhd-and-what-to-do-if-you-reach-its-max-size 
• If you are worried about CPU/Memory usage you put limits on memory/cpu/swap size on the WSL2 utility VM https://docs.microsoft.com/en-us/windows/wsl/release-notes#build-18945 

Your feedback needed!

We are excited to get your feedback on the first version of Docker Desktop for Windows Home and for you to tell us how we can make it even better.

To get started with WSL 2 Docker Desktop on Windows home today you will need to be on Windows Insider Preview build 19040 or higher and install the Docker Desktop Edge 2.2.2.0.

WSL 2 (or Windows Subsystem for Linux version 2) is Microsoft’s second take on shipping a Linux Kernel with Windows. The first version was awesome as it translated Linux system calls to the equivalent Windows NT call in real time. The second version includes a full fledged virtual machine. 

It was only natural that Docker would embrace this change and ship a Docker Desktop for Windows version that runs on WSL 2 (WSL 1 had issues running the Docker daemon). This is still a Technical Preview, but after using it for a couple of days, I’ve completely switched my local development to take advantage of it and I’m pretty happy with it.

In this blog, I’ll show you an example of how to develop Docker-powered applications using the Docker Desktop WSL 2 Tech Preview.

Why use Docker Desktop WSL 2 Tech Preview over the “stable” Docker Desktop for Windows?

The main advantage of using the technical preview is that you don’t have to manage your Docker VM anymore.  More specifically:

• The VM grows and shrinks with your needs in terms of RAM/CPU, so you don’t have to decide its size and preallocate resources. It can shrink to almost zero CPU/RAM if you don’t use it. It works so well that most of the time you forget there’s a VM involved.
• Filesystem performance is great, with support for inotify and the VM’s disk size can match the size of your machine’s disk.

Apart from the above, if you love Visual Studio Code like I do, you can use the VS Code Remote WSL plugin to develop Docker-powered applications locally (more on that in a bit). You also get the always awesome Docker developer experience while using the VM.

How does Docker Desktop for WSL 2 Tech Preview work?

When you install it, it automatically installs Docker in a managed directory in your default WSL 2 distribution. This installation includes the Docker daemon, the Docker CLI and the Docker Compose CLI. It is kept up to date with Docker Desktop and you can either access it from within WSL, or from PowerShell by switching contexts — see, Docker developer experience in action!

Developing applications with Docker Desktop for WSL 2 Tech Preview

For this example, we’ll develop a simple Python Flask application, with Redis as its data store. Every time you visit the page, the page counter will increase — say hello to Millennium!

Setting up VS Code Remote – WSL

Visual Studio Code recently announced a new set of tools for developing applications remotely — using SSH, Docker or WSL. This splits Visual Studio Code into a “client-server” architecture, with the client (that is the UI) running on your Windows machine and the server (that is your code, Git, plugins, etc) running remotely. In this example, we’re going to use the WSL version.

To start, open VS Code and select “Remote-WSL: New Window”. This will install the VS Code Remote server in your default WSL distribution (the one running Docker) and open a new VS Code workspace in your HOME directory.

Getting and Exploring the Code

Clone this Github repository by running git clone https://github.com/akalipetis/python-docker-example. Next, run code -r python-docker-example to open this directory in VS Code and let’s go a quick tour!

Dockerfile and docker-compose.yml

These should look familiar. The Dockerfile is used for building your application container, while docker-compose.yml is the one you could use for deploying it. docker-compose.override.yml contains all the things that are needed for local development.

Pipfile and Pipfile.lock

These include the application dependencies. Pipenv is the tool used to manage them.

The app.py  file contains the Flask application, which we’re just using in this example. Nothing special here!

Running the application and making changes

In order to run the application, open a WSL terminal (this is done using the integrated terminal feature of VS Code) and run docker-compose up. This will start all the containers (in this case, a Redis container and the one running the application). After doing so, visit http://localhost:5000 in your browser and voila — you’ve visited your new application. That’s not development though, so let’s change and see it in action. Open the app.py in VS Code and change the following line:

Refresh the web page and observe that:

1. The message was immediately changed
2. The visit counter continued counting from the latest value

Under the hood

Let’s see what actually happened.

• We changed a file in VS Code, which is running on Windows.
• Since VS Code is running on a client-server mode with the server running is WSL 2, the change was actually made to the file living inside WSL.
• Since you’re using the Technical Preview of Docker Desktop for WSL 2 and docker-compose.override.yml is using Linux workspaces to mount the code from WSL 2 directly into the running container, the change was propagated inside the container.
  • While this is possible with the “stable” Docker Desktop for Windows, it isn’t as easy. By using Linux workspaces, we don’t need to worry about file system permissions. It’s also super fast, as it’s a local Linux filesystem mount.
• Flask is using an auto-reloading server by default, which — using inotify — is reloading the server on every file change and within milliseconds from saving your file, your server was reloaded.
• Data is stored in Redis using a Docker volume, thus the visits counter was not affected by the restart of the server.

Other tips to help you with Docker Desktop for WSL 2

Here are a few additional tips on developing inside containers using the Technical Preview of Docker Desktop for WSL 2:

• For maximum file system performance, use Docker volumes for your application’s data and Linux Workspaces for your code.
• To avoid running an extra VM, switch to Windows containers for your “stable” Docker Desktop for Windows environment.
• Use docker context default|wsl to switch contexts and develop both Windows and Linux Docker-powered applications easily.

Final Thoughts

I’ve switched to Windows and WSL 2 development for the past two months and I can’t describe how happy I am with my development workflow. Using Docker Desktop for WSL 2 for the past couple of days seems really promising, and most of the current issues of using Docker in WSL 2 seem to be resolved. I can’t wait for what comes next!

The only thing currently missing in my opinion is integration with VS Code   Remote Containers (instead of Remote WSL which was used for this blogpost) would allow you to run all your tooling within your Docker container.

Until VS Code Remote Containers support is ready, you can run pipenv install --dev to install the application dependencies on WSL 2, allowing VS Code to provide auto-complete and use all the nice tools included to help in development.

Get the Technical Preview and Learn More

If you’d like to get on board, read the instructions and install the technical preview from the Docker docs.

For more on WSL 2, check out these blog posts:

• 5 Things to Try with Docker Desktop WSL 2 Tech Preview
• Get Ready for the Tech Preview of Docker Desktop for WSL 2

Developing #Docker Powered Apps on Windows with #WSL2 by Docker Captain @akalipetis
Click To Tweet