DEV Community

wmchurchill3 for Leading EDJE

Posted on

A Window into Docker, minikube, and containerd

Container Runtime Logos
Like many of you, I received an email from Docker notifying me of their changes to service. Having used Docker Desktop for many years as part of my work, I was a little concerned. My concern was not great enough to do anything... Until a co-worker suggested an article switching from Docker for Windows to containerd. This link from 2018 seemed to suggest containerd could run on Windows.

Spoiler Alert/TL;DR: This is not a post about getting containerd running on Windows. I was able to get a Windows nanoserver image running in containerd. I could not get that image connecting to any network. This post is a survey of the source code, GitHub issues, and dead links chased. All documented to show how close and far away we are to something useful.

Where does minikube fit in here?

In my research and frustration, I wanted to try running something else. I enabled Hyper-V on my machine. Followed the instructions at minikube quickstart. Things worked! Thank you to the maintainers of minikube! Great Job! I definitely will be using this more in the future.

The only place I deviated was in starting the minikube cluster. I used the command minikube start --driver=hyperv --container-runtime=containerd. For fun, I checked the Hyper-V Manager and saw a new virtual machine named 'minikube'. Then it hit me. A Linux VM hosts the minikube cluster complete with its own version of containerd. This means I could not run a Windows image!

The Journey Begins

The first stop was the Container Platform Tools on Windows. This is where the dead links begin (see the Links to CRI Spec). My second stop was the containerd site. I downloaded and installed the requirements and release tarball. When the compiling started, I ran into an issue with make looking for gcc. This seemed odd since 1) it is a Go application, 2) having gcc on Windows seems like a high bar for running containers.

Some more Googling brought me to James Sturtevant's site. This made me aware pre-built Windows containerd binaries exist. Now I was making some progress.

The following code snippet will download and configure containerd as a service. Each line does the following:

  1. Download the latest (as of 20210924) release of containerd
  2. Make a directory for the containerd binaries and configs
  3. Expand the containerd tarball
  4. Move the binaries to the directory created above
  5. Add containerd to the Path environment variable
  6. Create a default containerd configuration in the containerd directory
  7. Tell Windows Defender not worry about the containerd executable
  8. Register containerd as a service
  9. Start containerd

In a Admin PowerShell window,

curl.exe -LO https://github.com/containerd/containerd/releases/download/v1.5.5/containerd-1.5.5-linux-amd64.tar.gz
mkdir "C:\Program Files\containerd"
tar -xzf containerd-1.5.5-linux-amd64.tar.gz
mv .\bin\* "C:\Program Files\containerd"
$env:Path = $env:Path + ';C:\Program Files\containerd'
containerd.exe config default | Set-Content "C:\Program Files\containerd\config.toml" -Force
Add-MpPreference -ExclusionProcess "$Env:ProgramFiles\containerd\containerd.exe"
.\containerd.exe --register-service
Start-Service containerd
Enter fullscreen mode Exit fullscreen mode

To verify containerd is running:

  1. Open the Task Manager
  2. Go into the More Details view
  3. Scroll to Background Processes
  4. You should see a containerd.exe process Task Manager Process Listing

Running a Container

Under ideal circumstances, we would pull an image using the ctr command.

.\ctr.exe pull docker.io/library/mcr.microsoft.com/windows/nanoserver:10.0.19042.1165-amd64`
Enter fullscreen mode Exit fullscreen mode

Unfortunately, there is some authentication around the Microsoft images. Assuming you have one downloaded using Docker, we can

  1. Save the image
  2. Import the image using ctr
  3. Run the image. From the Admin PowerShell window,
docker save mcr.microsoft.com/nanoserver:10.0.19042.1165-amd64 -o nanoserver.tar
.\ctr.exe image import  --all-platforms c:\wherever\you\put\this\nanoserver.tar
.\ctr.exe run -rm mcr.microsoft.com/windows/nanoserver:10.0.19042.1165-amd64 test cmd /c echo hello
Enter fullscreen mode Exit fullscreen mode

If you see hello on the next line immediately after the command, Success!

That's it, right?
Lee Corso, Not So Fast Gif

We have a container running a Windows image, but no network.

Creating A Network for the containers

We need extra setup for networking our pods. CNI (Container Networking Interface) will provide NAT'ing for our dev environment. We also must get a helper script to set up the network. The steps:

  1. Get the CNI tools executables
  2. Get the helper script hns.psm1
  3. Create some directories
  4. Expand the CNI tools into the created directories.
  5. Allow your machine to execute scripts
  6. Unblock the helper script, hns.psm1
  7. Import hsn.psm1 for use. Disregard the warning about verbs. This is a naming convention.

From the PowerShell window,

curl.exe -LO https://github.com/microsoft/windows-container-networking/releases/download/v.0.2.0/windows-container-networking-cni-amd64-v0.2.0.zip
curl.exe -LO https://raw.githubusercontent.com/microsoft/SDN/master/Kubernetes/windows/hns.psm1
mkdir -force "C:\Program Files\containerd\cni\bin"
mkdir -force "C:\Program Files\containerd\cni\conf"
Expand-Archive windows-container-networking-cni-amd6464-v0.2.0.zip -DestinationPath "C:\Program Files\containerd\cni\bin" -Force
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope LocalMachine
Unblock-File -Path .\hns.psm1
ipmo .\hns.psm1 
Enter fullscreen mode Exit fullscreen mode

Now to configure the network. From the Admin PowerShell window,

$subnet="10.0.0.0/16"
$gateway="10.0.0.1"
New-HNSNetwork -Type Nat -AddressPrefix $subnet -Gateway $gateway -Name "nat"
Enter fullscreen mode Exit fullscreen mode

In this case, the name must be nat.
Let's check our work. From the PowerShell window:

netsh lan show profiles
Enter fullscreen mode Exit fullscreen mode

You should see the new 'nat' network.

Profile on interface vEthernet (nat)
=======================================================================
Applied: User Profile

    Profile Version        : 1
    Type                   : Wired LAN
    AutoConfig Version     : 1
    802.1x                 : Enabled
    802.1x                 : Not Enforced
    EAP type               : Microsoft: Protected EAP (PEAP)
    802.1X auth credential : [Profile credential not valid]
    Cache user information : [Yes]
Enter fullscreen mode Exit fullscreen mode

If you get an error about dot3svc not running, run net start dot3svc and run the netsh command again.

Configure containerd to use that network. From the Admin PowerShell window,

@"
{
    "cniVersion": "0.2.0",
    "name": "nat",
    "type": "nat",
    "master": "Ethernet",
    "ipam": {
        "subnet": "$subnet",
        "routes": [
            {
                "gateway": "$gateway"
            }
        ]
    },
    "capabilities": {
        "portMappings": true,
        "dns": true
    }
}
"@ | Set-Content "C:\Program Files\containerd\cni\conf\0-containerd-nat.conf" -Force
Enter fullscreen mode Exit fullscreen mode

Container Runtime Interface (CRI)

We are in the endgame now. I promise. From the README, crictl provides a CLI for CRI-compatible container runtimes.
The following snippet performs the following:

  1. Download the crictl executable.
  2. Creates the default location for crictl to look for a configuration
  3. Creates the configuration

From a PowerShell,

curl.exe -LO https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.20.0/crictl-v1.20.0-windows-amd64.tar.gz                          
tar -xvf crictl-v1.20.0-windows-amd64.tar.gz
mkdir $HOME\.crictl
@"
runtime-endpoint: npipe://./pipe/containerd-containerd
image-endpoint: npipe://./pipe/containerd-containerd
timeout: 10
#debug: true
"@ | Set-Content "$HOME\.crictl\crictl.yaml" -Force
Enter fullscreen mode Exit fullscreen mode

The Payoff

Using a pod.json of

{
    "metadata": {
        "name": "nanoserver-sandbox",
        "namespace": "default",
        "uid": "hdishd83djaidwnduwk28bcsb"
    },
    "logDirectory": "/tmp",
    "linux": {}
}
Enter fullscreen mode Exit fullscreen mode

The magic happens with this command:

$POD_ID=(./crictl runp .\pod.json)
$CONTAINER_ID=(./crictl create $POD_ID .\container.json .\pod.json)
./crictl start $CONTAINER_ID
Enter fullscreen mode Exit fullscreen mode

The Problem

Running the .\crictl runp .\pod.json creates a sandbox pod for use in creating a container in the next command. The runp command fails setting up the network adapter for the pod. The output is as follows:

time="2021-09-22T09:25:29-04:00" level=debug msg="get runtime connection"
time="2021-09-22T09:25:29-04:00" level=debug msg="connect using endpoint 'npipe://./pipe/containerd-containerd' with '10s' timeout"
time="2021-09-22T09:25:29-04:00" level=debug msg="connected successfully using endpoint: npipe://./pipe/containerd-containerd"
time="2021-09-22T09:25:29-04:00" level=debug msg="RunPodSandboxRequest: &RunPodSandboxRequest{Config:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:nanoserver-sandbox,Uid:hdishd83djaidwnduwk28bcsb,Namespace:default,Attempt:0,},Hostname:,LogDirectory:,DnsConfig:nil,PortMappings:[]*PortMapping{},Labels:map[string]string{},Annotations:map[string]string{},Linux:&LinuxPodSandboxConfig{CgroupParent:,SecurityContext:nil,Sysctls:map[string]string{},},},RuntimeHandler:,}"
time="2021-09-22T09:25:29-04:00" level=debug msg="RunPodSandboxResponse: nil"
time="2021-09-22T09:25:29-04:00" level=fatal msg="run pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox \"e4cc6fc22dbdf8ccde0035239873cb9f31b074fca4650acc545a8af5a51d814c\": error creating endpoint hcnCreateEndpoint failed in Win32: IP address is either invalid or not part of any configured subnet(s). (0x803b001e) {\"Success\":false,\"Error\":\"IP address is either invalid or not part of any configured subnet(s). \",\"ErrorCode\":2151350302} : endpoint config &{ e4cc6fc22dbdf8ccde0035239873cb9f31b074fca4650acc545a8af5a51d814c_nat 11d59574-13be-4a14-b3e8-11cc0d5a7805  [] [{ 0}] { [] [] []} [{10.0.0.1 0.0.0.0/0 0}]  0 {2 0}}"
Enter fullscreen mode Exit fullscreen mode

There is a GitHub Issue that hints to a problem with the pod network workflow on Windows

Conclusion

There is a good possibility this issue will remain for a while. It has been around for the better part of a year. If one is running Linux containers, there is a great substitute in minikube. It is easy to setup, well documented, maintained, and simulates a production environment. It appears Windows images will still need to run on Docker. Please leave a comment below if you are able to find a workaround.

Relevant Links

GitHub Issue: Windows CNI plugin has no chance to create and configure container VNIC
James Sturtevant's Windows Containers on Windows 10 without Docker using Containerd
PowerShell Execution Policies
minikube
crictl README has pod.json samples


Smart EDJE Image

Discussion (0)