After some failed attempts of adding GPU support to k3s, this article describes how to boot up a worker node with NVIDIA GPU support.
k3s, for those who are new to it, is a very small kubernetes distribution.
There are a few reasons why adding GPU support is not that easy. Main reason is that k3s is using containerd as it's container runtime. Most tutorials and also the official NVIDIA k8s device plugin assume docker as the container runtime. While you can easily switch to docker in k3s, we didn't want to change the runtime itself.
Kubernetes itself has a guide for adding GPU support which outlines the basic steps.
The following recipe has been tested on GCP instances n2-standard-1 with a NVIDIA Tesla T4 GPU attached.
It assumes a running master node. Each worker with an attached GPU needs a few additional steps which are outlined below.
Create device plugin DaemonSet
The device plugin is responsible for advertising the nvidia.com/gpu
resource on a node (via kubelet).
This needs to be done on the kubernetes node only once. Every node with the label cloud.google.com/gke-accelerator
then gets automatically a pod from this DaemonSet assigned.
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/kubernetes/release-1.14/cluster/addons/device-plugins/nvidia-gpu/daemonset.yaml
The following steps are necessary for each node which needs GPU support. Placing it in a startup script is a good option.
Install drivers
# required kernel module
modprobe ipmi_devintf
# add necessary repositories
add-apt-repository -y ppa:graphics-drivers
curl -s -L https://nvidia.github.io/nvidia-container-runtime/gpgkey | \
apt-key add -
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-container-runtime/$distribution/nvidia-container-runtime.list | \
tee /etc/apt/sources.list.d/nvidia-container-runtime.list
apt-get update
# install graphics driver
apt-get install -y nvidia-driver-440 nvidia-container-runtime nvidia-modprobe
Ensure nvidia driver is loaded and device files ready
From: https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#runfile-verifications
/sbin/modprobe nvidia
if [ "$?" -eq 0 ]; then
# Count the number of NVIDIA controllers found.
NVDEVS=`lspci | grep -i NVIDIA`
N3D=`echo "$NVDEVS" | grep "3D controller" | wc -l`
NVGA=`echo "$NVDEVS" | grep "VGA compatible controller" | wc -l`
N=`expr $N3D + $NVGA - 1`
for i in `seq 0 $N`; do
mknod -m 666 /dev/nvidia$i c 195 $i
done
mknod -m 666 /dev/nvidiactl c 195 255
else
exit 1
fi
/sbin/modprobe nvidia-uvm
if [ "$?" -eq 0 ]; then
# Find out the major device number used by the nvidia-uvm driver
D=`grep nvidia-uvm /proc/devices | awk '{print $1}'`
mknod -m 666 /dev/nvidia-uvm c $D 0
else
exit 1
fi
Install k3s
curl -sfL https://get.k3s.io | \
INSTALL_K3S_SKIP_START=true \
K3S_URL=https://IP_OF_MASTER_ADDRESS:6443 \
K3S_TOKEN=CONTENT_OF_/var/lib/rancher/k3s/server/node-token_ON_MASTER \
sh -s - \
--node-label "cloud.google.com/gke-accelerator=$(curl -fs "http://metadata.google.internal/computeMetadata/v1/instance/attributes/gpu-platform" -H "Metadata-Flavor: Google")"
INSTALL_K3S_SKIP_START
prevents k3s from starting, as we need first to change containerd config (see below)
node-label
should be set to that key, the value is only important if you want to schedule pods based on the GPU available. The example here uses a metadata attribute on the instance within GCE. Feel free to change that to something else or just true
.
Configure containerd
Containerd needs to be changed to use a different container runtime. This can be achieved by adjusting the config.toml
or rather creating a config.toml.tmpl
file.
mkdir -p /var/lib/rancher/k3s/agent/etc/containerd/
# why "EOF":
# https://serverfault.com/questions/399428/how-do-you-escape-characters-in-heredoc
# ($ signs would need to be escaped -> use "EOF" instead of EOF)
cat <<"EOF" > /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl
[plugins.opt]
path = "{{ .NodeConfig.Containerd.Opt }}"
[plugins.cri]
stream_server_address = "127.0.0.1"
stream_server_port = "10010"
{{- if .IsRunningInUserNS }}
disable_cgroup = true
disable_apparmor = true
restrict_oom_score_adj = true
{{end}}
{{- if .NodeConfig.AgentConfig.PauseImage }}
sandbox_image = "{{ .NodeConfig.AgentConfig.PauseImage }}"
{{end}}
{{- if not .NodeConfig.NoFlannel }}
[plugins.cri.cni]
bin_dir = "{{ .NodeConfig.AgentConfig.CNIBinDir }}"
conf_dir = "{{ .NodeConfig.AgentConfig.CNIConfDir }}"
{{end}}
[plugins.cri.containerd.runtimes.runc]
# ---- changed from 'io.containerd.runc.v2' for GPU support
runtime_type = "io.containerd.runtime.v1.linux"
# ---- added for GPU support
[plugins.linux]
runtime = "nvidia-container-runtime"
{{ if .PrivateRegistryConfig }}
{{ if .PrivateRegistryConfig.Mirrors }}
[plugins.cri.registry.mirrors]{{end}}
{{range $k, $v := .PrivateRegistryConfig.Mirrors }}
[plugins.cri.registry.mirrors."{{$k}}"]
endpoint = [{{range $i, $j := $v.Endpoints}}{{if $i}}, {{end}}{{printf "%q" .}}{{end}}]
{{end}}
{{range $k, $v := .PrivateRegistryConfig.Configs }}
{{ if $v.Auth }}
[plugins.cri.registry.configs."{{$k}}".auth]
{{ if $v.Auth.Username }}username = "{{ $v.Auth.Username }}"{{end}}
{{ if $v.Auth.Password }}password = "{{ $v.Auth.Password }}"{{end}}
{{ if $v.Auth.Auth }}auth = "{{ $v.Auth.Auth }}"{{end}}
{{ if $v.Auth.IdentityToken }}identitytoken = "{{ $v.Auth.IdentityToken }}"{{end}}
{{end}}
{{ if $v.TLS }}
[plugins.cri.registry.configs."{{$k}}".tls]
{{ if $v.TLS.CAFile }}ca_file = "{{ $v.TLS.CAFile }}"{{end}}
{{ if $v.TLS.CertFile }}cert_file = "{{ $v.TLS.CertFile }}"{{end}}
{{ if $v.TLS.KeyFile }}key_file = "{{ $v.TLS.KeyFile }}"{{end}}
{{end}}
{{end}}
{{end}}
EOF
Start k3s agent
Start k3s: systemctl start k3s-agent
That's it! :)
Top comments (2)
On cmd: systemctl start k3s-agent, it returns: Failed to start k3s-agent.service: Unit not found
Do we delete the config.toml or add config.toml.tmpl file, while keeping the config.toml file as well?