Kubernetes 1.19 on Fedora 33 with kubeadm and a GPU

Enable kubernetes repos

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

Disable SELinux becasue ... lazy

sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

Enable Fedora cri-o module

dnf -y module enable cri-o:1.20
dnf install -y cri-o

Install kube stuff

dnf  install -y --disableexcludes=kubernetes kubelet kubeadm kubectl

Enable cri-o and kubelet on boot

systemctl enable cri-o && sudo systemctl enable kubelet

Set kubelet extra args

echo "KUBELET_EXTRA_ARGS=--cgroup-driver=systemd" | sudo tee /etc/sysconfig/kubelet

Enable required modules on boot

tee /etc/modules-load.d/crio-net.conf <<EOF
overlay
br_netfilter
EOF

Set sysctl options

tee /etc/sysctl.d/99-kubernetes-cri.conf <<EOF
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

Set cgroup kernel arg

  • Edit /etc/default/grub and add the following parameter to GRUB_CMDLINE_LINUX
systemd.unified_cgroup_hierarchy=0

Update grub changes

  • If using BIOS
grub2-mkconfig -o /boot/grub2/grub.cfg
  • If using UEFI
grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg

If you have swap enabled, disable it.

touch /etc/systemd/zram-generator.conf

Disable firewall because ... lazy

systemctl disable firewalld.service

Reboot

init 6

Run kubeadm

kubeadm init --pod-network-cidr=10.244.0.0/16 --cri-socket=/var/run/crio/crio.sock

Copy the kubeconfig to your home directory

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Remove the taint from the master ( only ) node

kubectl taint nodes --all node-role.kubernetes.io/master-

Deploy Calico

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

Add Helm

curl https://get.helm.sh/helm-v3.3.4-linux-amd64.tar.gz | tar zxvf -
cp linux-amd64/helm /usr/local/bin/helm

Add a helm repo too

helm repo add stable https://kubernetes-charts.storage.googleapis.com/

nfs-client-provisioner

helm install --set nfs.server=NFS_SERVER --set nfs.path=NFS_SHARE_PATH nfs-client-provisioner stable/nfs-client-provisioner

Install Traefik

Previous post about Traefik

Enable a GPU

Host settings

I'm using libvirt/kvm with a GTX video card so I have to modify the VM to trick the Nvidia driver into thinking it's not running in a VM.

virsh edit VM_NAME

Add these hyperv and kvm values to the features section of the VM XML then shutdown && reboot the VM.

  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <vendor_id state='on' value='zewstech'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>

Additionally IOMMU has to be enabled in the host bios and host operating system. There are plenty of articles on the internet for that so I'm not going to cover it here.

VM Settings

Enable rpmfusion repos

dnf install https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm

Enable nvidia container runtime repo

curl -so /etc/yum.repos.d/nvidia-container-runtime.repo https://nvidia.github.io/nvidia-container-runtime/centos8/nvidia-container-runtime.repo

Install

dnf install xorg-x11-drv-nvidia xorg-x11-drv-nvidia-devel nvidia-container-runtime

Reboot

init 6

Add and update the nvidia helm repo

helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update

Install the nvidia device plugin

helm install --version=0.7.0 nvidia-device-plugin nvdp/nvidia-device-plugin

Credit:

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

https://jebpages.com/2019/02/25/installing-kubeadm-on-fedora-coreos/