Overview

I built this homelab on Ubuntu LTS (22.04 on the Lenovo nodes, 24.04 on the XPS worker). A proven Linux distribution with mature GPU support, extensive ecosystem tooling, and transparency that matters during troubleshooting.

The goal was a high availability Kubernetes cluster capable of running GPU-accelerated AI workloads (LLM inference, vector databases, real-time embeddings) on a budget, without sacrificing production-grade reliability.

This post covers the foundation: hardware, OS choice, kubeadm HA bootstrap, and GPU setup. Networking, GitOps, and storage are covered in the next post. The AI platform that runs on top is covered in the private AI platform post.

Hardware

XPS Ubuntu Server (GPU inference worker):

  • CPU: Intel Core i7 (6c / 12t)
  • Memory: 40Gi DDR4
  • GPU: NVIDIA GeForce GTX 1070 (CUDA 12.4, driver 550)
  • Storage: 1TB NVMe + 256GB NVMe

Physical Nodes (Lenovo ThinkCentres):

  • cp1 (192.168.1.102): 12 CPU, 31Gi RAM
  • cp2 (192.168.1.101): 12 CPU, 31Gi RAM
  • cp3 (192.168.1.103): 12 CPU, 31Gi RAM
  • wk1 (192.168.1.106): 8 CPU, 12Gi RAM

wk4 (XPS Desktop, joined as physical worker, Ubuntu 24.04 LTS):

  • wk4 (192.168.1.110): 12 CPU, 40Gi RAM, GTX 1070

The GTX 1070 handles LLM inference (quantized 7B models at good throughput). Ollama runs as a Kubernetes Deployment on wk4 using runtimeClassName: nvidia, with the NVIDIA device plugin scheduling GPU access.

Why Ubuntu 22.04 LTS

GPU and NVIDIA Ecosystem

GPU support on Ubuntu is well-established. The driver installation path is standard and well-documented:

1
2
3

apt install nvidia-driver-550
apt install nvidia-container-toolkit
nvidia-smi

For LLM inference on a GTX 1070, CUDA + cuDNN integration is straightforward. No compatibility surprises, no driver version mismatches.

Debugging

When inference slows, or a vector DB stops responding, or GPU utilisation drops, you need standard tools:

  • nvidia-smi / nvidia-smi dmon: GPU state and metrics
  • tcpdump: network traffic
  • journalctl: systemd logs
  • /proc: process inspection
  • strace: system calls

Ubuntu provides all of this natively. Transparent, auditable, accessible.

Ecosystem and Operational Familiarity

ML/AI packages (PyTorch, llama.cpp, Transformers) are built and tested against Ubuntu LTS. GPU monitoring tools (gpustat, gpu-burn) work out of the box. Community support is everywhere. GPU driver issues have thousands of answers online.

Ubuntu knowledge transfers. That matters when troubleshooting infrastructure at 2am.

Cluster Architecture

CONTROL PLANE (3 nodes, HA etcd quorum)
├─ cp1 (192.168.1.102) Physical Lenovo, 12 CPU, 31Gi
├─ cp2 (192.168.1.101) Physical Lenovo, 12 CPU, 31Gi
└─ cp3 (192.168.1.103) Physical Lenovo, 12 CPU, 31Gi

WORKERS (2 nodes)
├─ wk1 (192.168.1.106) Physical Lenovo, 8 CPU, 12Gi
└─ wk4 (192.168.1.110) Physical XPS, 12 CPU, 40Gi, GTX 1070

Three control planes give etcd quorum: the cluster survives one CP failure without intervention. wk4 (the XPS desktop) joined the cluster as a physical worker, providing the GPU for AI inference workloads.

Bootstrap

Node Preparation (Ansible)

1
2
3

# Runs on all nodes: installs kubeadm/kubelet/kubectl,
# configures container runtime, sets kernel params
ansible-playbook ansible/playbooks/node-bootstrap.yml -i ansible/inventory.yaml

Key steps per node:

  • Install containerd as the container runtime
  • Set net.bridge.bridge-nf-call-iptables=1, net.ipv4.ip_forward=1
  • Disable swap
  • Install kubeadm, kubelet, kubectl from the Kubernetes apt repo

For the XPS host, also:

1
2
3
4

apt install nvidia-driver-550 nvidia-container-toolkit
# Configure containerd to use nvidia runtime
nvidia-ctk runtime configure --runtime=containerd
systemctl restart containerd

HA Control Plane (kubeadm)

A Virtual IP at 192.168.1.100 provides the stable API server endpoint. keepalived handles VIP failover across the three CP nodes, haproxy load-balances API traffic.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Bootstrap the first control plane
kubeadm init \
  --control-plane-endpoint "192.168.1.100:6443" \
  --upload-certs \
  --pod-network-cidr=10.244.0.0/16

# Join the remaining two control planes
kubeadm join 192.168.1.100:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane \
  --certificate-key <cert-key>

# Join workers
kubeadm join 192.168.1.100:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

wk4: Joining the XPS Desktop as a Physical Worker

The XPS desktop was joined directly as a physical worker node with Ubuntu 24.04 installed fresh, then joined to the cluster with kubeadm join:

1
2
3

kubeadm join 192.168.1.100:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

After joining, the node was labelled for GPU scheduling:

1

kubectl label node wk4 gpu=true

This allows the NVIDIA device plugin DaemonSet and GPU workloads (Ollama) to target wk4 specifically via nodeSelector.

Cluster Health

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

$ kubectl get nodes -o wide
NAME   STATUS   ROLES           VERSION
cp1    Ready    control-plane   v1.31.3
cp2    Ready    control-plane   v1.31.3
cp3    Ready    control-plane   v1.31.3
wk1    Ready    worker          v1.31.3
wk4    Ready    worker          v1.31.3

$ kubectl exec -n ai-platform deployment/ollama -- nvidia-smi
NVIDIA GeForce GTX 1070
Driver Version: 550.x  |  CUDA Version: 12.4

Physical nodes have remained stable since initial bootstrap.

Lessons Learned

keepalived VIP timing matters. On first bootstrap, the VIP must be up before kubeadm init runs. If keepalived isn’t elected master yet, the API server binds to the wrong address and the join tokens reference an unreachable endpoint.

GPU inside the cluster via runtimeClassName: nvidia. Ollama runs as a Kubernetes Deployment on wk4 with runtimeClassName: nvidia and nvidia.com/gpu: 1 resource requests. The NVIDIA Container Toolkit handles library injection at runtime with no manual CUDA path wrangling in the container image. The device plugin DaemonSet targets nodes labelled gpu: "true".

What’s Next

The cluster foundation is running. The next posts cover what was built on top:

Building a similar cluster? We take on short-term infrastructure engagements and platform work.

Connect on LinkedIn →