Production Kubernetes Networking and GitOps: Cilium, Flux CD, and Democratic CSI

Overview

Once a Kubernetes cluster is running, the next layer of decisions defines how it actually behaves in production: how traffic flows, how workloads get deployed, and where persistent data lives. This post covers three of those decisions on a 5-node bare metal cluster:

• Cilium: eBPF-based CNI replacing Flannel, with L2 LoadBalancer replacing MetalLB
• Flux CD v2: GitOps from a GitHub repo, with SOPS+age encrypted secrets
• Democratic CSI: dynamic iSCSI persistent volume provisioning against TrueNAS

The cluster itself (kubeadm bootstrap, HA control plane, physical worker nodes) is covered in the previous post. This picks up from a running cluster.

Cilium: eBPF Networking + L2 LoadBalancer

The cluster started on Flannel: straightforward to get running, but limited. Cilium brought eBPF-based networking, native network policy enforcement, and built-in L2 LoadBalancer support that replaced MetalLB entirely.

Migration from Flannel

The migration runs node-by-node. Taking multiple nodes simultaneously risks a split-brain scenario: do it serially.

1
2

# Ansible playbook: serial: 1 enforced in the play
ansible-playbook ansible/playbooks/cni-cleanup.yaml -i ansible/inventory.yaml

For each node the playbook:

1. Drains the node
2. Removes Flannel CNI config from /etc/cni/net.d/
3. Deletes the flannel.1 VXLAN interface and cni0 bridge
4. Uncordons: Cilium picks up the node

Flannel and Cilium cannot coexist. The bridge and VXLAN interface must be gone before Cilium takes over or you get routing loops.

Cilium HelmRelease (via Flux)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: cilium
  namespace: kube-system
spec:
  chart:
    spec:
      chart: cilium
      version: "1.16.*"
      sourceRef:
        kind: HelmRepository
        name: cilium
  values:
    kubeProxyReplacement: false
    ipam:
      mode: kubernetes
    tunnel: vxlan
    l2announcements:
      enabled: true
    externalIPs:
      enabled: true

kubeProxyReplacement: false keeps kube-proxy running: a safe migration path. Full kube-proxy replacement is possible but unnecessary for this setup.

L2 LoadBalancer Pool (Replacing MetalLB)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: homelab-pool
spec:
  cidrs:
    - cidr: 192.168.1.200/28   # .200–.215
---
apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata:
  name: homelab-l2
spec:
  interfaces:
    - ^enp.*
  externalIPs: true
  loadBalancerIPs: true

Cilium announces LoadBalancer IPs via ARP on all enp* interfaces. Services get a stable LAN-routable IP without a dedicated load balancer appliance. Current assignments:

Post-Migration: Restart RWO-Backed Deployments

After a CNI migration, pods don’t automatically get new network interfaces assigned by Cilium. Anything backed by a ReadWriteOnce PVC needs a careful restart: the old pod must fully terminate before the new one starts, or the volume attach hangs:

1
2

# Restart all deployments: handles RWO attach ordering
kubectl rollout restart deployment -A

For deployments with RWO PVCs, also ensure strategy: Recreate is set. Rolling updates deadlock: the new pod can’t attach the volume while the old one holds it.

Flux CD: GitOps from GitHub

All cluster state lives in a GitHub repo (homelab-gitops). Flux watches the repo and reconciles the cluster to match. Nothing is applied with kubectl apply by hand.

Repository Layout

homelab-gitops/
├── clusters/prod/flux-system/     # Flux bootstrap manifests
│   ├── gotk-components.yaml
│   ├── gotk-sync.yaml
│   ├── infrastructure.yaml        # Kustomization ref
│   ├── monitoring.yaml            # Kustomization ref
│   └── apps.yaml                  # Kustomization ref
├── infrastructure/prod/           # CNI, ingress, DNS, storage, security
│   ├── cilium/
│   ├── cert-manager/
│   ├── nginx-ingress/
│   ├── technitium/
│   ├── pihole/
│   ├── velero/
│   └── falco/
├── monitoring/prod/               # Prometheus stack, Loki, ntfy
└── apps/prod/                     # AI platform, applications
    ├── ai-platform/               # Ollama, LiteLLM, Open-WebUI, Qdrant
    ├── nextcloud/                 # Nextcloud + PostgreSQL + Redis
    ├── n8n/                       # n8n workflow automation
    ├── vault-ingest/              # RAG ingestion pipeline
    └── vaultwarden/

Three top-level Flux Kustomizations (infrastructure, monitoring, apps) with explicit dependency ordering: apps waits on infrastructure, monitoring waits on infrastructure. This prevents race conditions on first bootstrap.

SOPS + age Secret Encryption

Secrets are encrypted in git with SOPS using an age key. Flux decrypts at reconcile time using an age private key stored in a cluster secret.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Generate age key
age-keygen -o age.agekey

# Store private key in cluster (one-time setup)
kubectl create secret generic sops-age \
  --namespace=flux-system \
  --from-file=age.agekey

# Encrypt a secret
sops --encrypt \
  --age <age-public-key> \
  secret.yaml > secret.sops.yaml

.sops.yaml: only encrypt data/stringData:

1
2
3

creation_rules:
  - encrypted_regex: "^(data|stringData)$"
    age: <age-public-key>

Targeting only data/stringData fields is important. Encrypting the whole file makes git diffs unreadable and breaks Flux reconciliation in subtle ways: the rest of the manifest (metadata, kind, namespace) stays in plaintext and is diffable.

Flux Kustomization decryption config:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-age
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/prod

Forcing a Reconcile

Flux polls the GitHub repo on its configured interval. To trigger an immediate reconcile without waiting:

1
2

flux reconcile source git flux-system
flux reconcile kustomization apps --with-source

Useful after pushing a change that needs to apply immediately.

Democratic CSI: TrueNAS iSCSI Storage

Persistent storage comes from a TrueNAS server at 192.168.1.108 via iSCSI, provisioned dynamically by Democratic CSI.

How It Works

Democratic CSI runs as a DaemonSet on all nodes plus a controller pod. When a PVC is created:

1. The controller calls the TrueNAS API to create a new ZFS dataset + iSCSI extent
2. The node daemon connects the iSCSI target and formats the volume
3. The PVC binds and the pod starts

No manual TrueNAS config per volume: it’s fully automated.

StorageClass

1
2
3
4
5
6
7
8

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: truenas-iscsi
provisioner: org.democratic-csi.iscsi
reclaimPolicy: Retain
volumeBindingMode: Immediate
allowVolumeExpansion: true

reclaimPolicy: Retain means data survives PVC deletion: requires manual cleanup on TrueNAS but prevents accidental data loss. allowVolumeExpansion: true lets you resize PVCs without pod restarts.

RWO Constraint

iSCSI volumes are ReadWriteOnce: one node at a time. This has two practical implications:

Deployments must use strategy: Recreate:

1
2
3

spec:
  strategy:
    type: Recreate

Rolling updates deadlock: the new pod can’t attach the volume while the old one holds the iSCSI session. Recreate terminates the old pod first.

During CNI migrations or node drains, pods with iSCSI PVCs may take longer to reschedule. The iSCSI session must fully close before another node can attach. If a pod gets stuck terminating, the volume appears attached even after the pod is gone:

1
2
3

# Force detach if stuck
kubectl delete pod <pod> --grace-period=0 --force
# Then on TrueNAS: disconnect the stale iSCSI initiator session

Talos-Specific Note

If you’re running Talos Linux rather than Ubuntu, Democratic CSI requires the iscsi-tools system extension and a specific machine config patch to enable the iSCSI kernel modules. On Ubuntu with kubeadm, the open-iscsi package handles this:

1
2
3

# On each Ubuntu node
apt install open-iscsi
systemctl enable --now iscsid

Lessons Learned

Don’t migrate CNI on multiple nodes simultaneously. Serial node-by-node migration with drain/uncordon is slow but safe. Parallel migration caused routing issues between nodes mid-migration on the first attempt.

encrypted_regex in SOPS saves headaches. Encrypting only the secret values (not the whole file) makes diffs reviewable, keeps namespace/metadata visible, and avoids Flux failing to parse the resource kind.

reclaimPolicy: Retain by default for stateful data. The extra cleanup step on TrueNAS is worth it. Delete removes the ZFS dataset immediately when the PVC goes away: there’s no recovery.

Flux’s GitLab PAT is a silent failure. When it expires, reconciliation just stops. No alert fires by default. Add a Prometheus alert on got_reconcile_error or just set a calendar reminder.

L2 announcements need a stable interface pattern. The ^enp.* regex covers all the physical interfaces. If a node has a different NIC naming scheme, it won’t announce on that interface: check with cilium l2announce list after any node changes.

References

• Cilium Docs: L2 Announcements
• Flux CD: SOPS Decryption
• Democratic CSI: TrueNAS Configuration
• SOPS

Running a similar stack? We take on short-term infrastructure engagements and platform work.

Connect on LinkedIn →