Private AI Platform: LiteLLM, Open-WebUI, Qdrant, and RAG on Kubernetes

Overview

This post covers the AI platform built on top of the Kubernetes cluster. The cluster foundation (kubeadm, HA control plane, GPU setup) is in Post 1. The networking and GitOps layer (Cilium, Flux, Democratic CSI) is in Post 2. This picks up from a running cluster with persistent storage and GitOps in place.

The goal was a fully self-hosted AI stack with no external API dependencies for inference:

• LiteLLM: OpenAI-compatible API gateway routing to local models
• Ollama: GPU inference on a GTX 1070
• Open-WebUI: Chat interface for team use
• Qdrant: Vector database for embeddings and RAG
• RAG pipeline: Knowledge base ingestion and semantic search
• sre-cli + MCP server: Tooling that connects the AI stack to cluster observability

Architecture

PLATFORM (in-cluster, managed by Flux)
├─ Ollama       → ollama.ai-platform.svc.cluster.local:11434 (wk4, GPU)
├─ LiteLLM      → litellm.home.<domain>
├─ Open-WebUI   → open-webui.home.<domain>
└─ Qdrant       → qdrant.home.<domain>

TOOLING
└─ sre-cli (Python) → MCP server → optional external AI assistant

Ollama runs as a Kubernetes Deployment on wk4 with runtimeClassName: nvidia. The NVIDIA Container Toolkit injects GPU access at runtime, no host-level Ollama install required. LiteLLM routes to the in-cluster service URL.

Ollama: GPU Inference on Kubernetes

Ollama runs as a Kubernetes Deployment on wk4 using the NVIDIA device plugin and runtimeClassName: nvidia. The NVIDIA Container Toolkit handles GPU library injection at runtime, no CUDA paths to manage in the image.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: ai-platform
spec:
  strategy:
    type: Recreate
  template:
    spec:
      runtimeClassName: nvidia
      nodeSelector:
        gpu: "true"
      containers:
        - name: ollama
          image: ollama/ollama@sha256:<pinned>
          resources:
            requests:
              nvidia.com/gpu: 1
            limits:
              nvidia.com/gpu: 1
          volumeMounts:
            - name: models
              mountPath: /root/.ollama

A 50Gi TrueNAS iSCSI PVC persists the model weights across pod restarts.

Models in use:

The GTX 1070 has 8GB VRAM. Quantized 7B models fit comfortably: the practical sweet spot on this hardware. 12B+ models offload layers to RAM and reduce throughput significantly.

Ollama exposes an OpenAI-compatible API at ollama.ai-platform.svc.cluster.local:11434. LiteLLM routes to this in-cluster endpoint.

LiteLLM: Unified API Gateway

LiteLLM presents a single OpenAI-compatible API endpoint to all consumers. Routing rules, model aliases, and spend tracking all live in one place. Swapping an underlying model doesn’t require changes to any client.

Helm Deployment (via Flux)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: litellm
  namespace: ai-platform
spec:
  chart:
    spec:
      chart: litellm
      version: "0.1.*"
      sourceRef:
        kind: HelmRepository
        name: litellm
  values:
    replicaCount: 1
    env:
      - name: LITELLM_MASTER_KEY
        valueFrom:
          secretKeyRef:
            name: litellm-secrets
            key: master-key

The master key is a SOPS-encrypted secret in the GitHub repo. Flux decrypts at reconcile time: same pattern as all other secrets in this cluster.

Model Configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

model_list:
  - model_name: mistral
    litellm_params:
      model: ollama/mistral
      api_base: http://ollama.ai-platform.svc.cluster.local:11434

  - model_name: llama3.2:3b
    litellm_params:
      model: ollama/llama3.2:3b
      api_base: http://ollama.ai-platform.svc.cluster.local:11434

  - model_name: qwen2.5-coder:7b
    litellm_params:
      model: ollama/qwen2.5-coder:7b
      api_base: http://ollama.ai-platform.svc.cluster.local:11434

  - model_name: nomic-embed-text
    litellm_params:
      model: ollama/nomic-embed-text
      api_base: http://ollama.ai-platform.svc.cluster.local:11434

Any client that speaks OpenAI’s API works against LiteLLM unchanged. The gateway abstracts the underlying model routing entirely.

Open-WebUI: Chat Interface

Open-WebUI provides a browser-based chat interface connected to LiteLLM. It handles conversation history, model switching, and basic document upload.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

apiVersion: apps/v1
kind: Deployment
metadata:
  name: open-webui
  namespace: ai-platform
spec:
  strategy:
    type: Recreate    # iSCSI RWO: rolling update deadlocks
  template:
    spec:
      containers:
        - name: open-webui
          image: ghcr.io/open-webui/open-webui:main
          env:
            - name: OPENAI_API_BASE_URL
              value: http://litellm.ai-platform.svc.cluster.local:4000/v1
            - name: OPENAI_API_KEY
              valueFrom:
                secretKeyRef:
                  name: open-webui-secrets
                  key: litellm-api-key
          volumeMounts:
            - name: data
              mountPath: /app/backend/data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: open-webui-data

strategy: Recreate is required: Open-WebUI’s PVC is backed by an iSCSI RWO volume. A rolling update blocks indefinitely waiting for the old pod to release the volume before the new one can attach. This applies to every stateful workload in this stack.

Exposed at open-webui.home.<domain> via the nginx Ingress controller on the cluster’s Cilium L2 LoadBalancer IP.

Qdrant: Vector Database

Qdrant stores embeddings for the RAG pipeline. It runs in-cluster with a TrueNAS iSCSI persistent volume.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

apiVersion: apps/v1
kind: Deployment
metadata:
  name: qdrant
  namespace: ai-platform
spec:
  strategy:
    type: Recreate
  template:
    spec:
      containers:
        - name: qdrant
          image: qdrant/qdrant:latest
          ports:
            - containerPort: 6333   # REST
            - containerPort: 6334   # gRPC
          volumeMounts:
            - name: storage
              mountPath: /qdrant/storage
      volumes:
        - name: storage
          persistentVolumeClaim:
            claimName: qdrant-storage

Creating the collection for the knowledge base:

1
2
3
4
5
6
7
8

curl -X PUT http://qdrant.home.<domain>/collections/vault \
  -H 'Content-Type: application/json' \
  -d '{
    "vectors": {
      "size": 768,
      "distance": "Cosine"
    }
  }'

768 dimensions matches the output of nomic-embed-text. Cosine distance is the right metric for semantic similarity over text embeddings.

RAG Pipeline: Knowledge Base Ingestion

The knowledge base is an Obsidian vault (98 Markdown files covering runbooks, incident notes, infrastructure decisions, and project plans).

Chunking and Embedding

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

import httpx
import ollama

QDRANT_URL = "http://qdrant.home.<domain>"
OLLAMA_URL = "http://<xps-host>:11434"

def chunk_markdown(text: str, chunk_size: int = 500) -> list[str]:
    paragraphs = text.split("\n\n")
    chunks, current = [], ""
    for para in paragraphs:
        if len(current) + len(para) > chunk_size and current:
            chunks.append(current.strip())
            current = para
        else:
            current += "\n\n" + para
    if current.strip():
        chunks.append(current.strip())
    return chunks

def embed(text: str) -> list[float]:
    client = ollama.Client(host=OLLAMA_URL)
    resp = client.embed(model="nomic-embed-text", input=text)
    return resp["embeddings"][0]

Ingestion

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

import uuid

def ingest_file(filepath: str, client: httpx.Client):
    with open(filepath) as f:
        content = f.read()

    chunks = chunk_markdown(content)
    points = []
    for chunk in chunks:
        vector = embed(chunk)
        points.append({
            "id": str(uuid.uuid4()),
            "vector": vector,
            "payload": {
                "text": chunk,
                "source": filepath,
            }
        })

    client.put(
        f"{QDRANT_URL}/collections/vault/points",
        json={"points": points}
    )

Result: 247 chunks from 103 files ingested into the vault collection.

The chunking strategy is paragraph-based with a 500-character cap. This keeps chunks semantically coherent: a paragraph on one topic stays together rather than being split mid-sentence.

Semantic Search

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

def search_vault(query: str, top_k: int = 5) -> list[dict]:
    query_vector = embed(query)
    with httpx.Client() as client:
        resp = client.post(
            f"{QDRANT_URL}/collections/vault/points/search",
            json={
                "vector": query_vector,
                "limit": top_k,
                "with_payload": True,
            }
        )
    results = resp.json()["result"]
    return [
        {"text": r["payload"]["text"], "score": r["score"], "source": r["payload"]["source"]}
        for r in results
    ]

Direct httpx POST to the Qdrant REST API rather than using the qdrant-client Python library. The library introduced breaking API changes between minor versions that silently changed the response shape. The REST API is stable and the dependency is eliminated entirely.

sre-cli: Cluster-Aware AI Tooling

sre-cli is a Python command-line tool that combines cluster observability with AI-augmented workflows.

sre alert          - active Prometheus alerts
sre cluster        - node status, resource utilisation
sre runbook <q>    - semantic search across runbooks
sre incident <q>   - recent incident notes and post-mortems

runbook and incident both call search_vault with the query and return the top-scored chunks formatted as terminal output, no LLM inference required for the search itself. Relevant context surfaces in under a second.

MCP Server

The same tools are exposed as an MCP server. This is optional: the core platform (local inference, RAG, semantic search) is fully self-hosted and air-gapped. The MCP server is a bridge for any external AI assistant that supports the protocol.

1
2

claude mcp add homelab-sre \
  -- python /path/to/sre_cli/mcp_server.py

Four tools exposed:

With the MCP server active, an external AI assistant can query cluster state and search the knowledge base mid-conversation, no copy-pasting from terminal. The cluster data being queried stays on-prem; only the query and retrieved context leave the network if an external assistant is used.

How It Comes Together

A realistic workflow:

1. Alert fires: Alertmanager routes to ntfy, phone notification arrives
2. sre alert: Check what’s firing from terminal
3. sre runbook "etcd quorum": Pull relevant runbook chunks without opening a browser
4. sre-cli + MCP (optional): If using an external AI assistant via the MCP server, it can call get_alerts and search_vault to correlate the alert with knowledge base context. The cluster data stays on-prem; the assistant just receives the query results.

The AI platform isn’t replacing SRE judgment: it’s reducing the time spent finding information that already exists in runbooks and incident notes.

Lessons Learned

runtimeClassName: nvidia makes GPU scheduling straightforward. The NVIDIA Container Toolkit injects GPU libraries at runtime, no manual CUDA path configuration in the image. The device plugin handles GPU resource accounting. Ollama inside the cluster means the inference endpoint is a proper k8s Service, not a host IP dependency.

strategy: Recreate for every stateful workload. iSCSI RWO volumes block rolling updates indefinitely. Every deployment backed by a PVC in this stack uses Recreate. Make it a default, not an afterthought.

REST over SDK for Qdrant. The qdrant-client library introduced a breaking change between minor versions that silently changed the response shape. The REST API hasn’t changed. For a search endpoint called in production tooling, that stability matters.

768 dimensions must match at collection creation. Creating the Qdrant collection with the wrong dimension silently accepts points but returns garbage similarity scores. Verify the embedding model’s output dimension before ingestion: fix requires dropping and recreating the collection.

Paragraph-based chunking preserves coherence. Fixed-size character splits break context at arbitrary points. Splitting on double newlines (paragraph boundaries) keeps each chunk semantically self-contained, which improves retrieval relevance meaningfully.

SOPS for AI platform secrets. LiteLLM’s master key and Open-WebUI’s API key live encrypted in git. The same encrypted_regex: "^(data|stringData)$" pattern used across the cluster keeps diffs readable while protecting values.

What’s Running

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

$ kubectl get pods -n ai-platform
NAME                          READY   STATUS
ollama-77477f456b-dvdhq       1/1     Running
litellm-7bdf9787bc-nb9qm      1/1     Running
open-webui-7c64c65977-w5hl9   1/1     Running
qdrant-5ffbf5757f-px85d       1/1     Running

$ kubectl exec -n ai-platform deployment/ollama -- ollama list
NAME                       SIZE
mistral:latest             4.4 GB
llama3.2:3b                2.0 GB
qwen2.5-coder:7b           4.7 GB
nomic-embed-text:latest    274 MB

All inference stays on-premises. No data leaves the network.

Building a private AI platform? We take on short-term infrastructure engagements and platform work.

Connect on LinkedIn →