Private LLM Inference on Kubernetes: What It Costs, What It Takes, and What You Get

The Question Worth Answering Honestly

Most articles about self-hosted AI focus on the setup. This one focuses on the decision. If you’re evaluating whether to run LLM inference on your own infrastructure, rather than calling OpenAI, Anthropic, or another API, you need honest answers to three questions:

• What does it actually cost?
• What does it actually take to run?
• What do you actually get in return?

The answers depend heavily on your use case. For some organisations, self-hosted inference is a clear win. For others, it’s unnecessary complexity. This post gives you the information to tell the difference.

What It Costs

Hardware

The minimum viable GPU for running useful LLMs (quantized 7B models) is an NVIDIA card with 8GB VRAM. A GTX 1070, available second-hand for €150–€300, is the entry point. At 8GB, you can run:

• mistral:7b (4-bit quantized, ~4.4GB): general queries, coding assistance
• llama3.2:3b (~2.0GB): fast lightweight tasks
• qwen2.5-coder:7b (4-bit, ~4.7GB): code generation, YAML, debugging
• nomic-embed-text (~274MB): embeddings for RAG pipelines

A single model loaded at a time. Swapping models takes 5–15 seconds on NVMe storage.

For 13B models you need 12–16GB VRAM. A used RTX 3090 (24GB) costs €600–€900 and runs 7B and 13B models comfortably, with headroom for 34B at reduced quality. An RTX 4090 (24GB) runs 34B models well and costs €1,500–€2,000.

The server to host the GPU adds another €200–€800 for a mid-range desktop.

Power

A GPU inference server draws 150–250W under load, 40–80W at idle. At European electricity prices (~€0.30/kWh):

• Running 8 hours/day at 200W average: ~€14/month
• Running 24/7 at 100W average: ~€22/month

Power is not a significant factor below mid-scale usage.

Operations Time

This is the underestimated cost. A Kubernetes-based AI platform requires:

• Initial setup: 3–8 days depending on existing cluster maturity
• Ongoing: 2–4 hours/month for model updates, monitoring, occasional restarts
• Incident response: 1–4 hours when something breaks (rare, but it happens)

If you have no Kubernetes experience, add a significant learning curve or budget for an external engineer to build it.

Cloud API Comparison

For context, OpenAI API costs at the time of writing:

• gpt-4o-mini: ~$0.60/1M input tokens, ~$2.40/1M output tokens
• gpt-4o: ~$5.00/1M input tokens, ~$15.00/1M output tokens

At 1M tokens/month (moderate usage), gpt-4o-mini costs roughly €50–€100/month. A self-hosted GTX 1070 setup costs €15–€30/month in hardware amortization plus ~€15/month in power: roughly equivalent, but with much lower quality ceiling.

The economics shift decisively once you hit 5M+ tokens/month, need 24/7 availability at scale, or require data to stay on-premises.

What It Takes to Run

GPU Access in Kubernetes

The NVIDIA Container Toolkit enables GPU access inside containers. Once installed on the host, the NVIDIA device plugin DaemonSet exposes the GPU as a schedulable resource:

1
2
3
4
5

resources:
  requests:
    nvidia.com/gpu: 1
  limits:
    nvidia.com/gpu: 1

With runtimeClassName: nvidia, the container gets direct GPU access. No CUDA paths to manage in the image.

The constraint: a GPU is a non-shareable resource. Only one pod holds it at a time. If Ollama owns the GPU, nothing else can use it simultaneously. For dedicated inference infrastructure this is fine; for mixed workloads it requires planning.

Model Storage

Models are large. A 7B quantized model is 4–5GB. Storing multiple models means:

• Persistent volume sized appropriately (50GB+ for a useful model library)
• NVMe preferred: model load times drop from 60s to 8–15s
• The volume must survive pod restarts, iSCSI or local persistent volumes work, network block storage is better than NFS for this workload

Inference Throughput Reality

On a GTX 1070 with Ollama:

For a chat interface with one or two concurrent users, this is fine. For API consumers running parallel requests, it becomes a bottleneck. Ollama queues requests: multiple parallel callers wait for the previous generation to complete.

If you need higher throughput or true parallelism, you need multiple GPUs or a more capable single GPU (A10G, A100) with a batching inference server (vLLM, TGI).

Operational Stability

In 12+ months running Ollama on Kubernetes with a GTX 1070:

• No GPU driver issues after initial setup
• No Ollama OOMs with standard 7B models (VRAM well-characterised)
• Occasional pod restart required after cluster CNI migrations
• Model downloads require internet access; air-gapped environments need a registry mirror

The stack is stable. The failure modes are predictable and recoverable.

What You Get

Data Sovereignty

The most important reason to self-host is that data never leaves your network. Prompts, responses, documents fed into the RAG pipeline, none of it touches an external API. For organisations handling:

• Personally identifiable information under GDPR
• Financial data under MiFID II
• Healthcare data under HIPAA
• Legally privileged information
• Confidential commercial strategy

…this is not a nice-to-have. It is the only viable path. Most cloud AI providers, regardless of their data processing agreements, send your data through their infrastructure. A self-hosted stack eliminates that exposure entirely.

Model Control

You choose which models run, and you keep them. No deprecation notices, no API version changes, no model being replaced without your input. The model you deployed 18 months ago is still there.

You can also fine-tune or adapt models to your domain vocabulary, though that requires additional compute investment beyond inference.

No Rate Limits

Cloud API rate limits become a real operational constraint at scale. Self-hosted inference has no external rate limits, throughput is bounded only by hardware. For burst workloads or batch processing (document ingestion, nightly summarisation runs), this matters significantly.

Compliance Auditability

A self-hosted platform gives you full auditability: what models ran, what requests were made, what data flowed through the system. With proper logging, you can answer any compliance question about your AI platform’s behaviour. Cloud APIs make this significantly harder.

Predictable Costs

Once hardware is purchased, costs are effectively fixed (power + ops time). No per-token billing. No surprise costs from a prompt that generated an unexpectedly long response. No API price changes.

When Self-Hosted Makes Sense

Yes, self-host if:

• Your data cannot leave the premises (regulatory or legal requirement)
• You’re processing high volumes (5M+ tokens/month)
• You need model stability over time
• You have or can acquire Kubernetes expertise
• You’re building a product where AI is a core component, not a peripheral feature

No, don’t self-host if:

• Volume is low and growing slowly
• You have no infrastructure team or budget to build one
• Frontier model capability (GPT-4 class) is a hard requirement
• Time to deployment is the primary constraint

The middle path: a hybrid. Use OpenAI or Anthropic for frontier-model tasks (complex reasoning, long-context synthesis). Run local models for high-volume, lower-complexity tasks (classification, summarisation, embeddings). LiteLLM as an OpenAI-compatible gateway handles both, letting you route by model name without changing client code.

The Stack That Makes This Practical

The architecture that makes this maintainable:

Ollama (GPU inference)
  → LiteLLM (OpenAI-compatible gateway, model routing, spend tracking)
    → Open-WebUI (team chat interface)
    → Any OpenAI SDK client (Python, Node, curl)
    → RAG pipeline (Qdrant vector DB + embedding model)

Everything runs on Kubernetes, managed by Flux GitOps. Secrets encrypted in git with SOPS. Models persisted on a TrueNAS iSCSI volume. The full setup is covered in the private AI platform post.

For regulated industries, the architecture adds:

• Network policies (Cilium) isolating the AI namespace
• No egress from inference pods
• Audit logging to a tamper-evident store
• RBAC scoped to minimum required access

The Honest Summary

Self-hosted LLM inference on Kubernetes is:

• Viable on commodity hardware at 8GB+ VRAM
• Economical at medium-to-high token volumes
• Operationally stable with a well-configured stack
• The only option when data residency is a compliance requirement
• Not a weekend project if you’re starting from zero infrastructure

The gap between “I’ve read the tutorials” and “this is running reliably in a production environment” is real. The architecture decisions, storage class choice, GPU scheduling, secret management, CNI network policy, each has failure modes that only surface under real operational conditions.

Running a similar evaluation? We take on short-term infrastructure engagements and platform work.

Connect on LinkedIn →