Implementing Secrets & Zero-Trust in Kubernetes

Your database password is in a ConfigMap. Your API key is in a git repo. Your cloud credentials are in plaintext environment variables.

This is how most Kubernetes clusters look in the wild. And it’s how breaches happen.

Zero-trust means: don’t trust anything by default. Encrypt secrets at rest. Network-isolate workloads. Control access explicitly. Only then grant it.

This article covers a complete secrets and zero-trust strategy: encryption, network policies, access control, credential rotation, and scanning for leaks. We’ll use real examples from a production-grade setup.

Part 1: Secrets

The Problem: Unencrypted Secrets in etcd

By default, Kubernetes stores secrets in etcd as base64-encoded text. Base64 isn’t encryption: it’s just encoding. Anyone with etcd access reads your passwords:

1
2
3
4
5
6
7

# This is how a secret is stored by default
etcdctl get /registry/secrets/production/database-password
# Output: base64-encoded plaintext

# Base64 is trivial to decode
echo "dGhpcyBpcyBub3Qgc2VjcmV0" | base64 -d
# Output: this is not secret

If your etcd is compromised (stolen backup, network sniffing, insider access), your secrets are exposed.

The fix: Encrypt secrets at rest using SOPS + age.

SOPS: Secrets Operations

SOPS is a tool that encrypts individual secret values in YAML files. It keeps your infrastructure code in git but encrypts sensitive data using your encryption key.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Plaintext YAML (what you write)
apiVersion: v1
kind: Secret
metadata:
  name: database-creds
  namespace: production
type: Opaque
stringData:
  username: admin
  password: super-secret-password-12345

SOPS encrypts it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Encrypted YAML (what goes to git)
apiVersion: v1
kind: Secret
metadata:
  name: database-creds
  namespace: production
type: Opaque
stringData:
  username: ENC[AES256_GCM,data:7m2k+g==,iv:nUe/Zq9K...]
  password: ENC[AES256_GCM,data:a3F9xK2m...,iv:hJ8Qb2Lx...]
sops:
  kms: []
  age:
    - recipient: age1xxxxxxxxxxxxx...
      enc: -----BEGIN AGE ENCRYPTED FILE-----
           ...
           -----END AGE ENCRYPTED FILE-----

Only your age private key can decrypt it. Commit the encrypted YAML to git. Your git repo is now safe.

Setup: SOPS + age

Step 1: Generate an age key

1
2
3
4
5
6

# Generate once, keep private
age-keygen -o ~/.age/infrastructure.txt

# Extract the public key
cat ~/.age/infrastructure.txt | grep "public key:"
# age1xxxxxxxxxxxxx...

Step 2: Create .sops.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# .sops.yaml in your repo root
creation_rules:
  - path_regex: .*\.sops\.yaml$
    encrypted_regex: ^(data|stringData)$
    age:
      - recipient: age1xxxxxxxxxxxxx  # Your public key
        enc: |
          -----BEGIN AGE ENCRYPTED FILE-----
          ... (encrypted key material)
          -----END AGE ENCRYPTED FILE-----

The encrypted_regex: ^(data|stringData)$ tells SOPS to only encrypt the actual secret values, not metadata. This keeps your YAML readable and git-diffable.

Step 3: Create a secret and encrypt it

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# Write plaintext secret
cat > database-secret.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: database-creds
  namespace: production
type: Opaque
stringData:
  username: admin
  password: my-super-secret-password
EOF

# Encrypt it
sops -e -i database-secret.yaml

# Now the file contains encrypted values
cat database-secret.yaml
# stringData:
#   username: ENC[AES256_GCM,...]
#   password: ENC[AES256_GCM,...]

# Commit to git (it's encrypted)
git add database-secret.yaml
git commit -m "Add encrypted database credentials"

Step 4: Deploy with Flux CD + SOPS

Flux automatically decrypts SOPS secrets when applying them to the cluster.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# flux-system/sops-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: sops-age
  namespace: flux-system
type: Opaque
stringData:
  age.agekey: |
    AGE-SECRET-KEY-...  # Your private age key

Then configure Flux to use it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# clusters/prod/flux-system/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - sources.yaml
  - kustomization.yaml
secretGenerator:
  - name: sops-age
    namespace: flux-system
    files:
      - age.agekey=~/.age/infrastructure.txt

Flux decrypts secrets on apply. Encrypted values never touch your CI/CD pipeline unencrypted.

What to Encrypt

Encrypt anything that grants access:

• Database passwords
• API keys (cloud, third-party services)
• SSH keys
• TLS certificates (private keys only)
• OAuth tokens
• Backup credentials
• Registry credentials

Example from a production setup:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

apiVersion: v1
kind: Secret
metadata:
  name: kube-prometheus-stack-secrets
  namespace: monitoring
type: Opaque
stringData:
  grafana-admin-password: ENC[AES256_GCM,data:...]
---
apiVersion: v1
kind: Secret
metadata:
  name: democratic-csi-credentials
  namespace: storage
type: Opaque
stringData:
  truenas-api-key: ENC[AES256_GCM,data:...]
  truenas-username: ENC[AES256_GCM,data:...]
---
apiVersion: v1
kind: Secret
metadata:
  name: velero-backup-credentials
  namespace: velero
type: Opaque
stringData:
  cloud-credentials: ENC[AES256_GCM,data:...]

Don’t Encrypt

Don’t encrypt non-sensitive data:

• ConfigMaps (configuration, not secrets)
• Namespace labels and annotations
• RBAC resource definitions
• Deployment specs (unless they reference secrets, which is fine)

Encrypt only the stringData/data fields that contain credentials. Keep metadata readable for git diffs.

Part 2: Network

Secrets protect data at rest. Network policies protect data in flight and control who can talk to what.

Default-Deny + Explicit Allow

Zero-trust networking means: block all traffic by default. Allow only what you explicitly need.

Step 1: Apply default-deny to production namespaces

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# apps/prod/production/network-policies/default-deny.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress

This blocks all incoming traffic to all pods in the production namespace. No exceptions.

Step 2: Add explicit allows for known traffic

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

# Allow ingress-nginx to reach this namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-ingress-nginx
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx

---
# Allow pods within the namespace to talk to each other
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-intra-namespace
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector: {}

---
# Allow monitoring to scrape metrics
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-monitoring
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - port: 8080
          protocol: TCP

---
# Allow kube-system (DNS, kubelet healthchecks)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-kube-system
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system

Now traffic flow is explicit:

• Ingress traffic → Only from ingress-nginx, kube-system, or same namespace, or monitoring (for metrics)
• DNS → Allowed from kube-system (kubelet sends DNS queries)
• Egress → Unrestricted by default (you can add egress policies if needed)

Namespace Isolation Pattern

Apply this pattern to every sensitive namespace:

production/
├── network-policies/
│   ├── default-deny-ingress.yaml
│   ├── allow-from-ingress-nginx.yaml
│   ├── allow-from-kube-system.yaml
│   ├── allow-intra-namespace.yaml
│   ├── allow-from-monitoring.yaml
│   └── kustomization.yaml
└── kustomization.yaml (includes network-policies)

Sensitive namespaces that get this treatment:

• production: Your main apps
• databases: If you run databases in K8s
• payments: If you handle payments
• secrets: Vaultwarden, secret stores
• monitoring: Prometheus scrapes everything, but shouldn’t be scraped by random pods
• ai-platform: If running LLMs or proprietary models

Namespaces that don’t need network policies:

• kube-system: Core Kubernetes components (exempt from network policies)
• flux-system: GitOps controller (exempt from network policies)
• dev: Development/testing (can be more permissive)

Debugging Network Policies

Network policies are hard to debug. Tools help:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Test pod-to-pod connectivity
kubectl run -it --rm debug --image=nicolaka/netshoot --restart=Never -- bash
# Inside the pod:
curl -v http://api-service.production.svc.cluster.local:8080

# Check which network policies apply to a pod
kubectl get networkpolicies -n production
kubectl describe networkpolicy allow-from-ingress-nginx -n production

# Use Cilium's network policy viewer (if using Cilium CNI)
cilium policy list -n production

# Simulate a policy change without applying it
# Create the policy in a test namespace, verify behavior, then promote

Part 3: Access Control

Secrets are encrypted. Network is isolated. Now control who can access what.

RBAC: Role-Based Access Control

RBAC answers: what can this service account do?

The concept:

• ServiceAccount: Identity (who)
• Role/ClusterRole: Permissions (what)
• RoleBinding/ClusterRoleBinding: Link them (who can do what)

Example: Monitoring stack needs to read pod metadata

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

# Role: "read pod info"
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-reader
rules:
  - apiGroups: [""]
    resources: ["pods", "nodes", "endpoints"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets"]
    verbs: ["get", "list"]

---
# ServiceAccount: "prometheus" identity
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring

---
# RoleBinding: give prometheus the reader role
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-read-pods
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus-reader
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: monitoring

RBAC best practices:

1. Least privilege: Only grant what’s needed. Prometheus needs to read pods, not delete them.
2. Namespace-scoped where possible: Use Role + RoleBinding instead of ClusterRole + ClusterRoleBinding when scoped to one namespace.
3. Service account per workload: Don’t share service accounts across deployments.
4. No cluster-admin for apps: cluster-admin is for infrastructure operators, not applications.

Dangerous patterns to avoid:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# ❌ DON'T: Grant all verbs
- apiGroups: [""]
  resources: ["*"]
  verbs: ["*"]  # Too permissive!

# ❌ DON'T: Share service accounts
# Bad: 100 deployments using the same "default" service account

# ✅ DO: Specific roles
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list"]
  resourceNames: ["app-config"]  # Only this configmap

Pod Security Standards

Pod security controls what a container can do: mount host filesystems, run as root, change kernel parameters.

PodSecurityPolicy (deprecated) → PodSecurityStandard (now)

Apply labels to namespaces:

1
2
3
4
5
6
7
8
9

# Mark production namespace as "restricted"
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

This enforces:

• Non-root user
• No privileged containers
• No hostPath mounts
• No host networking
• Immutable root filesystem
• Read-only containers

Example deployment that fails restricted policy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bad-app
  namespace: production
spec:
  template:
    spec:
      containers:
        - name: app
          image: my-app:latest
          securityContext:
            runAsUser: 0  # ❌ Running as root - violates restricted policy
            privileged: true  # ❌ Privileged container - violates
          volumeMounts:
            - name: host-fs
              mountPath: /host
      volumes:
        - name: host-fs
          hostPath:
            path: /  # ❌ Host path mount - violates

Kubernetes rejects this in a restricted namespace.

Correct deployment:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

apiVersion: apps/v1
kind: Deployment
metadata:
  name: good-app
  namespace: production
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 2000
      containers:
        - name: app
          image: my-app:latest
          securityContext:
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir: {}

Part 4: Credential Rotation

Encrypted secrets are good. But if a secret is compromised, you need to rotate it quickly.

Rotation Strategy

Principle: secrets should have expiration dates.

For each type of credential, define a rotation schedule:

Manual Rotation Process

For credentials not automated (e.g., database passwords):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# 1. Generate new credential
# (e.g., in your database, cloud console, or service UI)

# 2. Add the new secret to Kubernetes
kubectl create secret generic database-creds-v2 \
  --from-literal=username=admin \
  --from-literal=password=new-password-xyz \
  -n production

# 3. Update deployments to use new secret
kubectl patch deployment my-app -p \
  '{"spec":{"template":{"spec":{"containers":[{"name":"app","env":[{"name":"DB_PASSWORD","valueFrom":{"secretKeyRef":{"name":"database-creds-v2","key":"password"}}}]}]}}}}'

# 4. Verify new deployments are running
kubectl rollout status deployment/my-app -n production

# 5. Delete old secret (once confident)
kubectl delete secret database-creds -n production

# 6. Update your SOPS file with new credentials
# Add to database-secret.yaml, encrypt, commit

Automated Rotation with External Secrets Operator

For cloud credentials and API keys, use External Secrets Operator (ESO) to sync from a secret manager:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

# Install ESO
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets-system --create-namespace

# Create SecretStore pointing to AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets

---
# Create ExternalSecret that syncs from AWS
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-password
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: database-creds
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: prod-db-password

This automatically syncs credentials from AWS Secrets Manager every hour. Rotate in AWS, and Kubernetes picks up the change.

Part 5: Scanning for Credential Leaks

Despite your best efforts, credentials slip into code. Git history, logs, old commits.

Tools to Find Leaks

truffleHog: Scans git history for credential patterns

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Scan entire repo history
trufflehog filesystem . --json > leaks.json

# Scan remote repo
trufflehog github --org my-org --token $GITHUB_TOKEN

# Output: finds API keys, private keys, passwords
# Example:
# {
#   "Match": "AKIA2XXXXXXXX",
#   "File": "terraform/variables.tf",
#   "Commit": "abc123def456",
#   "Type": "AWS Access Key"
# }

git-secrets: Prevents commits containing secrets

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Install
brew install git-secrets

# Add to repo
cd my-repo
git secrets --install
git secrets --register-aws

# Now git commit fails if credentials are detected
git commit -m "Add AWS key"
# ❌ Error: AWS credentials detected in config.tf

SOPS pre-commit hook: Ensures all secrets are encrypted before commit

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/mozilla/sops
    rev: v3.8.1
    hooks:
      - id: sops-checker
        entry: sops-audit
        language: python
        types: [yaml]
        # Fails commit if unencrypted secrets found

Audit Existing Code

1
2
3
4
5
6
7
8

# Find plaintext passwords in YAML
grep -r "password:" . --include="*.yaml" | grep -v "ENC\["

# Find hardcoded API keys
grep -r "api.key\|api_key\|apikey" . --include="*.py" --include="*.go" --include="*.js"

# Find environment variables with credentials
grep -r "export.*PASSWORD\|export.*KEY\|export.*TOKEN" . --include="*.sh"

Fix found credentials by:

1. Rotate the leaked credential immediately
2. Move to secrets (encrypt with SOPS)
3. Commit the fix with the credential removed
4. Scan again to verify removal

Putting It All Together: Zero-Trust Architecture

Here’s how all pieces work together:

┌─────────────────────────────────────────────────────────────────┐
│ Production Namespace                                            │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌─ Pod Security Standard: restricted ──────────────────────┐  │
│  │  - Non-root user                                         │  │
│  │  - No privileged containers                             │  │
│  │  - Read-only root filesystem                            │  │
│  └──────────────────────────────────────────────────────────┘  │
│                                                                 │
│  ┌─ Network Policies: default-deny + explicit allows ──────┐  │
│  │  ✓ Allow from ingress-nginx (external traffic)         │  │
│  │  ✓ Allow from monitoring (metrics scraping)            │  │
│  │  ✓ Allow intra-namespace (pod-to-pod)                  │  │
│  │  ✗ Block everything else                                │  │
│  └──────────────────────────────────────────────────────────┘  │
│                                                                 │
│  ┌─ RBAC: least privilege ───────────────────────────────────┐  │
│  │  app-sa can only:                                        │  │
│  │  - Read configmaps named "app-config"                   │  │
│  │  - Read secrets named "app-db-password"                 │  │
│  │  - Cannot list/delete/modify other resources            │  │
│  └──────────────────────────────────────────────────────────┘  │
│                                                                 │
│  ┌─ Encrypted Secrets ───────────────────────────────────────┐  │
│  │  Secret: app-db-password                                 │  │
│  │  - Stored encrypted in etcd (SOPS + age)                │  │
│  │  - Only decrypted when pod requests it                  │  │
│  │  - Rotated every 90 days                                 │  │
│  │  - Leaked credentials detected by truffleHog            │  │
│  └──────────────────────────────────────────────────────────┘  │
│                                                                 │
│  Pod: api-server                                               │
│  │                                                              │
│  ├─ Runs as user: 1000 (not root)                             │
│  ├─ SA: app-sa (RBAC enforced)                                │
│  ├─ Mount: /etc/app-secrets (from encrypted secret)           │
│  ├─ Filesystem: read-only (except /tmp)                       │
│  │                                                              │
│  └─ Can only:                                                  │
│     - Read database password from secret mount                │
│     - Connect to allowed pods (network policy)                │
│     - Cannot access cluster secrets                           │
│     - Cannot escalate privileges                              │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Traffic in:

Internet
  ↓
Firewall (pfSense, cloud SG)
  ↓
LoadBalancer IP
  ↓
Ingress Controller (ingress-nginx)
  ↓
NetworkPolicy (allow-from-ingress-nginx)
  ↓
Service Endpoint
  ↓
Pod (restricted PSS, non-root, RO filesystem)

All layers enforce trust boundaries. Compromise at one level doesn’t cascade.

Implementation Checklist

Phase 1: Secrets Encryption (Week 1)

• Generate age key and store securely
• Create .sops.yaml in your repo
• Encrypt all existing secrets with SOPS
• Add SOPS decryption key to Flux
• Verify Flux can decrypt and deploy secrets

Phase 2: Network Policies (Week 2)

• Identify sensitive namespaces (production, databases, payments)
• Apply default-deny-ingress to each
• Add explicit allow policies (ingress-nginx, monitoring, kube-system)
• Test connectivity (curl from pods)
• Document which namespaces have policies

Phase 3: RBAC (Week 3)

• Audit existing service accounts and RBAC rules
• Create least-privilege roles for each workload
• Bind roles to service accounts
• Test that apps work with new RBAC (permission errors indicate overly strict)

Phase 4: Pod Security (Week 4)

• Label namespaces with pod-security.kubernetes.io/enforce: restricted
• Update deployments to pass restricted policy (non-root, RO FS, no privileged, no hostPath)
• Test deployments in non-production first
• Roll out to production

Phase 5: Credential Rotation (Ongoing)

• Set up credential rotation schedule (90d for passwords, 180d for API keys)
• Document rotation procedures for each credential type
• Automate where possible (External Secrets Operator for cloud credentials)
• Test rotation in staging before rotating in production

Phase 6: Scanning (Ongoing)

• Install truffleHog and scan existing repo
• Fix any found credentials (rotate, encrypt, commit fix)
• Set up git-secrets pre-commit hook
• Add truffleHog to CI/CD pipeline (fail build if credentials detected)

Common Mistakes

❌ Storing decryption keys in git

SOPS age public key can be in git. The private key must not. Store it in:

• Your laptop only
• Your secret manager (AWS Secrets Manager, HashiCorp Vault)
• Never in git, CI/CD environment variables, or container images

❌ Network policies that break observability

If you restrict all traffic, monitoring can’t scrape metrics. Add explicit allow from monitoring namespace:

1
2
3
4
5
6
7

- from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
  ports:
    - port: 8080
      protocol: TCP

❌ Pod security policies too strict

Some apps need to write to disk or bind to privileged ports. Test in dev first:

1
2
3

pod-security.kubernetes.io/enforce: baseline  # Less strict, still safe
# vs
pod-security.kubernetes.io/enforce: restricted  # Very strict

❌ Forgetting RBAC for stateless jobs

Jobs need ServiceAccounts too:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

apiVersion: batch/v1
kind: Job
metadata:
  name: backup-job
spec:
  template:
    spec:
      serviceAccountName: backup-runner  # Explicit service account
      containers:
        - name: backup
          image: backup-tool:latest

❌ Rotating a secret but not restarting pods

Pods cache environment variables. Change a secret, and old pods still have old values. You must:

1. Update the secret
2. Restart the pods (via rollout restart or new deployment)

1
2
3
4
5

# Update secret
kubectl patch secret database-password -p '{"stringData":{"password":"new-pass"}}'

# Restart pods to pick up new value
kubectl rollout restart deployment/my-app -n production

What’s Next

This covers the core. From here:

Audit Logging

Log who accessed secrets and when:

1
2
3

# Enable audit logging in kube-apiserver
# Log all secret GET/DELETE operations
# Ship logs to ELK or Loki for investigation

Secrets Rotation at Scale

Coordinate rotation across multiple environments:

1
2
3
4

# Week 1: Rotate in dev
# Week 2: Rotate in staging
# Week 3: Rotate in production
# Alert on rotation failures

Zero-Trust Across Clusters

If you run multiple clusters:

1
2
3

# Implement network policies within each cluster
# Add service mesh (Istio, Linkerd) for cross-cluster mTLS
# Sync secrets securely across clusters (not recommended: keep per-cluster)

Summary

You now have:

• Secrets Encryption: SOPS + age, encrypted at rest in etcd and in git
• Network Isolation: Default-deny ingress, explicit allows for known traffic
• Access Control: RBAC enforcing least privilege, Pod Security Standards
• Credential Rotation: Scheduled rotation for all credential types
• Leak Detection: Automated scanning for credentials in code

This is production-grade zero-trust. Attackers need to compromise multiple layers (network, access, secrets encryption, pod security) to break in. Any single compromise doesn’t cascade.

Deploy it in phases. Test in dev first. And remember: security is a practice, not a destination.

Implementing zero-trust on Kubernetes? We take on short-term infrastructure engagements and platform work.

Connect on LinkedIn →