A detailed account of building a production-ready GitOps workflow - from containerizing apps to automated deployments with Flux CD, including all the debugging, fixes, and lessons learned.

This article documents the complete journey of setting up an automated GitOps pipeline, including every roadblock encountered and how they were resolved.

Table of Contents

  1. Introduction
  2. The Architecture
  3. Phase 1: App Containerization
  4. Phase 2: Setting Up the GitOps Repo
  5. Phase 3: Building the CI/CD Pipeline
  6. Phase 4: Debugging and Fixing
  7. Phase 5: Flux CD Integration
  8. The Complete Workflow
  9. Lessons Learned
  10. Resources

Introduction

Modern app deployment requires automation, reliability, and declarative config. GitOps provides all three by treating Git as the single source of truth for infrastructure and app state. This post documents building a complete GitOps pipeline from scratch for a Python study app.

GitHub Repos:

All code examples in this post can be found in these repos. Feel free to explore, fork, and adapt for your own projects.

The App: A full-stack Python study tracker app consisting of:

  • Backend: FastAPI REST API for managing study sessions
  • Frontend: Flask web app for the user interface
  • Both services containerized and deployed to Kubernetes

What we built:

  • Automated Docker image builds
  • GitOps repo for Kubernetes manifests (separate repo: devops-study-app-gitops)
  • GitHub Actions CI/CD pipeline
  • Flux CD for continuous deployment
  • Automated image tag updates
  • Pull request workflow for production changes

Development Tools Used:

  • uv: Ultra-fast Python package installer (10x faster than pip)
  • mise: Development env and tool version manager
  • commitizen: Standardized git commit messages with semantic versioning
  • k3d: Lightweight Kubernetes in Docker for local development
  • pre-commit: Automated code quality checks (ruff, ruff-format)
  • GitHub Copilot: Automated PR summaries and code reviews

The Challenge: Starting with a basic Python app, we needed to:

  • Containerize frontend and backend services
  • Structure Kubernetes manifests using Kustomize
  • Set up local k3d cluster for testing
  • Automate the entire deployment pipeline
  • Handle multiple envs (dev/prod)
  • Maintain separate repos for app code and GitOps manifests
  • Debug numerous workflow issues

The Architecture

Components Overview

┌─────────────────────────────────────────────────────────────┐
│                    Developer Workflow                        │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
                  ┌──────────────────┐
                  │  Push Git Tag    │
                  │ backend-v0.0.5   │
                  └──────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                   App Repo                     │
│  github.com/username/devops-study-app                       │
│                                                              │
│  ┌────────────────────────────────────────────────┐        │
│  │         GitHub Actions Workflow                 │        │
│  │                                                  │        │
│  │  1. Build Docker Image                          │        │
│  │  2. Push to GitHub Container Registry (GHCR)    │        │
│  │  3. Trigger GitOps Update Workflow              │        │
│  └────────────────────────────────────────────────┘        │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
                  ┌──────────────────┐
                  │  Docker Image    │
                  │ ghcr.io/user/app │
                  │  :backend-v0.0.5 │
                  └──────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                    GitOps Repo                         │
│  github.com/username/devops-study-app-gitops                │
│                                                              │
│  ┌────────────────────────────────────────────────┐        │
│  │    Automated Update Workflow                    │        │
│  │                                                  │        │
│  │  1. Clone GitOps repo                           │        │
│  │  2. Update kustomization.yaml with new tag     │        │
│  │  3. Create Pull Request                         │        │
│  └────────────────────────────────────────────────┘        │
│                                                              │
│  apps/                                                       │
│  ├── dev/kustomization.yaml                                │
│  └── prod/kustomization.yaml                               │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
                  ┌──────────────────┐
                  │   Merge PR       │
                  │  (Manual Review) │
                  └──────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                   Kubernetes Cluster                         │
│  (k3d-study-app-cluster)                                    │
│                                                              │
│  ┌────────────────────────────────────────────────┐        │
│  │              Flux CD                            │        │
│  │                                                  │        │
│  │  1. Monitors GitOps repo every 1 minute        │        │
│  │  2. Detects changes in kustomization.yaml      │        │
│  │  3. Applies changes to cluster                 │        │
│  │  4. Rolling update with zero downtime          │        │
│  └────────────────────────────────────────────────┘        │
│                                                              │
│  Namespaces:                                                 │
│  ├── study-app (dev)                                        │
│  │   ├── dev-backend-pod  (backend-v0.0.5)                 │
│  │   └── dev-frontend-pod (frontend-v0.2.0)                │
│  └── study-app (prod)                                       │
│      ├── prod-backend-pod  (backend-v0.0.5)                 │
│      └── prod-frontend-pod (frontend-v0.2.0)                │
└─────────────────────────────────────────────────────────────┘

Repo Structure

Two Separate Repos:

This setup follows GitOps best practices by separating app code from deployment manifests:

  1. App Repo (devops-study-app): Contains source code, Dockerfiles, and CI/CD workflows
  2. GitOps Repo (devops-study-app-gitops): Contains Kubernetes manifests and Flux config

App Repo Structure:

devops-study-app/
├── .github/
│   └── workflows/
│       ├── docker-build-push.yaml      # Main build workflow
│       ├── update-gitops.yaml          # GitOps update workflow
│       ├── backend-tests.yaml          # Backend testing
│       ├── frontend-tests.yaml         # Frontend testing
│       ├── e2e-tests.yaml              # End-to-end tests
│       └── copilot-code-review.yaml    # Automated PR reviews with Copilot
├── src/
│   ├── backend/
│   │   ├── Dockerfile                  # Multi-stage Python build
│   │   ├── pyproject.toml              # Backend dependencies (uv)
│   │   ├── uv.lock                     # Lockfile for reproducible builds
│   │   └── src/backend/main.py         # FastAPI app
│   └── frontend/
│       ├── Dockerfile                  # Multi-stage Flask build
│       ├── pyproject.toml              # Frontend dependencies (uv)
│       ├── uv.lock                     # Lockfile for reproducible builds
│       └── app.py                      # Flask app
├── scripts/
│   └── update_kustomize_tag            # Shell script for tag updates
├── kubernetes/
│   └── k3d-config.yaml                 # Local cluster config
├── mise.toml                           # Tool version management
├── .pre-commit-config.yaml             # Pre-commit hooks (ruff, commitizen)
└── .release-please-config.json         # Automated versioning

GitOps Repo Structure:

devops-study-app-gitops/
├── apps/
│   ├── base/                           # Base Kustomize configs
│   │   ├── backend/
│   │   │   ├── deployment.yaml
│   │   │   ├── service.yaml
│   │   │   └── kustomization.yaml
│   │   └── frontend/
│   │       ├── deployment.yaml
│   │       ├── service.yaml
│   │       └── kustomization.yaml
│   ├── dev/                            # Dev overlays
│   │   ├── kustomization.yaml          # Dev-specific config
│   │   ├── namespace.yaml
│   │   └── backend/kustomization.yaml
│   └── prod/                           # Prod overlays
│       ├── kustomization.yaml          # Prod-specific config
│       ├── namespace.yaml
│       └── backend/kustomization.yaml
└── clusters/
    └── dev/
        ├── flux-system/                # Flux config
        │   ├── gotk-components.yaml
        │   ├── gotk-sync.yaml
        │   └── kustomization.yaml
        └── apps.yaml                   # App deployment config

Phase 1: App Containerization

Backend Dockerization (FastAPI + Python)

The backend is a FastAPI app using Python 3.13. We used a multi-stage build with uv for fast dependency installation.

Why uv?

  • 10x faster than pip for dependency resolution
  • Drop-in replacement for pip/pip-tools/poetry
  • Deterministic builds with uv.lock
  • Excellent Docker layer caching support

See the complete implementation:

Backend Dockerfile (excerpt):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

# Build stage
FROM ghcr.io/astral-sh/uv:latest AS uv
FROM python:3.13-alpine AS builder

# Copy uv binary
COPY --from=uv /uv /uvx /bin/

WORKDIR /app

# Install dependencies (cached layer)
RUN --mount=type=cache,target=/root/.cache/uv \
  --mount=type=bind,source=uv.lock,target=uv.lock \
  --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
  uv sync --locked --no-install-project --no-editable

# Copy app code
COPY . /app

# Build app
RUN --mount=type=cache,target=/root/.cache/uv \
  uv sync --locked --no-editable

# Runtime stage
FROM python:3.13-alpine

# Create non-root user
RUN addgroup -S -g 1000 app && adduser -S -u 1000 -G app app

# Copy virtual env from builder
COPY --from=builder --chown=app:app /app/.venv /app/.venv

# Set up env
USER app
WORKDIR /app
ENV PATH="/app/.venv/bin:$PATH"
ENV PYTHONUNBUFFERED=1

# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"

EXPOSE 8000
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8000"]

Key optimizations:

  • Multi-stage build reduces final image size
  • Layer caching for dependencies (only rebuilds when dependencies change)
  • Non-root user for security
  • Health checks for Kubernetes readiness probes
  • uv for fast dependency resolution (10x faster than pip)

Frontend Dockerization (Flask + Python)

Similar approach for the Flask frontend.

See the complete implementation:

Frontend Dockerfile (excerpt):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

FROM ghcr.io/astral-sh/uv:latest AS uv
FROM python:3.13-alpine AS builder

COPY --from=uv /uv /uvx /bin/
WORKDIR /app

RUN --mount=type=cache,target=/root/.cache/uv \
  --mount=type=bind,source=uv.lock,target=uv.lock \
  --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
  uv sync --locked --no-install-project --no-editable

COPY . /app
RUN --mount=type=cache,target=/root/.cache/uv \
  uv sync --locked --no-editable --no-dev

FROM python:3.13-alpine
RUN addgroup -S -g 1000 app && adduser -S -u 1000 -G app app
COPY --from=builder --chown=app:app /app/.venv /app/.venv

USER app
WORKDIR /app
ENV PATH="/app/.venv/bin:$PATH"

EXPOSE 5000
CMD ["python", "-m", "flask", "run", "--host=0.0.0.0"]

Local Testing with k3d

Before pushing to CI, test locally using k3d - a lightweight Kubernetes distribution that runs in Docker.

Why k3d?

  • Fast cluster creation (~30 seconds)
  • Minimal resource usage
  • Perfect for CI/CD pipelines
  • Compatible with standard kubectl commands

Set up local k3d cluster:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# Create cluster using config
k3d cluster create --config kubernetes/k3d-config.yaml

# Verify cluster
kubectl cluster-info
kubectl get nodes

# Build and test images locally
docker build -t backend:dev -f src/backend/Dockerfile src/backend
docker build -t frontend:dev -f src/frontend/Dockerfile src/frontend

# Import images into k3d
k3d image import backend:dev -c study-app-cluster
k3d image import frontend:dev -c study-app-cluster

# Deploy with kustomize
kubectl apply -k kubernetes/manifests/dev

# Test endpoints
kubectl port-forward svc/dev-backend 8000:8000 -n study-app &
curl http://localhost:8000/health

# Check logs
kubectl logs -f deployment/dev-backend -n study-app

# Cleanup
k3d cluster delete study-app-cluster

k3d Config (kubernetes/k3d-config.yaml):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

apiVersion: k3d.io/v1alpha5
kind: Simple
metadata:
  name: study-app-cluster
servers: 1
agents: 1
ports:
  - port: 22111:80
    nodeFilters:
      - loadbalancer
  - port: 22112:443
    nodeFilters:
      - loadbalancer

Phase 2: Setting Up the GitOps Repo

Kustomize Structure

Kustomize allows us to define base configs and env-specific overlays without duplicating YAML.

See the complete manifests:

Base Backend Deployment (excerpt):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

# apps/base/backend/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: study-app
spec:
  replicas: 1
  selector:
    matchLabels:
      component: backend
  template:
    metadata:
      labels:
        component: backend
    spec:
      containers:
        - name: backend
          image: ghcr.io/username/study-app-api:latest
          ports:
            - containerPort: 8000
          resources:
            limits:
              memory: "512Mi"
            requests:
              cpu: "100m"
              memory: "128Mi"
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 10
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 5
            periodSeconds: 10

Base Backend Service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# apps/base/backend/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: backend
  namespace: study-app
spec:
  selector:
    component: backend
  ports:
    - protocol: TCP
      port: 8000
      targetPort: 8000
  type: ClusterIP

Base Kustomization:

1
2
3
4
5
6
7

# apps/base/backend/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml
  - service.yaml

Env Overlays

Dev Env:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# apps/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - namespace.yaml
  - ../base/backend
  - ../base/frontend

namespace: study-app
namePrefix: dev-

# Image tags are updated by CI/CD pipeline
images:
  - name: ghcr.io/username/study-app-api
    newTag: backend-v0.0.5
  - name: ghcr.io/username/study-app-web
    newTag: frontend-v0.2.0

Prod Env:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# apps/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - namespace.yaml
  - ../base/backend
  - ../base/frontend

namespace: study-app
namePrefix: prod-

# Prod uses stable, tested tags
images:
  - name: ghcr.io/username/study-app-api
    newTag: backend-v0.0.5
  - name: ghcr.io/username/study-app-web
    newTag: frontend-v0.2.0

Testing Kustomize Locally

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Preview what will be applied
kubectl kustomize apps/dev
kubectl kustomize apps/prod

# Apply to k3d cluster
kubectl apply -k apps/dev
kubectl apply -k apps/prod

# Verify deployments
kubectl get pods -n study-app
kubectl get svc -n study-app

# Watch pod status
watch kubectl get pods -n study-app

Testing the complete flow locally:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# 1. Start k3d cluster
k3d cluster create --config kubernetes/k3d-config.yaml

# 2. Install Flux (optional for local testing)
flux bootstrap github \
  --owner=username \
  --repo=devops-study-app-gitops \
  --branch=main \
  --path=clusters/dev \
  --personal

# 3. Verify Flux is syncing
flux get sources git
flux get kustomizations

# 4. Watch deployments
kubectl get pods -n study-app -w

Phase 3: Building the CI/CD Pipeline

Docker Build and Push Workflow

See the complete workflows:

Main Workflow (excerpt):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

name: Build and Push Docker Images

on:
  push:
    tags:
      - "*-v*"  # Matches backend-v1.0.0, frontend-v2.1.0, etc.

env:
  REGISTRY: ghcr.io
  BACKEND_IMAGE_NAME: ${{ github.repo_owner }}/study-app-api
  FRONTEND_IMAGE_NAME: ${{ github.repo_owner }}/study-app-web

jobs:
  build-and-push-backend:
    name: Build and Push Backend
    runs-on: ubuntu-latest
    # Only run for backend tags
    if: contains(github.ref, 'backend')
    
    permissions:
      contents: read
      packages: write
    
    outputs:
      tag: ${{ steps.tag.outputs.TAG }}
      image: ${{ env.REGISTRY }}/${{ env.BACKEND_IMAGE_NAME }}
    
    steps:
      - name: Checkout repo
        uses: actions/checkout@v4
      
      - name: Log in to the Container registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.repo_owner }}
          password: ${{ secrets.GITHUB_TOKEN }}
      
      - name: Extract tag name
        id: tag
        run: echo "TAG=${GITHUB_REF#refs/tags/}" >> $GITHUB_OUTPUT
      
      - name: Build and push Backend Docker image
        uses: docker/build-push-action@v5
        with:
          context: ./src/backend
          push: true
          tags: |
            ${{ env.REGISTRY }}/${{ env.BACKEND_IMAGE_NAME }}:${{ steps.tag.outputs.TAG }}
            ${{ env.REGISTRY }}/${{ env.BACKEND_IMAGE_NAME }}:latest

  build-and-push-frontend:
    name: Build and Push Frontend
    runs-on: ubuntu-latest
    if: contains(github.ref, 'frontend')
    
    permissions:
      contents: read
      packages: write
    
    outputs:
      tag: ${{ steps.tag.outputs.TAG }}
      image: ${{ env.REGISTRY }}/${{ env.FRONTEND_IMAGE_NAME }}
    
    steps:
      - name: Checkout repo
        uses: actions/checkout@v4
      
      - name: Log in to the Container registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.repo_owner }}
          password: ${{ secrets.GITHUB_TOKEN }}
      
      - name: Extract tag name
        id: tag
        run: echo "TAG=${GITHUB_REF#refs/tags/}" >> $GITHUB_OUTPUT
      
      - name: Build and push Frontend Docker image
        uses: docker/build-push-action@v5
        with:
          context: ./src/frontend
          push: true
          tags: |
            ${{ env.REGISTRY }}/${{ env.FRONTEND_IMAGE_NAME }}:${{ steps.tag.outputs.TAG }}
            ${{ env.REGISTRY }}/${{ env.FRONTEND_IMAGE_NAME }}:latest

  trigger_gitops:
    name: Trigger GitOps
    needs: [build-and-push-backend, build-and-push-frontend]
    if: always() && (needs.build-and-push-backend.result == 'success' || needs.build-and-push-frontend.result == 'success')
    uses: ./.github/workflows/update-gitops.yaml
    with:
      tag: ${{ needs.build-and-push-backend.outputs.tag || needs.build-and-push-frontend.outputs.tag }}
      image: ${{ needs.build-and-push-backend.outputs.image || needs.build-and-push-frontend.outputs.image }}
    secrets:
      GITOPS_DEPLOY_KEY: ${{ secrets.GITOPS_DEPLOY_KEY }}
      DEVOPS_STUDY_APP: ${{ secrets.DEVOPS_STUDY_APP }}

Phase 4: Debugging and Fixing

Issue 1: Wrong GitOps Repo Reference

Problem:

1

repo: mischavandenburg/devops-study-app-gitops  # Wrong repo!

Symptom: GitOps workflow succeeded but changes appeared in the wrong repo.

Fix:

1
2

sed -i 's|mischavandenburg/devops-study-app-gitops|username/devops-study-app-gitops|' \
  .github/workflows/update-gitops.yaml

Lesson: Always verify repo references match your actual repo names.

Issue 2: Output Variable Name Mismatch

Problem:

1
2
3
4
5
6
7
8
9

# Workflow defined outputs as 'tag' and 'image'
outputs:
  tag: ${{ steps.tag.outputs.TAG }}
  image: ${{ env.REGISTRY }}/${{ env.BACKEND_IMAGE_NAME }}

# But trigger job used different names
with:
  tag: ${{ needs.build-and-push-backend.outputs.tags }}  # 'tags' not 'tag'!
  image: ${{ needs.build-and-push-backend.outputs.app }}  # 'app' not 'image'!

Symptom: GitOps PR created with empty tag:

1
2
3

images:
  - name: ghcr.io/username/study-app-api
    newTag: ""  # Empty!

Fix:

1
2
3
4
5

# Fix plural to singular
sed -i 's/outputs\.tags/outputs.tag/g' .github/workflows/docker-build-push.yaml

# Fix app to image
sed -i 's/outputs\.app/outputs.image/g' .github/workflows/docker-build-push.yaml

Lesson: Output variable names must match exactly between job definition and usage.

Issue 3: Workflow File Extension Mismatch

Problem:

1

uses: ./.github/workflows/update-gitops.yml  # .yml extension

But actual file was named .yaml

Fix:

1
2

sed -i 's/update-gitops.yml/update-gitops.yaml/' \
  .github/workflows/docker-build-push.yaml

Lesson: Be consistent with file extensions throughout the project.

Issue 4: Implementing GitHub Copilot PR Automation

Goal: Automate PR summaries and code reviews using GitHub Copilot.

Implementation:

Copilot PR Review Workflow (.github/workflows/copilot-code-review.yaml):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

name: Copilot Code Review

on:
  pull_request:
    types: [opened, synchronize, reopened]

permissions:
  contents: read
  pull-requests: write

jobs:
  copilot-review:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Run Copilot Code Review
        uses: github/copilot-code-review-action@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}

Benefits:

  • Automatically generates PR descriptions from commits
  • Performs automated code review on every PR
  • Identifies potential bugs and security issues
  • Suggests improvements and best practices
  • Reduces manual review time

Example Copilot PR Comment:

GitHub Copilot Review

Summary:
- Updated backend image tag from v0.0.4 to v0.0.5
- Changes affect production kustomization.yaml
- No breaking changes detected

Code Quality:
Image tag format follows semantic versioning
Kustomization syntax is valid
 Consider adding resource limits in production

Suggestions:
- Add rollback strategy documentation
- Update changelog with deployment notes

Lesson: AI-powered code review catches issues early and improves PR quality without manual overhead.

Phase 5: Flux CD Integration

Installing Flux

Bootstrap Flux to your cluster:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Export GitHub token
export GITHUB_TOKEN=<your-token>

# Bootstrap Flux
flux bootstrap github \
  --owner=username \
  --repo=devops-study-app-gitops \
  --branch=main \
  --path=clusters/dev \
  --personal

Verifying Flux

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Check Flux components
flux check

# View sources
flux get sources git

# View kustomizations
flux get kustomizations

# Force immediate reconciliation
flux reconcile source git flux-system

The Complete Workflow

End-to-End Flow

  1. Developer makes changes using commitizen:
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

    # Make code changes
vim src/backend/main.py

# Stage changes
git add .

# Commit using commitizen (ensures semantic versioning)
cz commit
# Or use the commitizen CLI interactively
git commit  # Pre-commit hook will validate commit message

# Tag the release (semantic versioning)
git tag backend-v0.0.5
git push origin backend-v0.0.5

Commitizen ensures standardized commits:

feat(backend): add new study session endpoint
fix(frontend): resolve login page styling issue
docs: update API documentation
chore: bump dependencies to latest versions

  1. GitHub Actions builds image

    • Output: ghcr.io/username/study-app-api:backend-v0.0.5

  2. GitOps workflow updates manifests

    • Updates dev (direct push)
    • Creates PR for prod
    • GitHub Copilot automatically generates PR summary
    • Copilot performs automated code review

  3. Review PR with AI assistance:

    1
2
3
4
5
6
7
8

    # View PR with Copilot-generated summary
gh pr view 2 --repo username/devops-study-app-gitops

# Copilot provides:
# - Automated change summary
# - Code review comments
# - Potential issues flagged
# - Suggested improvements

  4. Merge PR:

    1

    gh pr merge 2 --repo username/devops-study-app-gitops --squash

  5. Flux deploys automatically

    • Detects new commit
    • Pulls new image
    • Rolling update with zero downtime

  6. Verify deployment:

    1
2

    kubectl get pods -n study-app
kubectl describe pod dev-backend-xxx -n study-app | grep Image:

Lessons Learned

Key Takeaways

  1. Separate repos for code and manifests - App repo and GitOps repo provide clear separation of concerns
  2. GitOps provides declarative infrastructure - Git is the single source of truth
  3. Automation reduces human error - Tag push → Deploy automatically
  4. Pull requests enable review - Production changes require approval
  5. Flux handles reconciliation - Continuous synchronization
  6. Multi-stage builds optimize images - Smaller, faster, more secure
  7. Kustomize reduces duplication - Base + overlays pattern scales well
  8. k3d enables local testing - Fast, lightweight Kubernetes for development
  9. uv accelerates Python builds - 10x faster than pip, perfect for CI/CD
  10. mise manages tool versions - Consistent envs across team
  11. commitizen enforces standards - Semantic versioning and clean git history
  12. GitHub Copilot reviews code - Automated PR summaries and reviews catch issues early
  13. Proper testing prevents issues - Test early, test often
  14. Debugging is part of the process - Learn from errors and iterate

Resources

Project Repos

Live Implementation:

Explore the code, workflows, and configs. Star the repos if you find them helpful!

Documentation

Tools Used

Development Env:

  • mise: Development env and tool version manager (replaces asdf/nvm)
  • uv: Ultra-fast Python package installer (10-100x faster than pip)
  • commitizen: Standardized git commits and semantic versioning
  • pre-commit: Automated code quality checks (ruff, ruff-format)
  • GitHub Copilot: AI-powered PR summaries and code reviews

Container & Orchestration:

  • Docker: Container runtime
  • k3d: Lightweight Kubernetes in Docker (perfect for local dev)
  • Flux CD: GitOps continuous delivery operator
  • Kustomize: Template-free Kubernetes config management

CI/CD:

  • GitHub Actions: Workflow automation
  • GitHub Container Registry (GHCR): Docker image hosting
  • Release Please: Automated changelog and version management

Repo Structure:

  • App Repo (devops-study-app): Source code, Dockerfiles, CI/CD
  • GitOps Repo (devops-study-app-gitops): Kubernetes manifests, Flux config

This post documents a real implementation journey, including all mistakes and solutions. GitOps is powerful, but requires understanding, patience, and iteration to get right.

Tags: #GitOps #Kubernetes #FluxCD #Docker #CICD #DevOps #Kustomize #Automation