Introduction:

Before we dive into our pipeline, let's first understand the concept of GitOps.

Why GitOps?

Consider a scenario where we have a Kubernetes cluster and someone alters configurations within it. How do we track what has changed? Who made the changes? When were these modifications made? The concept here is that if we can use a version control system like Git to manage application source code, why not use Git to manage Kubernetes cluster configurations? GitOps serves as a method for achieving Continuous Delivery for both applications and infrastructure using Git.

What is GitOps?

GitOps is a Continuous Delivery approach that employs Git as a single source of truth for both declarative infrastructure and applications. By defining our Kubernetes resources in a GitHub repository, we can utilize tools such as Argo CD to manage the deployment of these resources into our Kubernetes cluster. Argo CD ensures that the current state of resources is in alignment with our declared state consistently.

Architecture Overview:

We will create a CI/CD pipeline that is triggered by a push to the main branch of our repository. This pipeline will build the application, execute tests, and deploy the application to a Kubernetes cluster following GitOps principles. The pipeline will be established using GitHub Actions and ArgoCD.

The architecture of the project is as follows:

First, we have a GitHub repository that houses the application's source code. This repository is connected to GitHub Actions, which will facilitate the building and deployment of the application. The GitHub Actions workflow will be initiated by a push to the main branch of the repository.

The workflow will build the application, run tests, and push the Docker image to Docker Hub, subsequently updating the image tag in the manifest repository. The manifest repository is an independent GitHub repository that contains the Kubernetes manifests for the application. Argo CD will monitor this repository for changes. Once the image tag is updated in the manifest repository, Argo CD will automatically deploy the new version of the application to the Kubernetes cluster.

Architecture Diagram:

Prerequisites:

• A GitHub account
• A Docker Hub account
• A Kubernetes cluster
• Argo CD installed on the Kubernetes cluster

Steps:

Step 1: Launching an Ubuntu Instance:

Since I lack a Linux machine, I will launch an Ubuntu instance on AWS.

If you do not have one either, consider launching a Linux machine using cloud providers, VirtualBox, or any other method of your choosing. However, ensure that the Kubernetes cluster meets specific system requirements.

Remember to open the necessary ports on your EC2 instance's security group.

System Requirements for Kubeadm: - Linux operating system (Ubuntu, CentOS, etc.) - Minimum 2 GB of RAM - Minimum 2 CPU cores - 40 GB of disk space

System Requirements for Minikube: - Linux, macOS, or Windows operating system - Virtualization enabled - Minimum 2 CPU cores - Minimum 2 GB of RAM - 20 GB of disk space

Step 2: Setting up a K8S Cluster:

There are several methods to set up our K8S cluster. You can choose any method that suits you. I will use the Minikube method for ease of implementation. However, if you prefer Kubeadm or manual setup, that is also perfectly acceptable.

I will provide commands for Ubuntu. If you're using a different Linux distribution, please consult the official documentation for installation guidelines.

Let’s start by installing Docker on Ubuntu:

# Add Docker's official GPG key:

sudo apt-get update

sudo apt-get install ca-certificates curl

sudo install -m 0755 -d /etc/apt/keyrings

sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc

sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:

echo

"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu

$(. /etc/os-release && echo "$VERSION_CODENAME") stable" |

sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo systemctl enable docker

sudo usermod -aG docker ubuntu

Once Docker is installed, we can proceed with setting up Minikube and Kubernetes.

Install Minikube using the commands below:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb

sudo dpkg -i minikube_latest_amd64.deb

minikube start

Install kubectl using the commands below:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Step 3: Install Argo CD using kubectl:

There are various methods for installing Argo CD, including HA and Non-HA setups. However, for learning purposes, we will utilize the non-HA method below:

kubectl create namespace argocd

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Now, you can check all the resources created by Argo CD using the command:

kubectl -n argocd get all

Expose the Argo CD Server by editing the argocd-server service to change the type to NodePort.

kubectl edit svc argocd-server -n argocd

Change the type to NodePort and save the changes. Now, use:

kubectl -n argocd get svc

to retrieve the service name. Find the node port for your argocd-server service and access the web UI using http://node-ip:node-port.

If you set up your cluster using Minikube, follow the approach below:

• We need to perform some port-forwarding to access the Argo CD UI.

• In my case, I used Minikube to establish a Kubernetes Cluster. Therefore, I will install socat for port-forwarding.

  sudo apt-get install socat

Execute the following command to enable port-forwarding:

socat TCP4-LISTEN:8080,fork,reuseaddr TCP4:192.168.49.2:NodePort &

Now, we can access the Argo CD UI at http://NodeIP:8080. To retrieve the password for the admin user, use the command below in the cluster:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Now, you can log in to the Argo CD UI using the admin user and the password obtained from the command above.

We can now create Argo CD applications using the New App option.

Step 4: Creating an Application Repository with Workflows:

Next, we need to create a repository containing our application source code and define our CI Pipeline. I will use GitHub Actions as my CI tool because it is straightforward to set up, given that I am using GitHub as my source code repository. If you prefer Jenkins or another CI tool, you can implement the CI Pipeline with that tool. My aim is to make my approach clear.

The directory structure will look like this:

??? .github

? ??? workflows

? ??? test.yaml

? ??? docker-image.yaml

??? Dockerfile

??? app.py

??? requirements.txt

??? test_app.py

??? README.md

??? LICENSE

In the above-mentioned files, the README.md and LICENSE files are optional.

Let’s begin by creating our application configuration files:

• First, start with app.py. Create a file named app.py and paste the following code:

  from flask import Flask

  from urllib.parse import quote

  def create_app():

  app = Flask(__name__)

  @app.route('/')

  def home():

  return 'Well done! You Successfully deployed your flask app, Now you can start building your app.'

  return app

  if __name__ == '__main__':

  app = create_app()

  app.run(host='0.0.0.0', port=80, debug=True)

• Now, create a file named requirements.txt and paste the following code:

  flask==2.0.1

  pytest

  Werkzeug==2.0.0

• Then, create a test_app.py file and use the following code:

  import pytest

  from app import create_app

  from urllib.parse import quote

  @pytest.fixture

  def app():

  return create_app()

  @pytest.fixture

  def client(app):

  return app.test_client()

  def test_home(client):

  response = client.get('/')

  assert response.status_code == 200

  expected_text = 'Well done! You Successfully deployed your flask app, Now you can start building your app.'

  assert expected_text.encode() in response.data

Now we will create a `Dockerfile` for building a Docker image. This file will be utilized by our workflows to create a Docker image:

FROM python:3.9-slim

WORKDIR /app

COPY . /app

RUN pip install -r requirements.txt

EXPOSE 80

ENTRYPOINT ["python", "app.py"]

Next, let’s create our workflow files. For this, we need to create a `.github/workflows/` folder:

• Create a file named test.yaml, which will test our application code using various Python versions:

  name: Upload Python Package

  on:

  push:

  branches: ["main"]

  pull_request:

  branches: ["main"]

  jobs:

  build-and-test-python-app:

  runs-on: ubuntu-latest

  strategy:

  fail-fast: false

  matrix:

  python-version: ["3.9", "3.10", "3.11"]

  steps:

  • uses: actions/checkout@v4

  • name: Set up Python ${{ matrix.python-version }}

    uses: actions/setup-python@v4

    with:

    python-version: ${{ matrix.python-version }}

  • name: Install dependencies

    run: |

    python -m pip install --upgrade pip

    python -m pip install flake8 pytest

    if [ -f requirements.txt ]; then pip install -r requirements.txt; fi

  • name: Lint with flake8

    run: |

    flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics

    flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics

  • name: Test with pytest

    run: |

    pytest

• Then create a file named docker-image.yaml, which will build our Docker image with the tag as commit-id and push it to Docker Hub. After that, it will update our manifest repository with that new tag:

  name: Publish Docker image and Update Manifest Repo

  env:

  CONFIG_REPO_NAME: helm-python-flask

  on:

  push:

  branches: ["main"]

  jobs:

  image-push:

  runs-on: ubuntu-latest

  steps:

  • uses: actions/checkout@v4

  • name: Build the Docker image

    run: docker build --tag memathesh/pythonflaskapp:${{ github.sha }} .

  • name: Login to Docker Hub

    uses: docker/login-action@v3

    with:

    username: ${{ secrets.DOCKERHUB_USERNAME }}

    password: ${{ secrets.DOCKERHUB_PASSWORD }}

  • name: Build and push

    uses: docker/build-push-action@v5

    with:

    push: true

    tags: memathesh/pythonflaskapp:${{ github.sha }}

  Update-Config-Repo:

  runs-on: ubuntu-latest

  needs: image-push

  steps:

  • run: |

    echo "promoting into dev environment!"

    git config --global user.email [email protected] && git config --global user.name GitHub Actions

    echo "cloning config repo $CONFIG_REPO_NAME"

    git clone https://oauth2:${{ secrets.CI_TOKEN }}@github.com/${{ github.repository_owner }}/$CONFIG_REPO_NAME.git

    cd $CONFIG_REPO_NAME

    echo "checkout main branch"

    git checkout main

    echo "Updating Image tag in values.yaml file"

    sed -i "s,tag:.*,tag: ${GITHUB_SHA}," pythonflaskapp/values.yaml

    git add . && git commit -m "Updated Image tag to ${GITHUB_SHA}"

    git push

If you examine the above file, you'll notice I've defined three secrets: DOCKERHUB_USERNAME, DOCKERHUB_PASSWORD, and CI_TOKEN. You need to create GitHub repository secrets. To do this, follow these steps:

1. Navigate to your GitHub repository.
2. Click on the Settings tab.
3. Click on the Secrets link in the left sidebar.
4. Click on the New repository secret button.
5. Add the secret name and value, then click on the Add secret button.

Remember to keep the secret names as DOCKERHUB_USERNAME, DOCKERHUB_PASSWORD, and CI_TOKEN.

You can also fork my repository if you prefer not to create the files yourself. However, you will still need to create the secrets on your own.

Repository Link: Click here

Step 5: Creating the Manifest Repository:

Next, we will create a manifest repository. This repository will contain our K8S resource files. We can store our manifest files in various formats such as Helm, Kustomize, or as plain manifests. I will use the Helm format, but you may adjust it according to your preference.

The structure of our manifest repository will be:

??? pythonflaskapp

??? templates

? ??? NOTES.txt

? ??? _helpers.

? ??? deployment.yaml

? ??? service.yaml

???

??? Chart.yaml

??? values.yaml

??? LICENSE

??? README.md

In the listed files, README.md and LICENSE files are optional. We only need to create Chart.yaml, values.yaml, and a templates/ folder.

• Let’s create the Chart.yaml file with the following code:

  apiVersion: v2

  name: pythonflaskapp

  version: 0.0.1

  description: A Simple Helm Chart for Python Flask Application

  type: application

  keywords:

  • python
  • flask

  # sources:

  # - https://github.com/mathesh-me/helm-python-flask-app

  maintainers:

  • name: Mathesh M

    email: [email protected]

    url:

  icon: https://helm.sh/img/helm.svg

  appVersion: 0.0.1

• Please use the link below to obtain the files for the templates/ folder, as including all of them here would extend this blog significantly. Create files according to our directory structure and copy the code from the link provided.

Repository Link: Click here

• Next, create a file named values.yaml, which is used for providing values to variables defined in our K8S resource manifest files:

  deployment:

  replicas: 2

  image:

  owner: <dockerhub-username>

  repository: <image-name>

  tag: 9fc652bfc6d4cbc0a9d388b28e624e4012dab81c

  container:

  name: pythonflaskapp

  port: 80

  service:

  type: NodePort

  port: 80

  targetPort: 80

  nodePort: 30112

Step 6: Creating an Argo CD Application:

Now we will create our Argo CD application. There are three methods to create an application in Argo CD:

1. Web UI: Create an application using the Argo CD Web UI.
2. CLI: Create an application using the Argo CD CLI.
3. Git Repository: Define the application in a Git repository, and Argo CD will automatically create it. This is the recommended declarative approach.

I will demonstrate the declarative approach by creating our Argo CD app using a Kubernetes manifest file.

• Create a new file named application.yaml and paste the following code inside:

  apiVersion: argoproj.io/v1alpha1

  kind: Application

  metadata:

  name: pythonflaskapp

  namespace: argocd

  spec:

  destination:

  namespace: default

  server: https://kubernetes.default.svc

  project: default

  source:

  path: pythonflaskapp

  repoURL: https://github.com/mathesh-me/helm-python-flask

  targetRevision: HEAD

  syncPolicy:

  automated:

  prune: true

  selfHeal: true

• Now apply the command kubectl apply -f application.yaml. This will create our Argo CD application, and it will automatically sync our cluster with the resources declared in our manifest repository.

• Next, push some changes to the application repository. This will trigger the workflow and automatically update the image tag in the manifest repository. Since our Argo CD application is monitoring it, Argo CD will deploy the application to the K8S cluster automatically.

• You can check our application using the command:

  kubectl -n argocd get app

You will see that our application is synced automatically.

• Now, go to the Argo CD Web UI and check our application.
• If you look at the picture above, you can see the commit updated by our actions user to Update Image tag to 37e7d.... This indicates our commit ID is 37e7d, and our workflow has updated our image tag accordingly.
• You can also view your image tag in your Docker Hub account.
• Now let’s take a look at the K8S resources deployed by our application.

Step 7: Accessing Our Flask Application:

Now we can access our application using the http://node-ip/node-port URL. If you are using a Minikube cluster, do not forget to perform port forwarding using the command below:

socat TCP4-LISTEN:8081,fork,reuseaddr TCP4:192.168.49.2:30112 &

Step 8: Experimenting with Our Application:

Now, try making some changes in the application configuration repository and observe the workflow by navigating to the Actions tab in the application repository.

If you check the manifest repository, you will see the commit made by the GitHub Actions user to update our image tag.

We have successfully built an end-to-end CI/CD pipeline based on the GitOps approach.

Application Repository Link: Click here

Manifest Repository Link: Click here

If you found this helpful, please give me a clap and follow my profile. I will be sharing more projects and insights about Cloud and DevOps. If you have any questions, feel free to comment or reach out to me on LinkedIn.

Let’s connect on LinkedIn: LinkedIn Profile

Explore hands-on projects: My GitHub Account