Quick Start Guide

For those eager to dive in without delay, here’s a streamlined setup to get your pipeline operational:

The provided YAML snippet sets up a GitHub Actions workflow that utilizes a reusable workflow from my repository. This approach eliminates the need for you to duplicate and maintain a lengthy YAML configuration filled with actions. All you need to do is place this YAML file in the .github/workflows/ directory of your repository and adjust the parameters in the with: section as desired.

The parameters come with sensible defaults, and you can skip the entire with: section if you trust my recommendations. Otherwise, feel free to modify them according to the examples shown above, along with other options found in the workflow definition. For guidance on locating values and configuring required secrets for integrations such as Sonar or CodeClimate, refer to the README in the repository or the following sections for detailed explanations.

The workflow makes certain assumptions about your repository's contents; it's expected that there is a source code directory and a Dockerfile, while other files are optional. To see an example layout, check out the testing repository.

As you might have noticed, the snippet references a specific version of the workflow using @v1.0.0 to prevent unwanted changes. However, I encourage you to periodically check the repository for updates, as improvements and new releases may be available.

The video titled "Day-4 | Real-Time CI CD Project Pipeline | DevOps Project" provides a comprehensive overview of setting up CI/CD for your projects, showcasing practical steps to streamline your development process.

The Basics of the Pipeline

By now, you might have your pipeline configured, but let’s delve into its inner workings and explore ways to further customize it. We begin with the fundamentals—checking out the repository, installing dependencies, and executing tests:

The steps in the above snippet have been simplified for clarity, but they should be straightforward if you're familiar with GitHub Actions. If you’re not, there’s no need to worry; the important takeaway is that the pipeline is compatible with all major Python dependency managers, including pip, Poetry, and Pipenv. Simply set the DEPENDENCY_MANAGER variable, and the pipeline will handle the rest, provided you have a requirements.txt, poetry.lock, or Pipfile.lock in your repository.

The aforementioned steps also establish a Python virtual environment used throughout the pipeline for isolated builds and tests, allowing for dependency caching to reduce runtime. Notably, cached dependencies persist across branches as long as the lock file remains unchanged.

For testing, we utilize pytest to execute your test suite. pytest will automatically detect any configuration files present in the repository, following this order of precedence: pytest.ini, pyproject.toml, tox.ini, or setup.cfg.

Code Quality Assurance

In addition to basic functionality, it’s crucial to enforce code quality standards. Numerous tools are available to ensure your Python code remains clean and maintainable, and this pipeline integrates all of them:

We begin by running Black, a code formatter for Python. It’s advisable to use Black as a pre-commit hook or to execute it each time you save a file, so this step serves as a verification to catch any oversights.

Following that, we apply Flake8 and Pylint to implement additional styling and linting rules beyond what Black covers. Both tools are configurable through their respective config files, which will be automatically recognized. Flake8 can utilize options from setup.cfg, tox.ini, or .flake8, while Pylint can refer to pylintrc, .pylintrc, or pyproject.toml.

All the aforementioned tools can be set to enforcing mode, causing the pipeline to fail if issues are detected. The configuration options for this are ENFORCE_PYLINT, ENFORCE_BLACK, and ENFORCE_FLAKE8.

Furthermore, the pipeline features two popular external tools—SonarCloud and CodeClimate. While these are optional, I highly recommend using them for any public repository. When enabled (using ENABLE_SONAR and/or ENABLE_CODE_CLIMATE), the Sonar scanner will analyze your code and send the results to SonarCloud, while CodeClimate will generate a coverage report based on the output from pytest.

To integrate these tools, simply include their respective configuration fields in your pipeline setup and follow the instructions in the repository README to establish the necessary secret values.

The video "Intro to CI/CD with Python" presented by Chris Arceneaux covers the foundational principles and best practices for implementing CI/CD in Python projects, providing valuable insights for developers.

Packaging Your Application

Once we’ve confirmed that our code meets the required standards, it’s time to package it, specifically in the form of a container image:

This pipeline defaults to using the GitHub Container Registry associated with your repository. If this is your preference, no additional configuration is necessary, aside from providing a Dockerfile.

If you wish to utilize Docker Hub or another registry, you can specify CONTAINER_REGISTRY and CONTAINER_REPOSITORY, along with credentials in CONTAINER_REGISTRY_USERNAME and CONTAINER_REGISTRY_PASSWORD (in the secrets section of the pipeline configuration), and the pipeline will manage the rest.

In addition to the standard login, build, and push sequence, this pipeline generates supplementary metadata for the image, tagging it with the commit SHA and any applicable git tags.

To enhance pipeline efficiency, caching is employed during the Docker build process to avoid unnecessary image layer rebuilding. Additionally, the Dive tool assesses the image’s efficiency, allowing you to provide a config file in .dive-ci to set thresholds for its metrics. As with other tools in this pipeline, Dive can also operate in enforcing or non-enforcing modes via ENFORCE_DIVE.

Security Measures

Ensuring the integrity of your codebase is crucial, and this workflow incorporates several tools for vulnerability scanning:

The first tool is Bandit, which identifies common security vulnerabilities within your Python code. Bandit comes with a default set of rules but can be customized using a configuration file specified in the BANDIT_CONFIG option of the workflow. Like the other tools, Bandit runs in non-enforcing mode by default, but can be switched to enforcing mode with the ENFORCE_BANDIT option.

Another tool included in the pipeline is Trivy by Aqua Security, which scans the container image for vulnerabilities beyond just the Python code. The findings are uploaded to GitHub Code Scanning, where they will be visible in the repository’s Security tab.

It's imperative to not only secure the application but also to verify the authenticity of the final container image to prevent supply chain attacks. To achieve this, we employ a tool that signs the image with GitHub's OIDC token, linking the image to the identity of the user who pushed the code to the repository. This signature is generated without any key requirements and is pushed alongside your image to the container registry.

For further details about signing container images with Cosign, check out GitHub's blog.

Notification System

An additional feature of this pipeline is the Slack notification, which can be configured to trigger for both successful and failed builds—provided you enable it with ENABLE_SLACK. You only need to supply the Slack channel webhook using the SLACK_WEBHOOK repository secret. Follow the README instructions to generate the webhook.

Closing Thoughts

That covers everything you need to establish a fully configured CI pipeline from start to finish. If the features or customization options do not align perfectly with your requirements, feel free to fork the repository or submit an issue for feature requests.

In addition to new features, more pipeline configurations for Python (or other languages) may be added in the future, as the pipeline outlined here may not suit every application type. Be sure to periodically check the repository for updates and new releases.

This article was originally published at martinheinz.dev.