Your Definitive GitOps Guide For DevOps Experts And Newbies

GitOps is a concept that not everyone knows about. I only realized this when I started preparing a paper and article on the GitOps topic. The basic definition of GitOps refers to "storing state in Git," but it is far from the only one and not the most important one.

Weaveworks first introduced the term, but its meaning is broader than just storing data in a repository. The word "GitOps" probably sounds more like a marketing ploy than a complete reflection of the methodology. The basic idea behind GitOps is to keep the state of an application continuously synchronized with its real-world environment, be it a Kubernetes cluster or another environment.

Let me introduce myself. I am Tanya Chayka. Previously, I worked for the Ukrainian hosting company CityHost. It is the largest hosting provider in the Ukraine. In this article, I want to share my GitOps implementation story, using CityHost as an example. My previous articles on Hackernoon are:

• Containerization In 2023: Strive for Maximum Modularity via the Cattle Model
• 8 Things Developers Forget in Application Migration To Kubernetes

I encourage you to read the articles above as they are no less valuable than this guide on GitOps.

Small intro

We had 2 in-house data centers where we hosted hundreds of thousands of websites and virtual machines. To manage our infrastructure and services, we made heavy use of Kubernetes. It allowed us to describe the infrastructure declaratively and manage the physical servers and all workloads.

Our infrastructure included:

• Physical servers without virtualization (bare metal)
• Vanilla Kubernetes
• 15 individual clusters
• Over 1000 nodes
• Significant number of stateful applications
• Several development teams worked together on this infrastructure

It's important to note that our specialty was using Kubernetes primarily to deploy stateful applications, of which we had almost 80%. Kubernetes played a key role in managing dozens of stateful applications. To ensure that multiple development teams could work together seamlessly, we needed to set up effective collaboration.

Let's break it down in more detail. To eliminate misunderstandings and ensure common terminology, let's define key terms.

Continuous Integration vs. Continuous Delivery

Let's start with the terms Continuous Integration and Continuous Delivery. Not everyone can tell how one differs from the other. Let's get to the bottom of it.

When discussing CI/CD, we usually mean some kind of Pipeline.

It runs and automates the build and testing, creates the release, and frequently delivers our application to the end environment (staging, prod).

Unfortunately, it doesn't always happen the way we want it to. Because we are dependent on external factors, the Pipeline can break, and the application stack can disconnect and have to be fixed manually. One of the important things that GitOps helps to solve is to greatly simplify pipelines by doing away with the stage of delivering the application to the final environments.

This diagram shows the complete life cycle of an application, from its planning and code development to the moment when the application is successfully deployed and functioning in the production environment. The 2 main environments highlighted here are Staging and Production. It is important to note that the cycle is not considered complete even after deployment to production, as it is necessary to continuously monitor the application and ensure it is running smoothly.

Continuous Integration

Continuous Integration is a set of practices that help customize the collaborative development process and automate many steps, including building, testing, and publishing releases. When developers push new changes into a branch, merge them into a master, or create a tag, automation starts the process: application images are built, automated testing is performed, and other steps are performed independent of the specific target environment.

At this stage, the application does not yet know exactly where it will be deployed. In theory, temporary test environments can be created at this stage for validation, but the main purpose of this stage is to provide automation of the build and test processes. However, the process of delivering the application to the real environment itself is not usually described by the CI methodology and may even be done manually.

Continuous Delivery

Continuous Delivery includes practices related to the specific delivery of the application to specific environments, such as staging or production. The delivery process in this stage is automated but may include manual activities.

For example, deployment to a staging environment may occur automatically, while deployment to a production environment may require manual manipulation. Deployment to production may require pressing an abstract "button,” provided that on the staging environment, the current version of the application has been successfully deployed and is running smoothly.

These are 2 important steps in the application development and delivery cycle, ensuring that the application continues to evolve and run smoothly in real-world environments.

There is a third term, Continuous Deployment. It means automating the delivery of the application to the production environment as well.

These highlighted steps are the tasks GitOps solves when delivering the application to the environment.

What is GitOps?

GitOps is a concept that combines Git as a single source of truth and a continuous process of synchronization from that source. In the context of GitOps, a Git repository contains a declarative description of the state of an application as it should be in a particular environment. The continuous synchronization process ensures that the environment always matches this desired state. An essential part of GitOps is observability, which provides real-time assurance that the application works correctly.

Let me explain the scheme in detail.

Suppose you have a Git repository with an application and manifest to deploy it to different environments. You also have a pipeline that automatically reacts to changes in the repository and deploys the application to Kubernetes. For example, it executes the kubectl apply or helm install commands to align the state of the deployed application in Kubernetes with the state described in the Git repository.

This may seem sufficient to call it GitOps, but according to the concept created by Weaveworks, there is another important element.

GitOps, as presented by Weaveworks, is based on the idea of a continuous reconciliation loop. This loop continuously monitors the source of truth (Git repository) and keeps its state constantly synchronized with the real world (Kubernetes cluster).

If you examine how controllers work in Kubernetes, you'll find similar logic, except that they keep track of state in the Kubernetes API rather than in the Git repository.

For example, there is a Deployment object that generates a ReplicaSet resource. Each ReplicaSet controller keeps track of its object and manages the creation and deletion of pods. If any pod is manually deleted, the ReplicaSet controller notices this and tries to restore the state according to the description in the object. The same happens with a ReplicaSet object. If it is deleted, the Deployment controller will create a new ReplicaSet to match the new state described in the Deployment. This logic works in the opposite direction as well. If we update the specification of a Deployment object, this results in the creation of a new ReplicaSet and pods.

This idea of continuous consistency inspired the creators of GitOps, based on the principles underlying Kubernetes controllers’ work.

That's right, GitOps assumes a dedicated GitOps operator or controller that performs functions similar to controllers in Kubernetes. However, instead of monitoring the Kubernetes API, this operator monitors a specific Git repository. It takes the declarative state stored in the Git repository and ensures that it is synchronized with the actual state in Kubernetes. This means that the GitOps operator takes data from the Git repository, applies it to the Kubernetes cluster, and ensures that the state always matches the description in the repository.

It is important to emphasize that GitOps is not necessarily tied to Git as a data source. While Git is the most common source, GitOps can use other systems to store declarative descriptions, such as S3 buckets or even Helm repositories. The key is that the system must be capable of providing a declarative application state that can be used for automatic synchronization and infrastructure management.

Thus, GitOps is a methodology that provides continuous alignment of the application state with its declarative description. This can be implemented using various data sources, not just Git.

5 best GitOps practices

Let's look at delivery automation solutions for Kubernetes using ArgoCD and Flux as examples.

2 repositories: 1 for sources, 1 for manifests

Using 2 Git repositories within GitOps, as suggested by the ArgoCD reference implementation, is an excellent practice to effectively manage application state and deployment.

Here's how it works:

1. First Git repository (Application Source Repository). This repository holds your application's source code, as well as Helm charts, Docker files, and other resources that are environment-independent (like Production or Staging). This repository contains everything you need to build and publish your application. This is typically the application source code and components.
2. Second Git repository (Application Manifest Repository). This repository contains the manifests that describe a specific application and how to deploy it to a Kubernetes cluster. There can be Helm charts with different values for different environments (e.g., Production and Staging), or they can be simple YAML manifests organized by folders. Importantly, this repository describes how the application should be deprecated in a particular environment. A GitOps operator (e.g., ArgoCD) uses this repository as a "single point of truth" and ensures that the Kubernetes state is synchronized with the description from this repository.

This approach provides a clear separation between the application and its deployment, which simplifies management and ensures deployment consistency. At the same time, the first repository (Application Source Repository) remains stable and environment-independent, allowing development teams to work on the application regardless of the infrastructure.

Also, the second repository (Application Manifest Repository) can be updated from the first repository using pipelines or other automation tools. This provides flexibility and the ability to respond quickly to changes in application source code, the need to make deployment changes, or the addition of new environments.

Select the appropriate method for organizing the repository

Using ArgoCD and Flux as fully multitenant systems provides flexibility in organizing the GitOps process for multiple development teams and applications. Here are the key aspects of this multitenant model:

1. Multiple Git repositories and Kubernetes clusters. ArgoCD and Flux can manage multiple Git repositories and synchronize their state with different Kubernetes clusters. This means you can deploy applications to different environments and manage them using the same GitOps operator.
2. Multi-project approach. You can organize repositories so that each development team has its own Git repository containing manifests for their application. This allows each team to manage their application independently and not be dependent on the others. This approach also simplifies permissions management and access control.
3. Monorepository and hybrid solutions. You can use a mono repository that contains manifests for all applications, or you can create separate Git repositories for each application. The hybrid option balances ease of management and isolation between teams. Each team has its mono repository that contains manifests for all of their applications within a single project.

Notably, the choice between a mono repository and separate repositories depends on the specifics of your organization and projects. Most importantly, ArgoCD and Flux can work with different models of Git repository organization to meet the needs of your development team.

This multitenant approach, with a central dashboard to monitor the status of all the repositories, allows you to centrally manage multiple applications and environments. It provides transparency and efficient infrastructure management. This approach has proven to be the simplest and avoiding Dependency hell.

Check manifests before committing

Validating manifests before they are deployed is an essential aspect of GitOps practices that helps ensure the reliability and security of application deployments. Let's look at methods for validating and parsing manifests in more detail:

1. Validation in the CI/CD Pipeline. Running helm template or kubectl apply --dry-run in the CI/CD Pipeline before merging changes into the main branch is a great practice. This helps identify syntax errors and manifest issues before they are applied to the actual cluster. It also helps to detect problems early and reduce recovery time.
2. Interactive Validation in ArgoCD. The ability to validate and verify manifests through the ArgoCD CLI in real-time, as well as the ability to update applications from the developer's local environment, provide convenience and flexibility for developers. However, this may violate GitOps principles as changes will not be reflected in the Git repository until they are committed.
3. Flux2 and nativity to Git. Flux implies that all changes must be committed to the Git repository before being applied to the cluster. This is consistent with the principles of GitOps, where Git serves as the source of truth and the only source of configuration. While Flux does not provide an option for interactive validation, it places a strong emphasis on ensuring the security and consistency of the deployment process through Git.

These practices have their benefits and can be utilized depending on your specific needs and level of confidence in the security of your deployment. The key is paying attention to validating and testing manifests before deploying them to a live cluster. This helps mitigate risk and ensures the stability of your GitOps process.

External factors should not alter manifests

This point is rather Argo-specific. It primarily concerns passwords and certificates. Before applying any changes to the cluster, ArgoCD generates a YAML file to apply to the cluster.

In ArgoCD terms, there is a desired and live state of your application. Desired is the manifest obtained from our source of truth (Git repository). Live is what is already there and running in the cluster.

ArgoCD compares them and tells you which fields should be changed. So if the helm template returns a different value every time, for example, a string with a random password, ArgoCD will endlessly try to bring them to the OutOfSync state. So manifests should be idempotent.

Remember that every field described in the Git repository will be checked against the one in Kubernetes.

Flux2 doesn't have this problem because it uses native helm to apply helm charts. Thus, all native helm functions will work in Flux2 as if you were deploying with a normal helm. Even an abstract hook to generate a random password won't give you any problems.

These 2 tools have different logic. Each of them has a right to exist. It's all about what is more important for you - strict control over the manifests you use or the convenience of using the tool you are used to.

Think about how and where you will keep secrets

Managing and securely storing secrets is an important part of any infrastructure, especially in Kubernetes and GitOps. ArgoCD does not provide a built-in mechanism for secret management and leaves it up to the users. In this context, it is important to choose an approach that meets your security and secret management requirements.

However, ArgoCD can be extended with custom plugins, or you can use separate controllers running on the Kubernetes side. Let's take a look at a few of them:

1. SOPS and git-crypt. These tools are suitable for encrypting files in a Git repository. SOPS has the advantage of integrating with various secret managers. However, both tools may require careful key management and access policies.
2. Bitnami Sealed Secrets. Sealed Secrets provides a simple way to encrypt secrets in a Git repository and decrypt them automatically in a cluster. It can be a good choice to get started, especially if you need a simple solution.
3. Banzai Cloud Bank-Vaults. This is a more sophisticated and powerful solution based on HashiCorp Vault. It allows for more granular secrets management and provides a high level of security. However, its configuration and management may require more effort and understanding of how Vault works.

The choice between these solutions depends on your specific security, complexity, and flexibility requirements. It is essential to consider security and secrets control to avoid leaks of sensitive information and ensure regulatory compliance.

How ArgoCD works?

ArgoCD is an open-source declarative, GitOps continuous delivery tool for Kubernetes. It helps automate and simplify the deployment and management of applications in Kubernetes clusters by using a GitOps workflow.

Let's consider the scheme of ArgoCD.

There are several deployments in the system, but the most exciting one is argocd-repo-server. This container includes Git and various manifest generation tools such as Helm, Kustomize, and others. It is important to note that argocd-repo-server does not have access to the Kubernetes API. The way ArgoCD works is that we first generate deployment-ready YAML files by executing the binary in the repo-server container. Then, argocd-application-controller synchronizes them with the current state in the Kubernetes cluster.

Argocd-server provides an attractive user interface for interacting with the system. It uses argocd-dex-server and argocd-redis for its operation, but their functions can be considered internal details that do not need to be discussed in detail.

Custom plugin

Let's see how to add a custom plugin to ArgoCD.

We have a ConfigMap in which we can describe configManagementPlugins. Each plugin has 2 stages: init and generate.

The init stage runs a script to prepare the repository to generate the manifest. For example, we can run a helm dependency update to download all dependencies for our helm chart. If using repository-level encryption, a decryption operation can be added here.

The generate stage is the command to retrieve the YAML file. In the case of the helm, here would be the helm template command. The manifests obtained in this way are applied to the cluster by an already separate GitOps engine.

How Flux2 works?

Flux2 is a software project that is part of the Cloud Native Computing Foundation (CNCF) landscape. It is an open-source tool designed to help automate the deployment and lifecycle management of applications and infrastructure in Kubernetes environments.

Flux2 works a little differently. It implements the Kubernetes pattern with custom resources and controllers.

Flux2 has a source controller that downloads changes from the source repo. In addition to Git, this could be S3 or the Helm Chart Repository, as well as a separate Kustomize controller and Helm controller that handles applying manifests to the cluster. If we wanted to extend Flux, we would have to write a separate controller and CustomResource.

ArgoCD vs. Flux2

Thanks to Flux2 controllers, the system supports all Helm and Kustomize features. However, it is worth considering that ArgoCD has some limitations in using Helm, as it works with it via the helm template. This can cause problems when using Kustomize Helm Charts, especially for features like lookup that do not work in ArgoCD. Some helm hooks may need to be reconsidered, as they are not executed by Helm but by ArgoCD's GitOps Engine and function differently.

ArgoCD's Image Updater also differs from Image Updater. It generates a file that can only be used with Helm or Kustomize, while Flux2 uses conditional comments, allowing you to substitute image names almost anywhere in the YAML file.

Regarding the security model, both tools use RBAC, but Flux2 offers a more straightforward approach by reusing standard Kubernetes policies, while ArgoCD has its own format. In addition, ArgoCD provides the ability to customize sync windows, which allows you to limit the deploys of certain clusters to only certain time slots, such as at night.

Regarding handling secrets, Flux2 provides SOPS support out of the box. ArgoCD can be a bit more complicated with this.

An important aspect is the observability of the system. ArgoCD offers wonderful and convenient dashboards out of the box, which is a big advantage in terms of observability. Flux2, on the other hand, does not have such nice dashboards but provides similar features through CLI utilities, notifications-controller, and integration with Grafana for creating custom dashboards.

Continuous Integration

We've talked about Continuous Deployment, which is responsible for delivering our application to Kubernetes. But what about Continuous Integration?

The approach to Continuous Integration (CI) can be varied, and there are several popular tools and methods for implementing it. Let's take a look at a few options you can use:

1. GitHub Actions. This tool has become popular in the development world due to its integration with GitHub. If you are working with GitHub repositories, GitHub Actions provides convenient tools for CI/CD customization. However, it is only integrated with GitHub and depends on that platform.
2. GitLab CI/CD. GitLab CI also provides powerful CI/CD customization tools and has popular functionality for local GitLab deployment. This gives you more control over your infrastructure, including Git repositories and Docker registry, and supports many tools.
3. Tekton. Tekton is a Kubernetes-native platform for creating and executing CI/CD pipelines. It is becoming a standard in the Cloud Native community and provides powerful tools to automate the CI/CD process in a Kubernetes environment.
4. Argo Workflows. Argo Workflows also provides tools for creating and managing workflows, but its integration with ArgoCD is limited. It can be used within the Argo ecosystem but also functions independently.
5. Local tools. For simple projects, it is possible to do without a CI system, especially in the early stages of development. There are tools for local development, such as the kubectl-build plugin. It allows you to build and publish Docker images directly in a Kubernetes cluster, facilitating development without leaving your local environment.

The choice of tool depends on your needs, project complexity, and preferences. It is important to choose a tool that best fits your specific tasks and integrates with your infrastructure.

Werf review

Werf presents an interesting approach to the concept of GitOps and CI/CD automation. This utility was designed with CI/CD principles in mind and can be easily integrated into various CI/CD systems, offering solutions to typical problems encountered when building such processes.

One of the key features of Werf is its use of determinism. It is similar to GitOps in organizing declarative and versioned infrastructure. However, unlike the pull model used by Flux and ArgoCD, Werf favors a push model for applying changes.

Werf covers aspects of CI/CD but does not fully implement GitOps by itself. Therefore, integrating with ArgoCD provides an interesting solution for building a complete CI/CD cycle, including local development, building images, and running tests. This integration allows you to use ArgoCD to deploy to the production loop and, optionally, to deploy to a production-like environment.

Using Werf and ArgoCD together can provide a more complete and automated infrastructure and application management, combining the benefits of GitOps and CI/CD.

Conclusion

GitOps is an important methodology that is revolutionizing the way Kubernetes applications are managed. By storing configuration and state in a Git repository, development teams can achieve greater automation and transparency in managing applications in Kubernetes clusters.

GitOps provides a more secure and efficient way to manage Kubernetes applications, foster better collaboration within the development team, and increase process transparency. Properly implemented, GitOps can significantly improve infrastructure and application management, making it an important tool in today's development and DevOps world.

My previous articles on Hackernoon are:

• Containerization In 2023: Strive for Maximum Modularity via the Cattle Model
• 8 Things Developers Forget in Application Migration To Kubernetes

I encourage you to read the articles above as they are no less valuable than this guide on GitOps.