1. Introduction
GitOps is a term initially coined in 2017 by WeaveWorks to indicate the application of DevOps principles to infrastructure management and operations. This is achieved by employing git as a single source of truth, as well as actuated by means of declarative infrastructure management approaches. Accordingly, any difference between the repository and what is actually running can be detected and trigger a reconciliation loop, to automatically adjust the configuration to the desired setting. This ensures that the infrastructure is inherently reproducible from the state of the Git repository, as well as provides a unified method for both development and operations.
2. Kubernetes
Kubernetes is a popular container orchestration platform that, in spite of its complexity, (and contrarily to the Hadoop world) significantly automates and simplifies operations in distributed systems. GitOps has its intrinsic implementation on OCI-standard container orchestrators, especially K8s, given its declarative deployment approach that enables reproducible and trackable environment changes.
2.1 Declarative vs Imperative operations
In the DevOps organizational model, cross functional teams are able to undertake development, quality assurance and operational tasks. This has multiple beneficial effects, most of all the improvement of collaboration and information exchange, as well as a reduction of time to market, given the reduced number of interactions between what were previously separated teams or vertical departments (i.e "siloses"). This works by storing infrastructure configuration along with code on versioning tools, such as git, which consequently provide a single and trackable environment desired state. This is made possible by declarative operation paradigm, such as Infrastructure as a Code (IaC) tools, which define repeatable provisioning processes that can be retried and rollbacked at need. For this to work, however, team members are required to know basics of operations and have to spend a certain amount of their time preparing the IaC and deployment scripts for their developed software, under the slogan "you wrote it, you operate it". This can, one hand, lead to situations in which different solutions are undertaken across teams and the risk for compromising the infrastructure is high. In GitOps, developers do not have full control on the infrastructure, rather create pull and merge requests to inquire for feedback by other team members, perhaps from the operations team, who reviews and may ask changes until the request is accepted and merged into a superior branch. Until then, no side effects occur on the infrastructure. In this sense, declarative definitions provide one step forward automation, as contrarily to imperative approaches their result does not depend on the current environment state, but they rather define targeted configurations while formulation of actions is left to control loops.
2.2 Controllers
Declarative operations are possible thanks to the availability of operation controllers, i.e., control loops waiting and watching for specific kind of resource definitions and performing necessary steps to bring the system's state to that targeted in the definitions. Any standard K8s resource, such as Pods and Deployments are managed by default controllers, while the controller design pattern allows for the definition of additional application-specific controllers, i.e, namely a custom-resource definition (CRD) and a reconciliation control loop. We wrote about that in a previous post, have a look here.
2.3 CICD for Kubernetes
Plain K8s does not have means to manage builds and CICD pipelines.
Redhat openshift provides a BuildConfig resource that can be defined (see here) to trigger directly on a git repository, so that images are automatically rebuilt upon changes. The so called ImageTag can be delivered within a so called ImageStream.
A possibility is also to trigger the CI pipeline on the code repository and deploy the image to a docker registry or explicitly to the cluster.
3. Achieving GitOps
3.1 Multi-environment Management
After code is developed there are normally multiple target environments in which the code is being tested and validated: such as dev (also called test or QA), integration, pre-prod (or staging) and prod. This allows for changes to be carefully tested on increasingly realistic environments before they are brought to production and can have a real impact. In K8s a natural organization of environments is via namespaces, since they provide unique resource naming, pooling/quotas and access control. On git this structure can be achieved using multiple branches, one per each environment, along with inferior feature branch, depending on the adopted approach (e.g. gitflow). Depending on how tightly coupled applications in the repo are, one may also opt for a single repository approach over a multi-repository one, with the main drawback of the former being the release cycles (and their potential rollbacks) more interdependent.
3.2 Access Control Management
Although end to end application deployment to Kubernetes involves several systems, such as git, CICD, code quality tools, vulnerability scanners, container registry and the cluster itself, GitOps advocates for reducing at minimum the attack surface, by letting users interact with the least possible systems. Specifically:
- pull from external registries shall be disabled and vulnerability scans should be periodically run on the sole enabled internal one;
- only cluster admins should be accessing the git repository, as a push to a branch would mean potentially updating RBAC rules and affecting deployments; the repo should only affect a namespace and not the entire cluster;
- git shall not have direct access to the cluster, as anybody with access to git may potentially have also have access to the cluster; viceversa a GitOps operator may be set triggers on the repo to watch for updates; this is especially important in case the repo is shared across multiple teams and different roles and privileges on the cluster must be enforced;
- admission controllers, such as those based on the Open Policy Agent (OPA) shall be used to validate and restrict the creation of new resources (i.e. Pod) based on specific security policies;
- specific branches should be protected from a push (e.g. master) so that they can only be accessed via merge/pull requestes; commits should enforce for the inclusion of the committer identity by requiring the signing of commits with a cryptographic key;
3.3 Secret Management
Kubernetes secrets is a convenient approach to separate confidential information, e.g. passwords, from their actual use, mainly by specifying a secret resource that can be then used by the client Pod mounted as a file, as environment variable or kube-api access token; There are, however, multiple drawbacks to the use of K8s secrets, mainly given the absence of encryption and the lack of access control in spite of their separation from their user process. External secret and key management systems (KMS) aim at copying with those issues. Rather than defining the secrets as K8s resource, the idea is to dynamically query the KMS for necessary secrets. A very common KMS is Hashicorp Vault. Although the Vault has its own CLI and API to retrieve tokens, due to its popularity a sidecar injector is available. This standardizes the retrieval of secrets, namely, by letting developers annotating pods with secret references. The Injector automatically injects those values by mounting it on the Pod. The mechanism also reduces the Vault's attack surface, as the injector is the only one interacting with the KMS.
4. Tools
We report in this section a couple of tools to implement GitOps practices.
4.1 Argo CD
Argo CD is an open source GitOps operator for K8s.
4.2 Flux
Resources
- https://www.weave.works/technologies/gitops/
- https://about.gitlab.com/topics/gitops/
Bibliography
- B. Yuen, A. Matyushentsev, T. Ekenstam, J. Suen. GitOps and Kubernetes. Mannings 2021.
No comments:
Post a Comment