DevSecOps for K8s: integrating security in modern CICD pipelines

We already discussed in previous posts of the benefits of employing continuous integration and deployment (CICD) pipelines to increase the frequency of testing and validation as well as deployment of new features in modern IT infrastructures. To this end, a crucial role is played by code repositories as one single source of thruth, as well as git as main versioning and interaction tool. This has eventually led WeaveWorks to the naming GitOps in 2017. See also here. 

The main concept can be summarized in the picture below:

 

The idea is to have developers push modifications to a code repository.  I leave the reader to investigate further how developers should interact with a code repo. I tend to use gitflow as a way to structure the code repository. The idea there is to use feature branches, whose size is more or less related to a feature or a user story (shall you use scrum for instance). Feature branches are then merged to upwards branches, generally development and master but more commonly a set of branches very specific for the target environments available in your infrastructure, such as development, test and production, or something like development, integration, pre-prod, prod. Frankly, the naming varies a lot as IT depts tend to be very creative.

Once modifications are pushed, triggers are set on specific branches. So on an integration branch you may want to trigger integration tests, perform some code quality checks and finally deploy the code to an test/integration cluster for additional checks. If all is fine, the user can merge on superior branches and start similar processes. Eventually, an actual release can then be triggered, for instance on tags or master.

The main effect is then to push the artifact and its wrapping container image to a registry. Being the registry available to the deployment, modifications can trigger rollout of new versions.

You can use the following tools within your CICD pipelines to increase the security of deployed artifacts on K8s:

• trivy - a dockerized vulnerability scanner; you may run this as a step in your code CICD to spot known vulnerabilities on imported images.
  
• kubesec - you can set up a CD pipeline on your infrastructure config repo to directly change the infrastructural setup. Kubesec can be applied as a predefined set of rules that can spot security flaws in your K8s yaml configurations; think it of a list of best practices;
  
• kube-score - similarly to kubesec, it does apply common rules to spot flaws in yaml definitions;

A more complete list of static analysis tools for docker images is also discussed in this post.

After those steps are passed, we got the artifact tested and its wrapping image validated and stored on a registry. In fact, very often the build of the image takes also place on the target cluster. For instance on openshift you may use a build config triggered directly by code changes and push to an internal registry. In this case you may apply container validation outside and, for instance, only allow for the merge to a superior branch to take place and the consequent trigger of the build config to start.

Assuming this is fixed there are certain things you can apply to still prevent ill-defined configurations to get in the cluster. The open policy agent (OPA) is a policy engine that can be used for enforcing policies by directly acting on the admission controller, i.e., the component in charge for the validation of k8s resources before they are created. The same rego language powering opa policies can be also used with OPA conftest, a tool allowing for static policy enforcement, for instance on Dockerfiles and k8s configurations. Thus it provides another static analysis tool that can be easily integrated in a CICD pipeline.