Secrets management in Kubernetes - an overview

Talking about Secrets Management in Kubernetes is about how to handle sensitive data (also called "secrets"), such as passwords or private keys in a Kubernetes environment.

In Kubernetes API, there is already a resource called "Secret", which is intended to store sensitive data in it. can be stored in it. The Kubernetes Secrets are always stored base64 encoded in the cluster. You might therefore think that they could not be "secure". Base64 can be easily encoded and encoded. However, it is important to note that here we mean "encoding" and not "encryption". The actual reason for using Base64 according to the Kubernetes developers, is not a security issue, but the fact that it is much easier and more reliable to unknown characters, line breaks or even binary data much more easily and reliably.

The Kubernetes secrets are stored in the Etcd key-value store.
Access is secured via Role-Based Access Control (RBAC).
It is therefore also okay or safe enough to use simple Kubernetes secrets.

Why should you worry about Kubernetes Secrets "management"?

It is recommended to create and manage the Kubernetes resources in a declarative way. A well-known approach for this is the GitOps concept. In this case, sensitive data such as passwords could also be stored in text or yaml files. files. However, you don't want to store sensitive data in plain text locally or in a Git repository. But how can the Secrets still be created or managed in Kubernetes?

There are already some solutions and experiences to solve the problem:

Sops

https://github.com/getsops/sops

The "Sops" tool can be used to encrypt the corresponding files that contain sensitive data so that they can then be encrypted in a Git repository, for example. If you work according to the GitOps concept with the help of ArgoCD, there is an ArgoCD Vault plugin for this purpose.

The complete file is encoded in a Helm chart values.yaml with Sops and pushed to the corresponding Git repository pushed. ArgoCD can then use the Sops plugin to decrypt this encrypted values.yaml again and thus roll it out in roll out to Kubernets.

Working with Sops is very straightforward. It is particularly suitable for small projects, as it can also be can be implemented quickly.

However, it also has some disadvantages. Here are our own experiences with Sops:

• Every developer must install and configure Sops in their local development environment (e.g. store Sops keys). store).
• When encrypting and decrypting with sops, formatting (e.g. indentations or quotation marks) and comments are often destroyed.
• If errors are generated in the Yaml structure, e.g. due to incorrect indentation, Sops and therefore also the editor were "frozen" for several minutes when encrypting the file.
• With automated CI/CD pipelines, everything became much more complex with SOPS, as you need the Sops binaries here, to be able to decrypt and encrypt automatically.

Bitnami Sealed Secrets

https://github.com/bitnami-labs/sealed-secrets

Sealed Secrets allows you to store encrypted secrets in a Git repo. As soon as you use kubectl to create a SealedSecret in a Kubernetes cluster using kubectl, it is automatically decrypted by a Sealed Secrets controller and converted into a native Kubernetes secret. The SealedSecret can only be decrypted by the controller that is running running in the target cluster, and no one else (not even the original author) can obtain the original secret from the SealedSecret. from the SealedSecret.

Things that we noticed negatively here:

• Every developer who wants to create secrets must set up appropriate tools in their local development environment. in their local development environment.
• As soon as you want to automatically generate secrets, e.g. through a CI/CD pipeline or an external application, using use with Sealed Secrets becomes very complicated.

Sealed Secrets, like Sops, is well suited for smaller projects and quick successes. It is uncomplicated and quickly set up.

In our opinion, Sealed Secrets performs slightly better than Sops, as it is less prone to errors. Furthermore, you only encrypt the pure secrets here and can, for example, leave the other files in a Helm chart as they are. as they are.

External Secrets Operator

https://external-secrets.io

External Secrets Operator is a Kubernetes operator that integrates external secret management systems such as AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, CyberArk Conjur and many more integrated. The operator reads information from external APIs and automatically inserts the values into a Kubernetes secret secret.

This solution therefore requires an existing secrets management system and is therefore somewhat more complex. However, you are However, the use of External Secrets Operator makes you very flexible, as you are not tied to a specific secrets management system. system. For example, if you are using the AWS cloud infrastructure anyway, it makes sense to also use their secret manager. You could also host your own HashiCorp Vault and connect the external Secrets Operator here too.

The fact that real Kubernetes Native Secrets are generated at the end also makes it easy if, for example, you use third-party Helm charts or tools such as Cert-Manager, Kube-Prometheus-Stack or Ingress-Nginx and you only have to refer to these existing secrets.

Hashicorp Vault

https://www.vaultproject.io

The "Vault" tool comes from the company Hashicorp and is initially "only" a Kubernetes-independent system for managing of secrets. Therefore, you need another tool in addition to Vault itself to distribute the secrets in Kubernetes:

Vault sidecar agent injector

https://developer.hashicorp.com/vault/docs/platform/k8s/injector This injector uses the sidecar concept to modify pod specifications to include a Vault Agent container that mounts Vault secrets on a shared storage volume. It can be deployed together with the Vault Helm Chart.

It is probably one of the safest ways to handle secrets in Kubernetes, as they do not even appear as native Kubernetes Secrets, but are injected directly into the pod. However, to use the sidecar agent you have to change the pod manifests. Annotations must be stored here. If you then also want to use "third-party applications that want to have the secrets available as environment variables, this is only possible via via workarounds, which makes the whole thing very complicated.

Vault CSI provider

https://developer.hashicorp.com/vault/docs/platform/k8s/csi The Vault CSI Provider is a "store driver" that, at a higher level, allows pods to mount and access a CSI Secrets Store volume and access it. The volume then contains the corresponding secrets. Here, too, you have to, similar to the sidecar agent injector, the pod manifests must be changed for use.

Vault secrets operator

https://developer.hashicorp.com/vault/docs/platform/k8s/vso The Vault Secrets Operator is currently the latest method of integrating Vault Secrets into Kubernetes. It functions similar to the external secrets operator and is also deployed in Kubernetes. It can read the secrets from Vault and generate and synchronize them as native Kubernetes secrets. It supports the synchronization of the entire lifecycle of the secret management of one or more Vault instances. The operator supports secret rotations for deployments, ReplicaSets or StatefulSets. Using the Vault Secrets Operator has the advantage of being very flexible with the help of the real native Kubernetes Secrets. is very flexible. This also allows easy integration into third-party Helm charts.

Argocd vault plugin

https://argocd-vault-plugin.readthedocs.io/ This plugin is installed in the GitOps tool ArgoCD. The plugin makes it possible to replace passwords in the manifests or Helm charts with placeholders. The placeholder then corresponds to the path to the password in the Vault instance. It is therefore very easy to use.

Which secret management solution you should use always depends on your own factors. Which public cloud do do I use? What data protection regulations do I have? Do I also need to access my secret management from external or CI/CD pipelines? access? How large / complex is my project, is it worth the effort of a complex solution? Do I work according to the GitOps concept concept?

For us, the combination of Hashicorp Vault and the Vault Secrets Operator has prevailed.

The following reasons speak for this:

Vault can be rented on the Hashicorp cloud platform or hosted and operated by yourself. There are already ready-made Helm charts from the developers themselves. Because you can set up Vault in any cloud, you are not you are not dependent on the respective cloud's own secret management services. You can therefore rely on the Vault secret management solution and not have to familiarize yourself with various other cloud solutions first.

German public clouds do not always offer ready-made secrets management services, which would be the case with AWS or Azure, for example. would be. We therefore often have to operate our own secrets management software.
Hosting Vault ourselves is necessary for us so that we can comply with the legal data protection regulations in Germany. in Germany. We also sometimes need to manage secrets from external tools or CI/CD pipelines, so we need secret management with an API.

Because "Vault" has already been defined as a tool by the points mentioned above, the only remaining question was what other tool should be used to get the secrets into the Kubernetes cluster? The Sidecar Agent Injector and the Csi provider were out of the question for us, as it was very complicated, or impossible to integrate the respective specifications into the pods in ready-made third-party Helm charts. The ArgoCD Vault plugin would have been suitable for us, as we mainly use ArgoCD, but we wanted to keep however, we wanted to keep the option open to use other GitOps operators instead of ArgoCD, such as Flux or the Gitlab Kubernetes Agent. This left us with the Vault Secrets Operator or the "External Secret Operator". However, the Vault Secrets Operator comes from Hashicorp itself. This seems to guarantee the greatest possible compatibility with Vault seems to be guaranteed.

In the next blog post from the DevOps category, we will show you how to deploy and use Vault and the Vault Secrets Operator in a Kubernetes cluster yourself.