Kubernetes - Secrets
This is a quick post about something I was asked about recently for a customer - Secrets within Kubernetes. The ask from the customer was:
Our engineer says that standard Kubernetes secrets aren’t secure, what are our options?
This is an overview of those options. I won’t be going into too much depth as there is no right or wrong answer here, it depends on the requirements of each individual organisation. It’s probably quite important to say that the engineer’s statement is both true and false (of course!).
What’s so wrong with secrets?
When you create a secret for Kubernetes it’s stored as a base64 encoded string, either in a YAML/JSON file or in the etcd store. Clearly base64 is not encryption and can easily be decoded. We’ll come back to the manifest later, let’s focus on etcd for now.
Secrets in etcd
So we know that secrets are stored in etcd in effectively plain text. There are two means of accessing the data in etcd - either direct to the datastore or via the kubernetes apiserver. What can we do to protect it?
Access to Infrastructure
As with any Operating System running in your datacentre/cloud, you should be controlling access to it in all the usual ways. Doing this will restrict access to the filesystem that etcd is stored on.
Also think about the other infrastructure components involved, are your backups secure? What about the underlying storage if it’s on premises?
Role Based Access Control
Role Based Access Control or RBAC is the primary method to protect the data within etcd when access via the api server. Structure the access control within your cluster so that only the necessary service accounts and users have got access to the secrets they require. This is something that’s fairly easy to implement but is often overlooked.
This method will also help protect access the datastore if etcd is installed within a kubernetes controlled container as your attacker won’t be able to access the container to get to the database. They’ll still be able to get to the database if they can get to the filesystem where etcd is store though, which brings us back to access to the infrastructure.
Encryption
Etcd can be encrypted while under rest if you’re concerned about the accessibility of your data. This is done by generating a configuration file and passing the --encryption-provider-config flag to the apiserver. If this is done without a Key Management Service then the encryption key is stored within this configuration file. This effectively only slightly improves the security of your secrets. The recommended solution is to use the KMS provider, which will take care of the encryption keys, adding another layer of protection.
Secrets in Manifests
So we’ve covered our options with the etcd security but what about storing the secret manifest? First question if you should store it? Does your existing secret storage have an automated method of pulling the secret out during your CI/CD pipeline?
If not, a good option here could be Bitnami’s Sealed Secrets. The option here enables you to store an encrypted version of the secret in with your deployment manifests. When this is send to the api server, the Sealed Secret controller will decrypt the secret and store it within etcd as a standard Kubernetes secret.
With that in mind, you still need to consider your etcd security as part of this solution.
Other Options
Your other option is some sort of api driven secrets storage. You can get a secret by meaned of an api call and inject it straight into the container when it starts. Thus completely avoiding storing the secret in etcd.
The main player in this space is Hashicorp’s Vault. If you haven’t got around to writing the api calls, they also provide a secret injector sidecar container. I’m not going into detail in this post, but Vault is a great option for secrets management.
Summary
So what should you do? As always, it depends on your requirements. Your best solution is likely to be not storing them in etcd at all with a solution like Vault. You should always protect your infrastructure by only providing minimal permissions, and that should be extended to Kubernetes. Sealed Secrets? Encrypting etcd? That’s a design decision to be made with the requirements in mind. Ultimately it’s like any security strategy, it’s the amalgamation of each layer that protects the infrastructure as a whole.