Tuesday, 4 February 2014

Encrypted Chef Node Data

When dealing with Chef, you frequently want to store data in an encrypted form so that your Chef Server doesn't become a single point of vulnerability for the security of your entire infrastructure. For this purpose, Chef has the concept of an Encrypted Data Bag.

Also when dealing with Chef, you frequently want to generate certain bits of data and remember them for future runs. When dealing with a Chef Server, you also sometimes want to let other nodes grab this data for their own configuration. For this, you have standard, plain text Node data.

So what happens when you want to generate something that needs to be kept secure, such as the passwords for your database users? You could put them in an Encrypted Data Bag, but then you need to generate this data in advance. You could put them in the node data, but then your passwords are stored in plaintext on the Chef Server.

Neither of these solutions are ideal. What we want is to be able to generate the data on the node and store it in an encrypted form. Unfortunately, Data Bags can't be updated during a run (and this does lead to some race conditions, as what would happen if several nodes tried to update the same data bag at the same time? Distributed race conditions are never fun).

How about encrypting the data in the node though? This seems like a good approach, as the node data is safe(ish) from distributed race conditions, but it would be fairly tedious to have to build our own encryption mechanism for nodes, so maybe we can re-use some of the encryption used for Encrypted Data Bags? This is surprisingly easy to do, as the data bag encryption is well factored into code for Secrets, Encryptors and Decryptors separately from the main Encrypted Data Bag code that binds them together (although the namespace doesn't lead to the clearest code at this point).

Without belabouring the point too much, by placing something like this:

into libraries/node_encryption.rb in a cookbook, you can then make use of this in your recipe in the following way:

This does still rely on your secret being shared between your servers, in the same way as Encrypted Data Bags, so it isn't perfect. Our own infrastructure makes use of multiple sets of secrets in order to mitigate this on an environment basis (the secrets for our development servers are separate from the ones for our production environment). What would be really fun would be if this could be integrated into a tool like Chef-Vault (see also this blog post on chef-vault),  so you could generate a value on a node and specify a search to specify which nodes are allowed to decrypt the value. This would extend the usefulness of Chef-Vault a bit, as you could then rely on your cookbooks to handle some of the re-encrypting that currently needs to be done manually with Chef-Vault (such as re-encrypting a value when a new node is added to grant access with the new node's public/private keypair).