Hi. We successfully store secrets in ansible variables files with either ansible-vault or sops. It is a good approach when Ansible itself configures something that requires a secret, such as configuring a database admin password.
But I’d like to ask you about how you store secrets meant to be used by applications. Example: we have a an application in PHP with a config.php file with all credentials needed by the application. Developers have a config.php setup to work with the test environment, while we maintain a different config.php for production in production machines. Nowadays this config.php file is stored in ansible repository, encrypted by ansible-vault or sops. We thought about moving the config.php production file to the application repository, so we could get advantage of the CI/CD pipeline.
It doesn’t smell right, because it would require to encrypt it somehow, and store keys to decrypt it in CI/CD, but I decided to ask you anyway what do you think of that and how you solved it yourselves.
Thanks!
Depends where you are deploying. If you’re deploying to AWS or Azure you can use their secret manager and replace the secret text in the config.php file with the path to the secret, then write a simple utility file to read the secret. If you are deploying to a Windows Server you can store an encrypted value in the registry and write a class to decrypt/read from the registry.
Thanks for the answer. I’m actually deploying to a local Linux server. Is there any tool I could use?
We use Hashicorp Vault to do this. A local Agent on the box can get the secret for you on your behalf and either you can use that in your app launch process, or have the app call the agent or the vault server itself.
That is exactly what I’m looking now! Thanks! I’m actually new to this paradigm. How is it more secure than storing password in plain text? I mean, in your application, you’ll have to store a token or something, so you can query Vault server, right? If an attacker have access to the server, it can query the Vault server with the token, retrieving relevant secrets, no?
One of our apps have vault write it to a file, but rotate it regularly (like every hour or so) and have their app watch the file for updates.
Other apps query vault directly and thus it never gets written to disk. As our apps are all containerized, the password for each instance is unique and only that ephemeral container can use that password.
In fact, when the vault agent requests a password for something like a database, the agent doesn’t just share what’s in vault, but uses what’s in vault to generate a password and username that is unique to only that container and expires previous ones on a rotation.
In truth, if you had low level access to a compromised box, you could probably still sniff it out of ram or off disk, but the rotation means it’s only good for a small amount of time, and the fact that only that machine and username can use it, combined with the Least Privilege, means it’s fairly secure.
You can probably write a utility in your PHP that just pulls the value from Ansible Vault and decrypts at runtime. Still not 100% secure but better than having the value in plaintext in your config.php.
Thanks. I would still have to store ansible-vault password locally or something like that, no?
Yeah, or just continue to have two separate config files.
My team is in a bit of a different situation as we are cloud hosted but I suspect it’s a similar approach. We have secrets hosted in AWS and our config file just references the secret name. We then have a SecretReader utility in our code which we reference to pull the secret from AWS at runtime, that way it doesn’t show up as plaintext in our code. Our CI/CD doesn’t touch the secret at all in that use case – we only use secrets in our CI/CD if there is a step such as connecting to a DB that is part of our pipeline.