HashiCorp Vault Secrets Management: 18 Biggest Pros and Cons
HashiCorp Vault is a phenomenal tool to store and dynamically create secrets, provide encryption for your applications on the fly and even manage certificates. There aren’t really any other tools quite as powerful and versatile.
However, it is not a perfect tool (is there such a thing!?).
In this blog, we will outline some of the benefits of using HashiCorp Vault and some of the drawbacks so you can decide whether it is the right tool for your use case.
But first of all, let’s start with a brief introduction to what HashiCorp Vault is and why you should be interested.
What Is HashiCorp Vault?
Vault is a tool that enables you to keep your secrets secured and encrypted. It can also do other things like encrypt data for your applications (so they don’t need a special library to do so) and generate temporary secrets and certificates for users and services.
All requests to Vault are done via an API and need a token—this token can be generated in multiple ways via different authentication methods such as OIDC, LDAP, username and password, etc.
Vault is also extensible, with many different secret stores and authentication mechanisms added regularly for various cases.
Overall, Vault is possibly the most comprehensive tool for secret management available today and it can be a powerful asset to an organisation that has complex and dynamic requirements for secret management.
Pros of HashiCorp Vault
1. It provides an easy way to managing secret sprawl
Managing secrets and credentials can be a very big headache. If you don’t have a simple tool to do it for your developers to use, you are likely to find them in plaintext in everyone’s computers or worse—in the repository.
Vault allows an easy way to encrypt, generate and store credentials for all your workloads across your organisation and it is well supported by most frameworks these days.
2. It allows dynamic secrets
Vault allows dynamically generated users and passwords for databases or other tools like SSH. This is a feature that’s enormously beneficial to contain the sprawl of users and passwords around your organisation and to keep track of who, when and what a user is doing.
Most importantly, it also protects you from leaked credentials causing damage to your organisation as the generated secrets can be configured to last however long you define.
3. It’s open source
Vault is open source, meaning that security experts worldwide can audit the code and contribute to it. It is also written in Go, which is a great language for scalable applications.
4. It is self-hosted
Recently HashiCorp released Vault as a service in the HashiCorp cloud platform but the main route to use Vault for companies is self-hosting it in their infrastructure.
Because Vault is open source and you can manage it completely, it also means you have complete control and ownership of your secrets—something that may appeal to banks and companies with stringent security requirements.
Also because Vault binary is so small, you can also easily run it on a laptop or anywhere you want.
5. It can be used to encrypt data of your applications
Vault can be used to do cryptography as a service for you. Meaning that applications don’t need to worry about encrypting data—they just use Vault to encrypt it for them. Vault can also sign and verify this data, but it doesn’t store it for you. It is just used to encrypt and decrypt it.
6. It can generate your PKI certificates
Vault can also generate your certificates for you. This is particularly handy if you have a need to automatically generate short-lived certificates for your organisation—an application could fetch them from Vault on demand.
7. Vault’s functionality can be extended with Secret Engines and Auth Engines
Vault supports many secret engines that either encrypt, store encrypted data or generate dynamic secrets for you as well as Authentication Engines that allow different ways to log into Vault. Because these are extensible engines, the release cycle for new methods is fast and you are very likely to find what you need.
8. It can be configured to be highly available
You can configure Vault to be highly available—meaning that if one of your nodes goes down another one will take its place. This high availability can also be extended to your data layer with something like Consul.
9. There are many ways of interfacing with it
Whether you like to use the API, CLI or the UI, Vault has you covered. In that sense if you have a user that’s not very tech savvy, he or she could still use the UI to add or remove secrets. Other more tech savvy users can use the CLI, the API, Terraform or even a programming library.
10. It has a fast release cycle
Just like many other HashiCorp products, Vault is released very frequently which means any vulnerabilities are unlikely to last long and new features are introduced constantly
Data Maturity in the Public Cloud: Research Report 2021
We asked 272 IT decision-makers from around the globe about the state of data maturity in the public cloud in their organisation.
Get the report for useful insights and the benchmark metrics you need to help drive a successful cloud data implementation.
Cons of HashiCorp Vault
1. Vault is powerful but complex
Vault is deceptively simple. When you look at the single binary and the things you can do with it right away you may be tempted to think it is ‘plug and play’ in your infrastructure and you are done.
However, when you start to take in consideration everything you need to do, you will see that it is not a task you can complete in a couple of days, even with all the Helm Charts and Terraform scripts available online, there is still a lot to be done:
- Backups
- Disaster recovery, completely automated recovery from backup.
- Key management, unsealing vault will either require something like KMS or key scrowing
- Secret management for Vault. This is the chicken and egg problem, Vault manages your secrets, but who manages Vault’s secrets?
- User permissions, this will probably have to be done with something like OAUTH or LDAP, either way setting this up requires thought and design.
- HA architecture design. This could be done with either Kubernetes or with MIGs, either way it is complex and you need to take care of your data layer.
- Permissions for groups, users, etc.
- Permission trees in KV and other stores.
- Automated creation of secrets based on paths with terraform or something like that.
- Etc.
The above is not an exhaustive list and you will likely encounter other issues as you implement Vault in your organisation, depending on how large and complex it is. So it is good to bear in mind that even though Vault is as simple as possible, it requires a lot of work and strategy to implement it.
HashiCorp now offers Vault as a service as part of their cloud offering. However, bear in mind that with this, you’ll lose control of where your secrets are stored, and you will also need to grant access from an external resource to your cloud infrastructure—but it may be a good option for some people.
Another thing to bear in mind is that if you use a cloud provider, you may already have various tools that take care of your secrets well enough not to need Vault. For example Google Cloud Platform (GCP) has a tool called Secrets Manager that already allows you to store your secrets securely within the GCP platform and it is fully managed and integrated with your GCP environment, which means you can get started a lot quicker.
One huge advantage of Vault over current cloud offerings for secret management is that it can create dynamic secrets for you, but even some of those needs could be circumvented with cloud offerings already existing. For example in AWS you can use IAM for RDS and GCP has a spectacular, out of the box way to manage SSH keys for your servers on the fly called OS login, so you probably wouldn’t have much use to generate them in Vault if that’s your cloud provider.
2. Everything is a path in Vault (but the path is not always so intuitive)
In Vault, everything is a path, and your policies can and will grant access to anything based on a path.
In practice, the above sounds amazing but I personally find the path structure counter intuitive and a little confusing.
For example, you may be misled into thinking you can list all paths—that to me would have been useful to find configuration endpoints—but this is not always possible, even with a root token:
curl --header “X-Vault-Token: $(cat ~/.vault-token)” --request LIST “${VAULT_AH_ADD}/v1/sys/”
will not list all the endpoints under the system backend, which would have been useful in some cases.
Let’s say I wanted to do something simple and list all the auth methods I have set up; let’s try and do it in a way that would make sense to me.
So auth is under sys, maybe I can do this:
curl -s --header "X-Vault-Token: $(cat ~/.vault-token)" --request LIST "${VAULT_AH_ADD}/v1/sys/auth/" | jq
{
"errors": [
"1 error occurred:\n\t* unsupported path\n\n"
]
}
Maybe I try this instead?
curl -s --header "X-Vault-Token: $(cat ~/.vault-token)" --request LIST "${VAULT_AH_ADD}/v1/auth/" | jq
{
"errors": [
"no handler for route 'auth/'"
]
}
Nope, that doesn’t work either. So I go to the documentation where I find that this is the right way to list auth backends:
curl -s --header "X-Vault-Token: $(cat ~/.vault-token)" --request GET "${VAULT_AH_ADD}/v1/sys/auth" | jq
{
"token/": {
"accessor": "auth_token_blah",
"config": {
"default_lease_ttl": 0,
"force_no_cache": false,
"max_lease_ttl": 0,
"token_type": "default-service"
},
So in order to do a listing here, we do a GET method instead of a LIST method. That for me would be okay if it was consistent, but then in other places like in kv you use the LIST method to list secrets. So yes, everything is a path in Vault, which is cool, but not an intuitive path structure.
3. The key value version 2 store is useful but a little confusing
There is another thing about paths that everyone I have worked with gets confused about: the kv2 path structure.
Kv2 allows you to have multiple versions of a secret, which is handy if you accidentally delete one. However the way it was implemented makes it confusing to use at first.
For example, if you have a path kv/database/ where all your db secrets reside and you create a policy like this:
path "kv/database/*"
{
capabilities = ["create", "read", "update", "delete", "list"]
}
The above will grant access to nothing because the path kv/database/ does not exist. Confused? So was I, and everyone else that I have witnessed using Vault with me.
The reason for this is that kv uses multiple “hidden” paths that perform different functions such as delete, update, metadata(to list), etc
So in order to gain the correct permissions to the above path your policy needs to look like this:
path "kv/+/database/*"
{
capabilities = ["create", "read", "update", "delete", "list"]
}
The plus is essentially a wildcard matching a single directory, so the above is giving access to all secrets under the database that reside under any parent directory.
These details are abstracted away for you when you use the UI or the CLI but not the API or any other programming library, so you need to make sure to communicate them to anyone who you are giving permissions to in Vault.
4. Inspecting audit logs can be tedious
One of the greatest things about Vault is that you can see who requested what secret, when and from where. This is particularly handy if you use Vault to generate dynamic secrets because if anyone were to steal those generated credentials, you could find out straight away what token generated them, from what user or service, and take action accordingly.
Vault sends all sensitive information hashed to the logs, which is a great thing, but sometimes it can be a little heavy handed with hashing and there is no simple way (that I know of) to selectively disable hashing for certain values.
For example, when using dynamic secrets for DBs, Vault will dynamically generate a username and password and both are hashed in the logs. It would have been nice if only the password was hashed, but that’s not the case, so in order to search for your generated DB user v-oidc-fer-test-A1blahblah, you need to hash that value and then search for it hashed.
The only way I know of to retrieve that value is by using an API endpoint like this:
curl -s --header "X-Vault-Token: $(cat ~/.vault-token)" --request POST --data \{\"input\":\"v-oidc-fer-test-A1blahblah\"\} "${VAULT_AH_ADD}/v1/sys/audit-hash/file" | jq .hash
"hmac-sha256:d3626e5969..."
Which even if scripted is a little tedious to do every time. It would be a little nicer if at least this function was available in the UI, but it isn’t because…
5. The User Interface is missing some useful features
Some developers and sysadmins look at UIs with disdain because they argue that everything should be done in code and the command line is more powerful. While I do agree with both of those arguments, I still think having a good UI is very useful for two main reasons:
- The UI is useful when you are delegating access to someone who is not familiar with Vault, and overall is just so much more convenient to use when updating your KV secrets, etc. Usually when updating Key Value type of secrets in Vault, the UI is very good.
- It is very useful when it comes to viewing existing and available configuration or to test something you will later implement in code. Here is where I find the Vault UI a bit underwhelming in terms of the configuration you can view and apply.
If you want to use all the features and configuration that Vault has available, the order is the following:
API → CLI → UI CLI → UI
Anything you want to do in Vault will be available via the API, followed by the CLI with fewer features, the UI CLI (A drop down menu in the UI that gives you access to a more limited CLI) and lastly the UI.
Many administrative tasks fall outside of the scope of the Vault UI and you won’t be able to do them there. This is totally fine with me as I prefer to do all the configuration in Terraform—however I do find UIs very useful to orientate myself, especially to find existing configuration options and see if they are correct and what else I can add, etc. It’s also quicker than going through the API documentation to find how I can fetch these config values.
For example, you cannot view what roles you have configured with AWS auth method in the UI, and it is the same with many other auth methods with a few exceptions. Instead you need to do that via the CLI, or API. You also cannot view the roles configured with DB dynamic secrets, etc.
6. Be mindful of upgrades
I love that Vault has fast release cycles and they are continuously improving it but sadly I have encountered a number of issues with upgrades—even with minor releases.
Some of these bugs, feel to me, like they could have easily been avoided with better automated testing, for example when they released version 1.5.1 they forgot to pack the UI with Vault…
Or when, in the next release 1.5.2, they patched a security issue with AWS auth method but they forgot to include allowed accepted headers for it...
I don’t want to be too judgmental with Hashicorp here because I really appreciate the great job they are doing overall. However, it’s important as a user to be mindful of this when doing your upgrades and perhaps apply them to a lower tier vault and wait for at least a couple of days of real life usage before applying them to production.
Also, always check their upgrade guides before upgrading, and bear in mind that if you don’t wait a few days before upgrading even to a minor version you may be the first one reporting something that will end up going on this guide.
7. You can’t search secret keys
If you have your secrets organised in a hierarchy and you do not remember where something is, tough luck. There isn’t an out of the box way to search for secrets in Vault. You can create a script for it, sure, but it will be very slow...as you have to traverse all “directories” and do multiple lists in each of them. See this for reference.
8. Miscellaneous issues
I have come across a handful of other issues or ‘bugs’ as I’ve used Vault. These seem to affect mostly edge cases or functionality that perhaps is not as widely used by everyone. This can give the impression that you are using a product that is still in beta.
For example, this issue I had with path permissioning with multiple plus signs, or this other issue with documented functionality not working as expected, etc. Neither one of these were really deal breakers, but they’re not ideal, especially when you spend a lot of time trying to figure out what you have done wrong, when in reality it wasn’t your fault.
So Should You Use HashiCorp Vault?
Vault is a great tool and it really has little competition in doing what it does better. If your organisation requires a lot of the features it provides and if there aren’t any simple cloud offerings that meet them, then Vault is a great tool and wholeheartedly recommend it.
However if you want to keep things simple and you have no use for dynamic secrets, encryption or generating certificates on the fly, using your cloud offerings may be all you need.
