The Less Understood Dangers of Secret Sprawl and What Devs Can Do to Fight It

When developers refer to secret sprawl they are typically referring to the unwanted distribution of secrets across multiple platforms, services and machines.

Once a secret ‘sprawls’ into other systems it can often have a follow on effect allowing attackers to use secrets to move laterally between services and uncover additional secrets.

To really understand how secrets sprawl, we need first to understand two key concepts, what is a secret and how we use secrets.

What is a Secret?

As the name suggests, a secret is really any data that is sensitive but when discussing secrets in the context of software development, developers are generally referring to anything that grants access to external services or data. These are most commonly API keys, credentials and security certificates.

What is a Software?

Software used to include everything needed to run internally, today as the world is much more reliant on the internet, this has allowed software architecture to fundamentally change with the introduction of new services such as Cloud Architecture, SaaS Platforms and Microservices.

These services allow a lot of development work not related to the core of an application to be offloaded, this reduces the upfront development costs while simultaneously making applications more robust and scalable.

As beneficial as this is, it does introduce a new challenge to overcome, how to establish a trusted and secure connection with each of these services. This is generally done through an exchange of secrets, namely API keys, security certificates and credentials.

The Challenge

Depending on the size and objective of an application, a project might need to connect to tens or even hundreds of services which all need individual secrets.

Developers not only need to store these secrets safely, they also need to be able to distribute and use secrets during their development process. Adding an even greater level of complication, secrets will often get rotated and revoked over time, this means that the distribution of secrets is a challenge that will last throughout the entire software development lifecycle.

How secrets sprawl across the internet

Secrets management requires a thoughtful understanding of what permissions to give secrets, who needs access to them, how to keep them in sync across multiple teams (often in different geographies), and what restrictions, tools and guidelines need to be in place when accessing and using them.

Strict secret management creates added procedures that are both difficult to implement and tempting to circumvent. This is why developers and organizations alike often store secrets in unsecure locations, usually unintentionally. Secrets can be hardcoded into source code and included in a git repository which can get cloned onto multiple machines (professional and personal), get sent via Slack or emailed for convenience, saved to an internal Wiki and uploaded into a google drive…. So on and so forth.

Secrets sprawl increases the 'attackable area'

Even if secrets don’t end up on public internet space (for example on a public git repository) they should still be considered compromised if they are sprawled. Having secrets on multiple services, email, Slack, git etc increases what is referred to as the 'attackable area’.

The attackable area refers to the amount of systems that could be exploited to find secrets. In a situation where an organization has secrets sprawled over multiple locations, it only takes one compromised developer's git account, one compromised email or one compromised computer for an attacker to suddenly gain access to a trove of highly sensitive secrets.

Secrets can be used to travel laterally between systems too, for example a secret allowing an attacker access to Slack messages might lead him to discover secrets with access to a cloud drive which might uncover secrets to a database….. So on and so forth. Secrets should remain centralized and encrypted.

How do you prevent secret sprawl?

Secrets management as stated previously, is difficult. But there are great tools available that can be implemented to help tackle the issue of secret sprawl.

Secrets Scanning

To combat secret sprawl it is essential to have visibility inside the systems where secrets may be located. .git repositories can contain a trove of secrets buried in the history including inside old versions of source code, application logs and config files. It is important to consider also that even the best secrets management systems and policies do not prevent newly generated secrets entering the code base or old secrets being extracted and included again, therefore all organizations should implement secrets scanning into their workflow.

GitGuardian offers a free secrets scanning tool for .git repositories which scans, in real time, every commit you make so you can immediately identify if your secrets have sprawled. GitGuardian also has an API so all office systems like slack or email can be scanned for secrets too.

Encrypting Secrets

Git repositories offer unmatched collaboration features for developers, git not only acts as a complete historic record of a project but also offers a single point of truth for the latest version and files, hence why it is so common for secrets to be stored within them. The good news is that there are ways to store secrets securely within git repositories.

Git-secret is a free open source tool that encrypts secrets within git repositories making them safe to distribute through git.

While encrypting secrets and storing them within git does provide the benefit of preventing secrets sprawling through the git, it does not prevent secret sprawl on other services and developers will still need to manage the secrets to decrypt the secret file (Secret sprawl can be like inception, secrets for files that contain secrets inside, which give access to services with secrets inside them).

Using secrets management solutions

One of the most popular secrets management tools on the market, Hashicorp Vault, offers both open-source and enterprise solutions to developers and organizations and provides the ability to tightly restrict and control access to secrets, enabling the easy rotation of secrets while also giving developers the ability to easily connect to external services.

Hashicorp vault can be difficult to roll out and implement and might not be appropriate for all types of secrets.

Conclusion

No organization big or small is immune from secret sprawl and the best policies and tools still won’t stop every possibility of secret sprawl. This is why to combat secret sprawl you need to combine a strategy to store secrets, manage secrets (rotate and distribute) with a strategy to gain visibility into your services and systems.

(Disclaimer: The Author is associated with the GitGuardian)