Whack-a-mole is dead

I am convinced that security has turned an important corner. Welcome to the post alerts and issues era, where whack-a-mole is dead, and teams are now embracing security insecticides. 

For decades everyone has complained about noisy tools, both monitoring and assessment. The approach to this was vulnerability management and SIEM tools, that for the most part focused on correlating and deduping alerts and issues. It simply hasn’t worked. People are drowning in issues and alert fatigue. 

A slightly updated version of those tools in the appsec space is ASPM, which again for the most part, and despite marketing claims to the contrary, still do largely the same; correlation and deduplication. Word on the street is there is a real demand for the promise of ASPM because the mess they are trying to solve is very real, but word is also they fall in the same trap and don’t solve the problem either. 

Given the surface area of what we have to secure, the number of tools that companies need to use to do it, and the fact that each vendor intentionally wants to generate alerts and issues so that users ‘see value’ in their technology (no one renews a tool that doesn't show value), the fundamental problem of noise and alert fatigue has been getting exponentially worse to the point it has now hit the tipping point. The current approach is a case of reducing noise, but only from eardrum bursting pandemonium, to a constant deafening yell.

When people get to this improved state of constant yelling, and not everyone does, then they start playing whack-a-mole with the alerts and issues they are left with and think matter. The reality is whack-a-mole is a game you can never win. It’s rigged. No matter how fast you are able to whack them, the moles always keep pace with your effort. Even putting out fires is better than whack-a mole. You only have to extinguish them faster than new fires appear, and eventually you will win the game. 

I have been learning from an increasing number of teams that I consider to be on the leading edge of security, that they are taking a different approach. Instead of trying to tackle threats and vulnerabilities one by one, they are using the same tools to create signals about a problem or set of problems, and then tackling those problems strategically. That may be things like building shared infrastructure, common libraries and services, or putting controls in the right places. They are tackling problems strategically at the root cause. This is security insecticide. 

Whack-a-Mole = Reactively solving the problem (moles). 

Insecticide = Solve the underlying cause of the problem (grubs).

In many ways there are parallels with when MSFT first introduced what became known as patch Tuesday. Before patch Tuesday, MSFT would publish security patches whenever vulnerabilities were found. The effect was that security teams had to run around, constantly scanning hosts to find out which ones were vulnerable, and then patch them as fast as they could, one by one. Some patches inevitably broke some apps and everyone was running around with their hair on fire, all the time. When patch Tuesday was first announced, a lot of people were skeptical. “How can we leave vulnerable hosts unpatched for weeks?”, but the reality was that teams could now calmly think about how best to fix issues strategically, be better prepared by testing beta patches before rolling them out, have the right change control in place, the right telemetry and the right resources available. Patch Tuesdays introduced real efficiency gains in the update process, an overall improvement in the patch deployment process and a step function of improved security.

Of course critical vulnerabilities still need to be patched as soon as possible but they are, alarmism aside, few and far between.

As an illustration, an example I recently came across was a team tackling vulnerabilities in containers. Their previous approach was to scan workloads for known vulnerabilities, and then try and patch the ones considered to be important. Their better approach, a security insecticide approach, is to use gold images served from a managed container registry. This puts control back into the supply chain, and allows them to update things once upstream and let the gold flow downstream. Teams can use something like the excellent commercial ones from Chainguard or homegrown ones. They can have the right change control process including testing and notifying before updating, and the right telemetry and resources available to make sure updates are successful. 

Organizations like this one, moving to this insecticide approach typically tackle it in a few phases. 

First they ensure they have the right controls in the right place, and that means initially finding and shutting down shadow engineering, so they can make sure containers are being pushed and pulled from registries that they manage. Next they find all of the images being used across their organization, and what they need to contain before building ‘gold’ versions. Some teams use this step to identify and eradicate entire technologies. No more Python 2 or heaven forbid PHP. Teams then build a robust operational process to roll out the gold images, and keep them updated and free of vulnerabilities. Knowing zero risk is a fallacy, they ensure that they then monitor compliance from their managed registry, and that it is always operating within an acceptable threshold. 

This approach applies to lots of things. Identity and access management where permissions are too liberal or multi-factor auth is spotty, expensive security tools deployed in places where they have little benefits, a lack of event logging in custom applications and the list goes on and on and on. 

Whack-a-mole is dead, and teams are now embracing security insecticides. Welcome to the post alerts and issues era. 

If you are interested in this approach but not sure how to start, you can tackle a lot of scenarios using the free open source project Chalk. Take a look at the info you can capture during a ‘build’ and post deployment, we call an ‘exec’. https://crashoverride.com/docs/chalk/config-overview/metadata

If you want help exploring this approach with Chalk, or even a followup article with step by step instructions, let me know. 

As always, articles are published to both our company blog and LinkedIn (for comments and discussion). Its always worth signing up to our newsletter for updates about products, open-source, events and articles. Its in the footer of the web site - https://crashoverride.com