On Social Media Witch Hunts and Learning Important Lessons

I have been meaning to write an article like this for over a year but it never made its way to the top of my TODO list. Along came an avalanche of what I consider irresponsible social media commentary about the recent CrowdStrike issue and that rapidly changed. Social media has now become the primary source of news for many people, but along with that, it has become a dangerous form of citizen journalism and even political information warfare, where if content is left unchallenged, it becomes ‘truth’ to a lot of people, often with unexpected or unintended consequences.

It's raw for me because in August, the UK where I now live, saw riots with far right fascists and racists taking to the streets stoning mosques, burning temporary housing for immigrants (with them inside) and setting light to public libraries. Why? Because a rumor spread on social media, including my Internet scumbag Andrew Tate, that the perpetrator of a heinous crime the week before was an illegal immigrant from Rwanda. He wasn't, he was a first generation British citizen, but mob culture didn't care about understanding the truth or taking an educated opinion based on fact and just went on the rampage. 

I don’t need to recap what happened in the CrowdStrike saga to anyone reading this. I can't remember a security event that has reached as far and wide as this, and I hope we don’t again, but if we are to avoid history repeating itself then I think some parts of the security community have to take a step back and act more responsibly on social media than they have, and think about the unintended consequences of their behavior. 

What I witnessed was akin to a social media witch hunt. People started making broad statements that this type of bug should never happen, CrowdStrike needed a QA process and that they should train their developers. I saw one comment (I can't find it) from someone who trawled LinkedIn profiles and asked if CrowdStrike had more than one QA person. I saw a picture of a person at BlackHat wearing a “CrowdStrike intern t-shirt” and a number of comments from people offering advice about the use of RegExes. 

Instead of throwing virtual rocks, chastising developers and offering naive advice from the peanut gallery, appsec people should have been reflecting on the fact that zero risk in software is the same fallacy as zero risk in software in general, and instead think about why the inevitable catastrophe had such a impact and think about how to minimize that in the future. The general situation reminds me when social media pundits dump on open-source developers for vulnerabilities in their libraries, and talk about holding them accountable. The same people work at companies where they continue to pull the latest version of a library down, let it flow straight into their builds and then straight into production. They then throw rocks. 

As Jen Easterly from CISA said, ‘this (CRWD saga) is a dress rehearsal for what China could do in the future’ and if we don’t learn the actual lessons here we are doomed to repeat history.

Here are a few things to consider instead of chucking rocks.

• Monocultures

• The shared responsibility of supply chains

• Being a d%^k doesn’t help developer relations

Monocultures

I have referred to Dan Geers paper on monocultures (1, and 2) more than any other article in the last 25 years that I have been doing this security stuff. If you haven't read it you should. The synopsis is that if everyone uses the same technology, Windows being the primary example used in the paper, then you have a global threat environment where adversaries can develop pervasive exploits that have a very large and a very predictable attack surface. It is even more pertinent today in that it is a concept taken from biology and we have all been through Covid.

While this was a self-inflicted injury and the impact was largely denial of service, it still fits this description of the threat perfectly. CrowdStrike clearly has been an incredibly successful company and as a result has (had?) a lot of customers, estimated to be 30,000, and a lot of end-points under management, estimated to be 8.5 million. CrowdStrike and technologies like it may be an even bigger target than Windows because if the host has a CrowdStrike agent running, you can probably assume it's worth protecting. My mother-in-law ain’t running CrowdStrike on her laptop.

So a post-mortem question for me is that if the issue has been discussed for more than twenty years, then why aren’t people looking at this and asking why everyone is not talking about monocultures and how to mitigate that threat? Why aren’t the social media pundits not suggesting strategies to mitigate that threat? Why is the security industry as a whole not taking some level of accountability for thinking the monoculture is someone else's problem and it shouldn't have happened to them? You used to hear people talking about multi-cloud strategies. Less so today, but I have never seen teams deliberately using diverse technology to project against the monoculture security threat, it's always to prevent vendor lock-in against being hostage to potential costs. 

It is worth pointing out that mono-cultures for security is not a binary bad thing of course.  Standardizing on a small set of gold image containers that are free of known defects makes perfect sense, but it also potentially introduces the same conditions. Its the same for VS Code, design tools like Figma, Docker desktop and the list goes on. 

The Crowdstrike saga should be a wake up call that we are all relying on and creating monocultures, and what that could mean. This is going to happen again. As an industry I think we need to be thinking more about the risk introduced by mono-cultures and actively having a strategy to understand and manage the risk and the reward. 

The shared responsibility of supply chains

“Crowdstrike should have tested the release” …. “They should have had QA”. Really, no shit Sherlock, you think they became a massive company without testing their software? I personally know and respect George so I may have a biased opinion, but that's because I know the man and his standards. I have no idea of the test and QA process at CRWD, but I am sure it is the same as 99.999% of the negative security pundits on social media. What I do know with absolute certainty is that if you have ever built enterprise software, then you will know that even if you take all reasonable precautions, some things will still fall through the cracks. That's the reality of scale and complexity, and people claiming otherwise are sharing naive, romantic and uninformed opinions. 

So once again, no matter how angry people are, this should be a wake up call that bugs will always exist and when consuming other people's software, you have some level of shared responsibility whether that is to understand the risk or to manage it. I appreciate the nature of anti-malware software means that the timely release of protection profiles is important, and that at the time, CRWD did not offer the ability to turn off auto-updates, but when you are the downstream consumer, running isolated tests and rollouts is your part of the shared responsibility and if you can’t do that it should be part of your threat model.  

There is a shared responsibility when consuming other people's software and you can’t just blame others. You can hold them accountable for doing their part and no doubt CRWD should be held accountable, they  fell short, period, but when you take a step back then it should be a wake up call that this had the potential to happen when the right controls to mitigate the threat didn’t exist in the software, and you better have had  a good disaster recovery plan. I wonder how many desktop software vendors still roll-out auto-updates? If I were a foreign adversary I would have my team of agents building that list and hacking the vendors right now, before the industry realizes the threat and reacts. You snooze, you lose. 

It is worth noting that when MSFT moved to patch Tuesdays, they showed that we can improve the patch process by waiting for updates on a predictable schedule, allowing time, resources and planning to test and roll them out.

Being a d%^k doesn’t help developer relations

The common theme among many of these people throwing virtual rocks on social media is that I doubt most of them have ever written and shipped enterprise software in their lives. If they had, they would understand that it's complex and have some bloody empathy. It used to be the case that developers would avoid security at all costs. They would hide issues, avoid engagement and therefore not get expert help. All of this had a negative effect on the security quality of software. 

It has taken decades to make inroads into developers trusting and actively working with security people, and as we know from breaches, trust is hard to learn and easy to lose. If we throw rocks they will just recede and that will set everyone back years.

While on the topic, it;s also worth thinking about who you should take your advice from. I will always remember talking at an appsec conference, and asking a series of questions to the audience. Granted this was well over a decade ago and things have moved on, but the first was how many people knew how HTTP worked, and would consider themselves experts. Almost 100%. The second was how many people would consider themselves as ‘competent developers’ and could code in a language of their choice proficiently. I asked them to be honest and I think it was about 25% of the audience. That's at a conference about software security made up of people providing guidance about how to secure it. I then asked how many people had been commercial developers and it was literally a handful. I would like to think it is significantly different today, but if it's anything like the last appsec conference I went to two years ago, I very much doubt it. 

Come on, we can do better than this. 

Footnotes

1. As always this article is posted on LinkedIn (here) and our company blog (here). 

2. I planned to publish this to our newsletter a week before publishing it online. I took a vacation and my automation failed. Another article will be posted to the list this week, and then published online next week. You can sign up to the newsletter in the footer of the homepage https://crashoverride.com