SBOMs for Production Incident Response Maybe a Killer Trojan Use Case for Security

SBOMs are more valuable for platform engineers than they are to security engineers today, and why this will help security in the long run.

I get why SBOMs will be powerful in the future, but as I wrote a while back, the SBOM frenzy is premature. As of today the frenzy has been largely driven by non-technical people lobbying politicians for adoption, but neither the lobbyists or the politicians understand the underlying issues with the technology, and so what they thought they would get, an accurate and transparent ingredients list of vendors software, that they could map against vulnerabilities and hold suppliers accountable, is, in reality, pure theater. Effective security attestation inside the build pipeline with SLSA is reality but outside is not. 

To be clear, this is not an SBOM problem, the CycloneDX spec and community is superb. It's the gaps and faults of the ecosystem needed to use SBOMs effectively that is. . 

Package managers for instance are complicated, and so you usually get different results each time you build from different build servers. That means different vulnerability reports. 

An SBOM must match a release and teams release often. If you use an SBOM as part of a business contract, then what is in production will be out of date and if you just specify SBOMs must exist, then you are effectively auditing vulnerability management programs for all of your vendors. 

SBOMs are only focused on third party code, the open source libraries. Sure the OSS code these days makes up a growing portion of a final piece of software, but it's hardly line-for-line value, and this ignores many classes of vulnerabilities and threats, including things like insider threats that are being actively exploited by bad actors. 

It is impossible to bind an SBOM and the custom code to a COTS application unless you share the source code with the purchaser or an escrow company. This point should not be lost on people. It’s huge.

Last and definitely not last given the core security use case is the reality that CVE’s are a radical subset of all the vulnerabilities out there, and while better than nothing, total theatre if you think being CVE free is the same as being vulnerability free. 

I believe that all of the problems above will be solved, or improved, to the point they are no longer show stoppers for anyone. Things like hermetic builds, zero knowledge SBOMs that cover custom code, maybe a digital trust hierarchy, code escrow schemes, better vulnerability analysis and vulnerability databases, are just a few of the options that will no doubt emerge.

But even taking into account the reality of these issues today, I still strongly believe that we should all be pushing for SBOMs everywhere across the DevSecOps pipeline. Everywhere. 

Today there is a gap in where SBOMs are usually deployed, which is production. This matters because what we learned from the Log4J security issues, was that companies deployed armies of technicians to figure out where it was actually running in production, rather than being present in a repo. Said another way, even with SCA today, there is a serious production visibility problem. 

But here is the good news. I believe we may well be seeing a new killer ‘trojan’ use case emerging, that is similar, but different, to the SCA one that I believe initially fueled vulnerability analysis for OSS libraries. I call it ‘trojan’ because it first and foremost solved a developer problem, and then it solved a security problem by proxy. That first killer use case was developers wanting to solve dependency hell, and in doing so solved vulnerability management of OSS libraries. Out of date libraries or vulnerable libraries, it's all the same. 

This was undoubtedly good for the source code end of the DevSecOps pipeline, but didn't touch the downwards part that happens after the build and where code is put into production. That is where the visibility gap exists today. The new killer use case I am seeing emerge is again a developer lead one and adds SBOMs to production, closing the visibility gap. 

What I am seeing as I observe Chalk deployments, is that when  a production incident occurs, such as a service down, or a service not behaving as intended, engineering teams rally to figure out what is going on, and immediately want to know exactly what code is deployed. Is it a particular branch from the custom code, or particular versions of libraries? There are many other things but not relevant to this article. They want to know this in order to replicate the production environment locally, and are working in a safe place, but here's the rub. Most developers don't have access to production. It's the visibility gap. That is usually by design but it's a problem all the same. Imagine if they had access to a production SBOM library, they could immediately have the right information at hand. 

I am excited that I am increasingly seeing software engineers view SBOMs as a solution to that lack of production visibility problem because this will also solve a core security use case that is not solved today. Where there are SBOMs everywhere, we will no longer have to run around like headless chickens searching for the next Log4J in production. 

Blue skies.

This article is cross-posted to LinkedIn here for comments and feedback.