Skip to content
Security 7 min read

Lines of Code Is a Terrible Productivity Metric. It's a Pretty Good Risk Metric.

We spent a decade learning not to count code. AI just made counting it matter again, for a different reason.

Mike Flouton
Mike Flouton
7 min read
Lines of Code Is a Terrible Productivity Metric. It's a Pretty Good Risk Metric.

Before Crash Override, I spent a few years at GitLab, and one conversation came up with customers more than almost any other: how do we measure whether our developers are productive?

The progression was almost always the same. First it was lines of code. When we pointed out that a developer can write 400 lines to do what a better developer does in 40 (and that the 40-line version is usually the one you want), the conversation moved to merge requests. That was better, and GitLab itself used MR rate as a headline engineering metric for a while. But we knew it could be gamed too, so it came with guardrails: never measure individuals, never build a leaderboard, always pair it with quality metrics. Split a change into five tiny merge requests and your number goes up. Nothing your customers can actually use has changed.

Over time the whole industry developed a healthy, instinctive allergy to counting output. Goodhart's Law did the rest. And I think that instinct is correct, for productivity.

For security, it might be exactly backwards.

More code is more to secure

Here is the thing we glossed over while we were busy being right about productivity. Every line of code that ships is something that has to be reviewed, tested, maintained and defended. Lines of code was never a good measure of value. It was always a decent leading indicator of attack surface.

The important word there is ships. Not all code makes it to production. Plenty of it sits in branches, experiments and test fixtures that never run anywhere that matters. What counts for security is the code that gets built into an artifact and runs in production, along with everything it pulls in. That's a smaller number than lines written, but it grows with them.

GitLab's own team said as much. In a 2024 post on measuring AI's impact, they wrote that "more code means more to review, test, and maintain." Their public product discussions go further. The design work for their AI impact dashboard lists the problems with measuring AI productivity, starting with "lines of code can be misleading" and noting that AI suggestions might increase speed "at the expense of code quality, maintainability, or security." A separate discussion rejects per-developer AI metrics outright, because measuring developers by output "does not account for code quality, maintainability, or collaboration."

So the people building the dashboards know. And yet, if you follow those same public discussions forward, you watch the market pull them right back to volume. Customers buying AI coding tools need to justify the spend, and the easiest number to show is how much code the AI wrote. So we get lines of AI-generated code versus human code, acceptance rates, and ROI formulas built on "lines accepted." That isn't a knock on GitLab (every vendor in this space is under the same pressure). It's the industry in miniature: we all agree volume is a bad productivity number, but we're measuring AI with it anyway.

What nobody is doing is counting that volume for what it means on the security side. Not every line is exploitable, and most of it never will be. But every line that reaches production is one more thing someone has to be able to vouch for, and you can't tell which ones matter until you know what's actually running.

AI broke the ratio

Security teams have always lived with a ratio: code being shipped on one side, people and process available to review it on the other. The ratio was never great. AI is breaking it.

Look at what the leading edge does about it. One of our customers, Blackstone, ships 98% of its production deploys automatically, and its AppSec team reviews only the 1 to 2% that warrant a human look. That isn't a team keeping up by working harder. It's a team that decided, ahead of most of the industry, that people should only see the exceptions and everything else should be checked automatically on the way to production.

That model is the one that survives AI. Double the code going into the pipelines and the automated 98% absorbs it; the humans still see only what needs them. Now picture the more common program, where every release still waits on a person. Double the volume there and AppSec headcount doesn't double with it. Something gives, and it's usually the review.

That's the real risk story, and it doesn't depend on AI writing worse code than people. It only depends on AI writing more of it.

The code isn't getting better on its own

To be clear, I'm not anti-AI. Far from it: I'm routinely the most pro-AI person in my friend circle, which has led to some awkward dinners. And this isn't a "the robots write bad code" piece. Plenty of human-written code is bad.

What AI changes is who writes code and how much. Over a lazy Sunday recently, I shipped 13 pull requests with Claude's help. I'm a CMO, and I hadn't written a line of production code since early in my career. Multiply that across every team in your company that just discovered it can build things, and you have the volume problem in one sentence.

The data we have says AI isn't fixing the quality problem either; it's producing the same kinds of flaws at much higher volume.

GitClear analyzed 211 million changed lines of code from 2020 through 2024 and found that 2024 was the first year copy-pasted code outpaced refactored code, with duplicated blocks rising roughly eightfold. Duplicated code means duplicated bugs, and a fix applied in one place but not the others. Veracode tested more than 100 models on security-sensitive coding tasks and found that, given a choice between a secure and an insecure way to write something, the models picked the insecure option 45% of the time, and bigger models didn't do meaningfully better. (That's a controlled test, not a measurement of production code, but it's a useful signal about defaults.)

The attack surface you didn't write

The most concrete version of this risk isn't in the code your team writes at all. It's in what that code pulls in.

AI coding tools add dependencies freely, and sometimes they recommend packages that don't exist. Researchers at USENIX Security 2025 found that nearly 20% of packages recommended across 16 models were hallucinated. Newer models do much better (closer to 5 or 6% in a 2026 replication), but attackers have noticed. They register the names models tend to invent and wait. It even has a name now: slopsquatting.

No productivity dashboard counts that. Lines of code doesn't either, which is the point: the volume problem is bigger than the code you can see.

So what should you count?

Not lines written. Not merge requests merged. Not acceptance rates.

Every metric I've described stops at the merge. Time to merge, cycle time, acceptance rate, AI versus human lines: they all measure the act of producing code. None of them follow it to the one place where it can actually hurt you: production, along with everything it brought with it.

I think the ground truth here is that security teams need to measure what reaches production and runs, and they need to stop bad code before it gets there, at the speed it's now being written. You're not going to review your way out of a ratio problem. You need to know, for every artifact running in production, where it came from, what's in it and whether it should be there.

Measuring whether AI is actually making engineering teams more productive is genuinely hard. What we really want to measure is the line from intent to usable value, and that's still out of reach. I don't think it gets solved soon. But security doesn't have to wait for it. We may not be able to measure how much value AI is adding. We can absolutely measure how much it's adding to what we have to defend.

Share:
Mike Flouton
Mike Flouton
CMO

Mike Flouton is CMO of Crash Override, a former engineer who has spent nearly three decades in DevOps and cybersecurity product and marketing.

More in Security

View all posts

See what Crash Override finds in your codebase.

Book a demo with an engineer. We'll show you real results, not slides.