# AI Promised to Fix the Vulnerability Backlog. It's Making It Worse.
Security teams are drowning in alerts. The fix everyone bet on isn't working.
The pitch was straightforward: large language models would cut through the noise in application security scanning — smarter triage, fewer wasted hours, faster remediation. The reality, according to testing of more than a dozen AppSec tools, is that over 60% of AI-flagged vulnerabilities are either false positives, sitting in dead code that nothing will ever call, or so low in severity they shouldn't be touching anyone's queue in the first place.
That figure comes from Arshan Dabirsiaghi, CTO and co-founder of AI security startup Pixee, who plans to detail those findings at Black Hat USA next month. The conclusion he's walking into one of the industry's biggest stages to deliver: stock LLMs aren't the solution, and the reason is structural.
## The Context Problem Nobody Solved
When a language model flags a vulnerability, it's doing pattern matching against training data. What it isn't doing — at least not reliably — is understanding whether the vulnerable code path is actually reachable in your application, whether your runtime controls already mitigate it, or whether the dependency it's screaming about is even compiled into your production build.
That distinction is everything. A vulnerable function buried in a dead branch of an abandoned library is not the same threat as an exploitable SQL injection sitting one hop from your authentication endpoint. Both look like findings. Only one should wake anyone up at 2 a.m.
Dabirsiaghi's formulation is blunt: "LLMs are dumb." Not unintelligent in the abstract sense — the latest frontier models are genuinely impressive at spotting code patterns associated with known vulnerability classes. The problem is that "dumb" in AppSec has a specific meaning: no awareness of what actually runs, what's exposed, what controls exist around it.
## Fifty Percent More CVEs and Counting
The timing of this reckoning is brutal. FIRST is projecting that CVE assignments will jump 50% this year. Microsoft has been setting volume records on Patch Tuesday for months. The pipeline of potential issues isn't slowing down — it's accelerating, partly because AI-generated code is producing novel security mistakes at scale and partly because the CVE system is finally catching up to the enormous backlog of legacy software issues.
So security teams face a numbers problem and a tools problem simultaneously. The volume of incoming findings is rising sharply. The tooling meant to help prioritize that volume is injecting more noise than signal.
Dabirsiaghi describes the fork in the road AppSec teams now occupy: let Dependabot run free and watch your CI pipeline choke on an endless stream of dependency-update PRs, most of which address vulnerabilities in code paths your application never touches. Or assign humans to manually evaluate every finding and try to surface the 8% that actually matter. Companies don't have the headcount for the second option. The first buries engineering in churn.
## Why Reachability Is the Hard Part
The gap isn't that AI is bad at finding vulnerabilities. Current models, including the latest generation, are genuinely good at pattern recognition across codebases at a scale no human team could match. The gap is in reachability analysis — the computationally expensive work of mapping which code paths are actually live, which inputs could reach them, and which controls sit between an attacker and the vulnerable function.
That kind of analysis traditionally requires static analysis tools built specifically for that purpose, often combined with runtime context from actual deployments. Layering an LLM on top of a scanner that lacks this foundation doesn't fix the underlying problem. It adds a language interface to the same flawed signal.
The models also run slower and cost more than established scanning tools for comparable coverage — a detail that tends to get buried in vendor marketing but matters enormously when you're running scans across a large codebase on every pull request.
## What Would Actually Help
Reachability-aware scanning isn't new. Tools that do call graph analysis to determine whether a vulnerable code path can actually be triggered exist, and some of them are genuinely useful. The challenge is integrating that analysis into developer workflows without introducing so much latency that engineers route around the gate entirely.
The honest answer for most teams right now: AI-assisted triage works best as a filter layered on top of context-aware static analysis, not as a replacement for it. Use the model to help explain findings and surface the most readable remediation paths. Don't trust it to distinguish critical from noise on its own.
---
## HackWire Analysis
The deeper story here isn't that LLMs fail at AppSec. It's that the industry rushed to deploy AI on a problem that was already structurally broken before AI entered the picture.
Vulnerability management has had a prioritization crisis for years. CVSS scores are notoriously poor proxies for actual exploitability in specific environments. Scanners have generated false positive rates that would get a human analyst fired. The backlog of unresolved findings at large organizations has been growing faster than teams can clear it since well before GPT-4 shipped.
AI tools landed into that environment and inherited its failure modes — then added new ones. The reachability problem is a perfect example. Rather than solving the context gap in existing scanners, many vendors wrapped a language model around the same underlying analysis and called it AI-powered triage. The output looks smarter. The signal quality is largely unchanged.
Dabirsiaghi's Black Hat presentation matters because it puts real numbers on a dynamic the industry has been dancing around: the emperor isn't clothed. Sixty percent false-positive or irrelevant findings isn't a quirk of a specific tool; it's a characterization of the current state of AI-assisted AppSec scanning as a category.
For defenders, the practical implication is to resist vendor pressure to treat AI vulnerability prioritization as a solved problem. Demand reachability data. Measure your actual false-positive rate before and after any AI tooling addition. And watch for whether the AI layer is genuinely reducing human triage time or just adding a chatbot on top of the same alert queue you already couldn't staff.
The CVE surge is real and it's accelerating. That means the cost of getting prioritization wrong — in both directions — keeps rising. Missing a reachable, high-severity finding buried in noise could be catastrophic. Spending engineering cycles chasing phantom vulnerabilities is slow-motion damage. Neither is acceptable.
AI isn't going to solve this on its own, and Black Hat's timing is appropriate: the hype cycle is about to meet the data.
— HackWire Editorial
---
## Related Coverage