Why Analysts Don't Trust AI Security Verdicts They Can't Verify
Security platforms now ship a confidence score next to every alert, but analysts keep pulling the raw logs anyway because the score alone gives them nothing to check.
Most security platforms now attach a verdict to every alert: malicious, benign, needs review. Analysts still open the raw logs before they act on it.
That habit isn’t paranoia. It’s the rational response to a conclusion with no visible work behind it.
The Confidence Score Nobody Takes At Face Value
AI is no longer a side feature in the SOC. Darktrace’s 2026 survey found 77% of security stacks now include generative AI, with agentic AI running inside two-thirds of them.
Adoption outran trust. The same survey found 74% of respondents still restrict what AI can do autonomously until its explainability improves. And 86% won’t let it take even a small remediation action without a person signing off first.
Arctic Wolf’s 2026 trends report surveyed 1,350 security and IT decision-makers. Just 53% trust AI to perform even a narrowly defined action, like blocking a malicious IP at the firewall. Only 14% have made AI central to their security strategy at all.
The pattern repeats across every survey worth reading. Teams adopt the tooling fast and extend it trust slowly, in that order.
A Verdict Isn’t The Same Thing As Evidence
This gap isn’t just about model accuracy catching up over time. It’s structural. A verdict by itself gives an analyst nothing to check.
AirMDR built an entire piece of research around exactly this failure mode, titled bluntly: don’t trust the AI, inspect the investigation. A polished conclusion that hides its evidence, its confidence level, and the data it never collected isn’t reviewable. It’s an opinion with better formatting.
That distinction matters more as pilots mature past the demo stage. Gartner projects that by 2028, 70% of large SOCs will pilot AI agents for tier-one and tier-two work, but only 15% will see measurable improvement. Explainability ranked as a barrier for 41% of surveyed teams, and 57% already require a human to review every AI decision before closing an alert. Not one respondent granted a tool fully unsupervised autonomy.
More Usage Hasn’t Bought More Confidence
If familiarity closed the trust gap on its own, 2026 should have been the year it closed. It wasn’t.
SANS’s 2026 AI survey of 536 practitioners found 63% report significant shortcomings in AI-driven detection and response, even as active generative AI use in security work climbed to 78%. Teams are using these tools more and trusting them the same amount, or less.
That’s the opposite of what a maturing technology usually produces. It’s a signal that the trust problem isn’t about model quality catching up with exposure. It’s about what analysts can and can’t verify once a verdict lands in their queue.
What Explainability Gets Wrong
Vendors hear “analysts don’t trust the verdict” and ship an explanation panel. That’s usually the wrong fix.
Vectra’s framing of the agentic SOC draws the real line. As its guidance puts it, “an agent reasoning over noisy or incomplete telemetry automates uncertainty rather than resolving it.” A fluent, after-the-fact explanation can describe a conclusion convincingly. It still doesn’t prove that conclusion held up.
Repeatability is a cleaner test than eloquence. If the same alert, run through the same model twice, produces two different verdicts, no amount of generated reasoning text fixes that. It’s a consistency problem, not a communication one.
Designing For Inspection, Not Persuasion
The fix isn’t a better-written explanation. It’s a case file an analyst can actually interrogate.
That means surfacing the evidence a verdict relied on, not only the conclusion it reached. It means disclosing what data was missing instead of quietly assuming the gap didn’t matter. It also means attaching a confidence score that behaves like a real signal. A high-confidence call built on partial data should read as riskier than a careful low-confidence one that says so upfront.
- Store the reasoning trace, not just the disposition. An analyst reviewing a closed case weeks later needs the evidence chain the model saw, not a label.
None of that has to live in dense text. A verdict card can show its supporting evidence as linked artifacts, flag missing telemetry with a visible badge, and let an analyst expand the reasoning chain inline instead of reading a paragraph of generated prose.
Where review happens matters as much as what’s shown. Reserve human sign-off for decisions that are hard to undo, such as access changes, service disruption, or irreversible containment. Let the model handle lower-stakes triage on its own, as long as its reasoning stays visible enough to audit afterward.
That’s a narrower ask than “fully explainable AI,” and a more honest one. Security leaders don’t need every verdict justified in plain English. They need to know that if they pulled on a thread, it wouldn’t come apart.
The Caraballo Group designs the verdict cards, case files, and investigation workflows that make AI features in security platforms something analysts actually trust, not just tolerate. If your product ships confidence scores nobody checks, we’d be happy to take a look. Book a call to get started.
Photo by Joshua Hoehne on Unsplash.