Usability Testing Enterprise Security Software: What Actually Works
SOC analysts won't give you an hour, so testing security software means redesigning the study around their time, not your script.
Most usability testing plans assume a participant has an hour free. Security analysts almost never do.
A standard SOC already runs on shift work, live incidents, and tool fatigue. One more hour mid-shift is a lot to ask. It’s why so many security UX studies end up with five confirmed sessions and two no-shows.
Why Standard Recruiting Breaks Down
Incident response pulls participants away with no warning. A confirmed session can vanish an hour before it starts. CleverX’s guide to cybersecurity product research recommends over-recruiting by 25 to 30 percent to absorb exactly that risk.
Role matters as much as headcount. A former SOC analyst now working in security sales won’t produce valid usability data. A study built for Tier 1 triage also needs different participants than one built for a CISO’s reporting view, according to CleverX’s companion guide on testing enterprise security tools.
Screening has to move fast too. Long screeners lose busy professionals before they finish. The research guide recommends five to seven screening questions, and five to eight participants per role, per round.
Sessions Have To Shrink To Fit
Thirty minutes, not sixty, is the realistic ceiling. Security professionals rarely commit to a full hour. CleverX backs that limit with reminder cadences at 48 hours, 24 hours, and two hours out, sent by text as well as email.
A cancellation doesn’t have to mean a lost session. The same guide suggests a fallback: give the no-show a 10-minute unmoderated task to finish within 48 hours instead.
Timing the study matters too. Scheduling around Patch Tuesday or a fresh vulnerability disclosure all but guarantees cancellations. That’s exactly when analysts get pulled into real work.
Five Honest Sessions Beat Fifteen Rushed Ones
Nielsen Norman Group’s argument for small samples was never really about budget. It’s about where the returns stop.
NN/g’s research on sample size models problems found as a function of participant count. One well-run session typically surfaces roughly a third of them. Five sessions find about 85 percent. Finding the rest takes closer to fifteen participants, and NN/g’s own recommendation is three rounds of five over one round of fifteen.
That logic fits security research better than most domains it was written for. Five honest, well-matched analyst sessions, with fixes made between rounds, teach you more than fifteen rushed ones. Especially fifteen squeezed into a single sprint before anyone agreed to show up.
Make The Test Environment Feel Real
A clean demo environment produces clean, useless findings. Analysts recognize a sanitized test the moment they see it. They stop behaving the way they would during a real shift.
CleverX’s research guide suggests loading a mock environment with 40 to 100 alerts. Keep the same 60 to 80 percent false-positive rate analysts triage every day, timestamped recently enough to feel current. Add real time pressure to the task itself, not just the alert count: triage the first five of 47 alerts from the last four hours, or contain a phishing report that came in ten minutes ago.
Scenarios should read like an incident, not a test script. CleverX frames this as presenting a MITRE ATT&CK-based situation in plain language. “Three users received suspicious emails and one clicked a link; investigate” works better than naming the technique being tested.
What Moderated Sessions Still Catch
Unmoderated testing scales. Dovetail’s guide to enterprise B2B testing, updated this year, puts moderated sessions at five to eight participants and weeks to results. Unmoderated runs reach ten to twenty-plus participants in days.
But scale isn’t the only variable that matters. Moderated, think-aloud sessions reveal why an analyst dismissed an alert or missed a pivot, not just whether they finished the task. Dovetail’s advice is to run a small moderated round first to find the real problems, then confirm the fixes with a larger unmoderated sample.
Censys describes its own research practice as mostly 30-minute moderated Zoom sessions against sandboxed test accounts, never a participant’s real data. Participants range from InfoSec analysts to CISOs. The team’s own read on why this works is blunt: “Cybersecurity professionals are almost always great interviewees or testers.”
Protect The Participant And The Data
Security researchers carry more risk into a study than most participants. Both CleverX guides call them high-value targets. Both recommend a bidirectional NDA before any session starts, sometimes requiring the participant’s own security team to approve it first.
Findings should get anonymized the same way: no names, no employer, no tool configuration. That caution matches what’s actually at stake in a SOC right now.
Devo’s survey of 200 security operations managers and directors found 84 percent of organizations have analysts unknowingly re-investigating the same incident more than once a month. Eighty-three percent feel overwhelmed by alert volume and missing context. Those are the conditions a well-run study exists to catch before a product ships, not statistics to recreate in a report nobody reads.
A participant who trusts the process comes back for the next round. One who feels exposed won’t. The next recruiting cycle gets harder for everyone building in this space.
The Caraballo Group designs and tests security product interfaces against the real constraints analysts work under, not a tidy demo script. If your last usability study produced a report nobody acted on, we’d be happy to talk through what a better one looks like. Book a call to get started.
Photo by Jefferson Santos on Unsplash.