What human testing still reveals

Automated tests are excellent at asking: does this still do what we already know it should? Human testing asks a different question: can a real person get through this path with confidence — and leave evidence you can act on?

Those jobs overlap less than marketing slides imply. You need both. This piece is about what humans are still uniquely good at revealing, and the research that supports running small, sharp human sessions instead of waiting for production to teach you.

What humans catch that scripts usually miss

• Confusion that isn’t a crash. The button works. The label is wrong. The empty state is blank. Automation green-checks the happy path; a person stalls and says why.
• Judgment calls. “Is this ready to publish?” “Does this feel trustworthy enough to enter a card?” Those aren’t assertion failures — they’re ship/no-ship signals.
• Multi-step reality. Staging logins, invite emails, WebGL quirks, browser permission prompts, “works on my laptop / fails on theirs.” Humans bring messy environments with them.
• Wrong mental models. People click the wrong primary action, skip a required step, or invent a workaround you never intended. That is gold for product and copy — invisible to unit tests.
• Severity as felt by a user. A layout shift might be a flake in CI and a trust-killer in a recording. Humans rank pain the way customers will.

In short: automation guards known contracts. Human testing surfaces unknown friction — especially the kind that never throws an exception.

The research that still holds

Small samples find most of the obvious problems

Jakob Nielsen and Tom Landauer’s classic model (INTERCHI ’93; popularized in Nielsen Norman Group’s “Why You Only Need to Test with 5 Users”) estimates that with a typical discovery rate, about five users uncover roughly 85% of the usability problems in an interface for formative research.

The important nuance: that is for finding problems, not for measuring precise conversion rates. For quantitative metrics (task success %, time-on-task averages with statistical confidence), NN/g guidance points much higher — often on the order of ~20+ participants, and commonly 40+ when you need tight confidence intervals.

For product teams shipping weekly, the practical takeaway is not “five is magic.” It is: run a few humans now, fix what they find, then run another small round — three studies of five beat one study of fifteen for design improvement.

Late fixes still cost more — carefully cited

Barry Boehm’s Software Engineering Economics (1981), drawing on dozens of projects, documented that the relative cost to fix a defect tends to rise as you move from requirements/design into test and operations. Later industry summaries often compress that into a neat 1∶10∶100 slogan.

Treat the round numbers as a direction, not a spreadsheet. The oft-repeated “IBM Systems Sciences Institute study” behind some 100× charts is thinly sourced (course notes, not a published dataset). Boehm’s broader finding is still useful: finding pain before customers do is cheaper than patching reputation, support, and hotfixes after release — especially for UX and workflow defects that never fail a unit test.

Diminishing returns are a feature

In Nielsen’s curve, the first user teaches you almost a third of what there is to learn about an interface’s problems; each additional person adds less new insight. That is why paying for endless identical sessions on an unchanged build is wasteful — and why a marketplace model with a clear brief, a few slots, and a structured form fits the research: enough humans to see the pattern, not a hundred to pad a slide.

What human testing is not for

• Guarding every edge case on every commit — that is CI and automation.
• Proving a 2% conversion lift — that is A/B or analytics at scale.
• Replacing live-traffic learning after you ship — session sampling and product analytics still matter.

Human testing earns its keep when you need evidence before a ship decision: a recording, a marked moment, and answers tied to the path you care about.

How to ask for evidence that travels

• Brief the mission. Numbered success criteria beat “explore and tell us how you feel.”
• Match the form to the brief. Ask about the steps you defined — not open-ended goals the instructions already answered.
• Require a trail. Screen recording plus marked stills beats a paragraph with no timestamp.
• Prefer severity over volume. Three solid submissions that agree beat twelve vague ones.

Bottom line

Human testing is not nostalgia for a pre-CI world. It is the cheapest way most teams have to answer: will a real person get stuck — and can I see how? The research says you do not need a lab full of participants to get that signal. You need a few careful ones, a clear task, and evidence you can ship (or stop) on.

If you want that as a paid, structured workflow — brief, slots, recording, approve/reject — see QA Testing on QATested. For how to keep those sessions short and decision-shaped, see brief, high-signal tests. For why that evidence still beats AI alone, see confidence from human testing. For why the “research is too expensive” instinct is often about the wrong package, see why user research costs so much. For learning from live visitors after you ship, see Usage Learnings.