When session replay is enough (and when it isn’t)

Session replay is a good tool. It is a bad ship gate. The mix-up is treating a recording of whoever showed up as proof that a stranger can finish the job you care about.

Replay answers what did this visitor do? A structured test answers can someone finish this path on this build, and can I see the proof? Keep those questions apart and both tools earn their keep.

When replay is enough

Use it after the product is live and people are already in it:

• You do not know which path to brief. Traffic wanders. Rage clicks, dead ends, and drop-offs show you stalls you would never have written into a mission.
• The environment is the real one. Dirty accounts, real data, the theme they actually use — not a clean staging user you invented.
• You need the movie of a bounce. Support said “checkout is confusing.” Replay can show the hover, the back-button, the field they never filled. That is cheaper than guessing.
• You are learning, not settling. There is no approve/reject. You are collecting clues for a later brief, a copy change, or a funnel you will measure in analytics.

That is post-ship learning. It is how you find the next question. It is not how you decide whether Friday’s build is ready.

When it is not enough

Replay fails as a gate for a named decision:

• No mission. The visitor came to read a blog post. They never attempted invite, checkout, or “create the first project.” You cannot treat their session as a pass or fail on those paths.
• No comparable answers. Ten replays are ten stories. You cannot line them up against the same form fields. You can only argue about what you think you saw.
• Staging has no audience. Pre-ship, the people who would bounce are not on the URL yet. Watching empty replay is not caution. It is waiting.
• Independence is accidental. Whoever landed is not a cold tester you paid to attempt the job. Fans, bots, and returning users pollute the sample without a brief to filter them.
• You cannot reject it. A replay is not a submission. There is no slot to reopen, no “incomplete — try again,” no settlement. If you cannot reject the evidence, you are not doing QA. You are watching.

If the decision is hold the release, rewrite the empty state, or fix the invite, you need people aimed at that path on purpose — usually before customers are the ones teaching you.

A simple split

After you ship: watch sampled live sessions to find stalls you did not predict. Turn a stall into a finding. Then, if the finding is a ship question, write a brief and have humans attempt that path on the next build.

Before you ship: do not wait for traffic that does not exist. Name the path, pay a few independent people to finish it, collect a recording plus answers you can approve or reject. Scripts still guard contracts. Replay still waits until there are visitors.

Do not expect replay to prove conversion. First-run friction shows up as a bounce; lift is an analytics claim. Replay can show the stall. A structured pass can confirm a stranger hits it on purpose. Neither replaces measuring the funnel.

On QATested

Those two jobs sit next to each other here: Usage Learnings samples live visitors after you ship; QA Testing is the structured pass before (or instead of) learning from production. Related: structured QA vs an open beta · what human testing still reveals · the testing gap between big companies and everyone else.