The Uncomfortable Questions · An AI FAQ for leaders

Question #1 · Judgment

When your AI is right 95% of the time, who on your team is still checking the other 5%?

The short answer

High accuracy is exactly what makes people stop looking.

Often, nobody is still checking. The danger isn’t the 5% error rate. It’s that by month six, nobody remembers how to catch it. Every AI deployment needs a named person whose job is to doubt it.

Norm E. Murray

The psychology

Why it happens

A system that is almost always right creates the conditions for its own mistakes to slip through.

01

Automation bias

Psychologists use this term for our tendency to trust an automated system over our own judgment, and to stop scanning for its mistakes. The more reliable the system, the stronger the pull.

02

Vigilance fades

People are poor at watching for rare events. When errors are uncommon, attention drifts, and the rare error slips through precisely because it is rare.

03

Skills decay

The people who used to do the work by hand were also the ones who could spot a wrong answer. Once the AI does the work, that checking skill weakens.

04

Nobody owns the doubt

Checking an output that is almost always right feels like wasted effort. Unless someone is explicitly paid and praised for doubting the system, the job quietly disappears.

The bottom line: Accuracy is not oversight. If nobody is paid to doubt the system, nobody will.

Your move

What to do about it

  1. Name a doubter.For every AI system that touches a real decision, assign a person whose job includes challenging its output.
  2. Know where the 5% lives.Ask which kinds of cases the system gets wrong. Errors usually cluster, and that is where review time belongs.
  3. Test the checkers.Every so often, put a known error in front of the reviewers. If nobody catches it, your oversight exists on paper only.
  4. Keep the hands-on skill alive.Have people do part of the work without the tool, so they still know what a wrong answer looks like.
  5. Reward the catch.When someone spots an AI mistake, make it visible and celebrate it. That is the behavior you need more of.