The Uncomfortable Questions · An AI FAQ for leaders

Question #22 · Governance

Can your board explain how your most important AI system makes decisions? Should it be able to?

The short answer

Boards don’t need to understand the math.

They do need to understand what the system optimizes for, where it’s likely to fail, and who notices when it does. If no director can answer those three questions, the board is governing on trust, not oversight.

Norm E. Murray

The psychology

Why it happens

Technical complexity gives boards a reason to defer, and deferring feels like delegation when it is really a gap in oversight.

01

Feeling informed is not understanding

Psychologists Leonid Rozenblit and Frank Keil named the illusion of explanatory depth: we believe we understand complex systems until we try to explain them step by step. A board that has seen a demo can feel informed without being able to answer a single hard question.

02

The board sees what it is shown

Management and vendors know far more about an AI system than directors do, a classic information asymmetry. Without deliberate effort, the board only sees what the people it oversees choose to show it.

03

Every system optimizes for something

Every model has an objective function: the one target it is built to maximize, such as conversion, approval rate or cost per case. If the board never asks what that target is, it cannot see the trade-offs the system makes on the company’s behalf.

04

Everyone assumes someone else is watching

When oversight is shared across the board, a committee and management, each group assumes another is watching. Psychologists call it diffusion of responsibility, and it is how failures go unnoticed until they are public.

The bottom line: If no director can say what the system optimizes for, where it fails and who notices, the board is trusting, not overseeing.

Your move

What to do about it

  1. Ask the three questions.For your most important AI system: what does it optimize for, where is it likely to fail, and who notices when it does?
  2. Require a one-page explanation.Management should describe each critical system in plain language a director can challenge.
  3. Name an owner.Make one executive accountable to the board for each critical system’s behavior, not just its uptime.
  4. Review failures, not demos.Ask to see the cases the system got wrong and what happened next.
  5. Build director fluency.Schedule regular briefings so at least some directors can ask informed follow-up questions.