"Human-in-the-loop" has become the go-to reassurance for AI in healthcare. Put a person at the end of the process, and the tool is safe. Right?
Not quite. A piece out this week made a point worth sitting with: adding a human reviewer doesn't automatically make a system safe. Often it just moves the liability onto someone who's already stretched thin.
If an AI generates a recommendation and a clinician is expected to catch every error before signing off, you haven't added a safeguard. You've added a rubber stamp with a name attached. The human becomes accountable for a decision they didn't really have the time, context, or tooling to review.
Real oversight isn't a checkbox at the end. It means:
- The reviewer has enough time and information to actually disagree.
- The system is built to surface its reasoning, not just its output.
- Someone is watching how often the human overrides it, and why.
If the answer to "who's responsible when this goes wrong?" is "the busy person at the end," that's not oversight. That's offloading.
Meaningful human oversight is a design choice, not a disclaimer.
What this means for health organizations
If staff are expected to check AI output, give them the time, context and tools to do it, and measure whether the review is real. A very high approval rate can be a sign that review has become a formality.
