There’s a natural excitement that comes with applying AI to healthcare. Every few months, we see another pilot that promises to push the boundaries of what’s possible - promising to strip away the administrative burdens and bottlenecks that have long defined clinical inefficiency.
The recent experiment in Utah, where AI was used to support prescription renewals, fits right into that narrative.
But what makes this story important isn’t just what the technology tried to do. It’s what happened when it hit reality.
The program was immediately suspended after regulators raised concerns that it wasn’t providing meaningful clinical value, and pointed to red flags around safety, oversight, and decision-making responsibility.
On the surface, the industry will label this as a failure. I see it differently.

What the Utah experiment actually revealed

The instinctive reaction to a halted pilot is to frame it as another example of “AI not being ready for medicine.” But that misses the more important signal.
The Utah pilot surfaced a fundamental truth about AI prescription safety in healthcare: the challenge isn’t whether AI can generate a recommendation, but whether the clinical system around it can absorb the accountability for that decision. (Aka, “human in the loop”)
Healthcare decisions don’t exist in isolation. Every prescription carries context like medical history, prior responses, and subtle clinical judgements that rarely live fully inside structured data. Even highly capable systems struggle when responsibility is distributed but not clearly defined.
In broader discussions around clinical AI governance, including guidance from the U.S. Food and Drug Administration (FDA) on AI-enabled medical devices, the central question is no longer just ‘can AI assist?’ but ‘where does the human accountability begin and end?’
This is where real-world deployments stall. It's not a failure of capability; it's a failure of defined responsibility.

Why “speed” is not the same as “safety”

A common justification for integrating AI into prescription workflows (or arguably any workflow) is efficiency. We talk about reducing administrative load, improving access, and closing gaps in overburdened systems. And those needs are very real.
But true AI prescription safety in healthcare cannot be measured by throughput alone.
Healthcare is one of the few domains where faster does not automatically mean better. A faster prescription loop still requires safeguards for the outliers, the exceptions, and the clinical uncertainty - areas where human clinicians consistently outperform systems that rely on probabilistic outputs.
Research from institutions like NEJM Catalyst consistently highlights the same tension: AI can improve throughput, but without strict governance and clinician oversight, it can also amplify risk at scale.
The Utah case didn’t fail because AI has no place in prescription workflows. It failed because it exposed how early we are in defining the rules of engagement.
And that might be the most valuable outcome of all.
Defining the rules of the human, at the N of 1.

Get insights like this delivered monthly.

Subscribe to the CancerGeek newsletter for exclusive perspectives on healthcare innovation, digital health trends, and more delivered straight to your inbox.