The safety you saw in the demo may not be the safety you bought.
A study published in October in the Journal of Medical Internet Research, led by researchers at RAND, sent the same 30 suicide-related questions to five widely used AI models, 100 times each, two ways: through the public chat app, and through the API that developers build products on.
Same models. Different behaviour.
Through the API, the models answered directly 77% of the time, compared with 62% in the consumer apps. The gap was widest on the highest-risk questions: 25% direct answers through the API, under 5% in the apps.
The reason is structural. Many safeguards live in the app, not the model: moderation filters, crisis responses, system instructions. A product built on the API doesn't inherit them. Its protections are whatever its developer added.
That matters for health organizations, because nearly every AI tool you'd license, whether it's a patient-facing assistant, a triage bot or a documentation tool, reaches its model through an API.
So "it's built on a leading model" isn't a safety answer.
The study measured whether a model answered, not whether the answer caused harm. The authors' conclusion still holds: safety observed in an app shouldn't be assumed to carry over to products built on the API.
What this means for health organizations
Put safeguard questions into procurement, not just demos. Ask vendors to document the safeguards they added, show test results for the sensitive topics your services deal with, and tell you when the underlying model changes.
If a tool talks to clients directly, test it yourself with the kinds of questions your clients actually ask.
