Skip to main content

CarePlan AI is a configurable care and service management platform with built-in AI, designed for healthcare, community, and service-based organizations across Canada.

Contact Info

Safe AI Sunday

The policy was in place. A coworker is what caught it.

The policy was already in place. What caught the problem was a coworker.

In July, an employee at a state-run youth mental health facility in Washington reported that a colleague was using ChatGPT to draft summaries for client case notes (opens in a new tab). When the state looked further, it found the same staff member had done it for other patients too, using a personal account. Names, diagnoses and medications may have been exposed. Families of 12 patients were notified in August, and a second state facility found four more patients affected.

The state's policy already said staff couldn't use AI without approval. The employee was told to delete the prompts and then the account. In its letter to families, the state said it doesn't know how the information is stored on ChatGPT's servers.

A few things stand out:

  • The work being shortcut was documentation, the part of the job where staff are most desperate to get time back.
  • The policy didn't detect anything. A colleague did.
  • Deleting the prompts was the remedy, and no one at the facility can confirm what that actually removed.

Banning a tool is the easy part. The harder questions are whether staff have an approved option for the task they're trying to speed up, and whether anyone would know if they went around it.

If one of your staff did this tomorrow, how would you find out?

What this means for health organizations

Most health organizations now have an AI policy. This case shows where a policy stops working on its own: the rule existed, and it neither prevented the use nor detected it. Detection depended on one colleague being willing to speak up.

The pressure behind it is real. Documentation is where staff lose the most time, and consumer AI tools are a phone away. Banning tools without addressing that pressure leaves it in place. A more durable approach pairs a clear policy with an approved option for the tasks people are trying to speed up, training on why consumer tools don't fit health information, and a way for staff to raise concerns that doesn't feel like an accusation.

Questions to bring to your team

  • Which tasks are staff most likely to hand to a consumer AI tool, and do we offer an approved alternative for them?
  • If a staff member put client information into a personal AI account, how would we find out?
  • Do staff know how to raise a concern about a colleague's AI use, and would they feel safe doing it?
  • If it happened here, what would our response be, and what could we actually confirm had been deleted?
Share:
CarePlan AI dashboard preview

Let's Chat

No pressure or pitch, just a conversation to see if we're a good fit.