Medical Chatbot Hacked Into Giving Dangerous Advice

Security researchers have demonstrated that a healthcare AI chatbot used in a US medical pilot can be manipulated into producing dangerous advice and misleading clinical notes, raising new questions about how safely AI can operate inside real healthcare systems.

What Happened?

Doctronic is a US telehealth platform built around an AI medical assistant (a medical chatbot) designed to help patients understand symptoms, manage conditions and connect with licensed doctors. The system is intended to act as a first point of contact in a digital care pathway, gathering patient information, offering guidance and preparing summaries for clinicians.

The idea of Doctronic is that patients can consult the AI about symptoms, medications or health concerns, and the system prepares structured information that helps doctors review cases more quickly.

Can Be Manipulated

However, the platform has recently attracted attention after being examined by Mindgard, an AI security company that specialises in testing the safety of AI systems.

In its research, Mindgard showed that the chatbot could be manipulated into spreading vaccine conspiracy theories, recommending methamphetamine as a treatment for social withdrawal, generating altered clinical guidance and even advising users how to cook methamphetamine.

According to the researchers, the issue stems from weaknesses in the chatbot’s internal instructions. As Mindgard explained: “System prompts are the ‘keys to the kingdom’ when it comes to chatbots.”

The issue is particularly sensitive because Doctronic is currently being used in a pilot programme in the US state of Utah. The project operates within a regulatory “sandbox”, which allows new technologies to be tested under controlled conditions. As part of the trial, the system can assist with managing patient queries and renewing certain existing prescriptions before cases are reviewed by a human clinician.

Why The Exploit Matters

The issue is more serious than a typical chatbot error or AI hallucination because Doctronic sits inside a healthcare workflow. The system generates structured medical summaries and guidance that clinicians may review as part of patient care. If that output is manipulated or incorrect, it could appear credible enough to influence how a case is interpreted.

The researchers warned that this creates a new type of risk. As they put it, “the most dangerous advice can come from the most well-intended of chatbots.”

How The Prompt Injection Works

According to Mindgard, the weakness it discovered involved a type of attack known as prompt injection.

Large language models (LLMs) operate based on internal instructions known as system prompts. These hidden instructions guide how the AI behaves, what rules it follows and what information it should refuse to provide.

Mindgard said it was able to trick the chatbot into revealing those internal instructions by manipulating how the conversation was framed. By convincing the system that the session had not yet begun, the researchers prompted it to recite its own internal instructions.

Once those instructions were exposed, the chatbot became easier to influence. The researchers then introduced fabricated regulatory bulletins and policy updates, which the system treated as legitimate information.

This allowed them to push the AI towards unsafe advice, including altered medication guidelines and fabricated medical guidance.

Why SOAP Note Persistence Raises The Stakes

The most concerning aspect of the experiment involved clinical documentation.

When users request a consultation with a human clinician, the system generates a structured medical summary known as a SOAP note. These documents summarise the patient’s situation and provide context before the appointment begins.

Mindgard found that manipulated information introduced during a compromised session could appear in these summaries and be passed on to clinicians.

In its report, the company warned that this could “actively undermine the human professionals who might trust its authoritative-looking output.”

While the document itself is not a prescription, it becomes part of the clinical context surrounding the patient. In busy healthcare environments, that context can influence how clinicians interpret a case.

In other words, manipulated AI output could enter a legitimate medical workflow.

What Utah Says About The Limits Of The Pilot

Officials involved in the Utah pilot have, however, been keen to point out that the programme includes safeguards.

The trial is limited to renewing certain existing medications and does not allow prescriptions for controlled substances. Additional checks are also applied before any prescription renewal is approved.

Doctronic has said it has reviewed the research findings and continues to strengthen its safeguards against adversarial prompts and manipulation attempts.

Those limitations reduce the immediate risk in this particular pilot. However, the research highlights the types of challenges developers may face as AI systems move deeper into healthcare processes.

The Wider Evidence On Medical Chatbot Risk

This incident also aligns with concerns raised by other recent academic research.

A major study led by the University of Oxford earlier this year examined how people interact with AI systems when seeking medical advice. The study compared people using AI chatbots with those using traditional sources of information.

Researchers found that participants using AI tools were no better at identifying appropriate courses of action than those relying on other methods such as online searches. In some cases, users struggled to interpret the mixture of correct and incorrect advice produced by the models.

The study concluded that strong performance on medical knowledge tests does not necessarily translate into safe real-world interactions with patients.

Crucially, the researchers argued that systems intended for healthcare use must be evaluated in real-world conditions with human users before being widely deployed.

What Does This Mean For Your Business?

For healthcare providers and regulators, the findings reinforce a familiar lesson from other safety-critical industries. Introducing AI into a workflow does not simply add automation. It changes how information flows and how people trust that information.

Healthcare systems already rely on structured documentation and clinical summaries. If AI systems begin generating those summaries, their reliability becomes a core safety question rather than a technical curiosity.

For organisations developing AI tools in high-trust environments such as healthcare, finance or legal services, the message is that technical accuracy alone is not enough. Systems must also be resilient to manipulation, misuse and subtle changes in context.

The Doctronic case illustrates that prompt security, audit trails and robust human oversight are not optional features but fundamental safeguards when AI systems begin influencing decisions that affect real people.

Although AI may eventually become a valuable support tool in healthcare, the evidence emerging so far suggests that the journey from promising technology to safe clinical practice is likely to be longer and more complex than first thought.