A new study has found that ChatGPT Health, an AI tool offering medical advice, frequently fails to recognise medical emergencies, with experts warning it could lead to unnecessary harm and death.
The first independent safety evaluation of the platform, published in Nature Medicine, found it under-triaged more than half of cases where immediate hospital care was needed. Researchers created 60 realistic patient scenarios, from mild illnesses to emergencies, and generated nearly 1,000 responses by varying patient details.
In 51.6% of cases where hospital admission was required, the platform advised staying home or booking a routine appointment. In one simulation, it sent a suffocating woman to a future appointment she would not live to see in 84% of attempts. Conversely, 64.8% of safe cases were told to seek immediate care.
Dr Ashwin Ramaswamy, lead author, said he was particularly concerned about the platform's under-reaction to suicidal ideation. When a patient mentioned normal lab results, a crisis intervention banner disappeared in all 16 attempts. “A crisis guardrail that depends on whether you mentioned your labs is not ready,” he said.
Alex Ruani, a doctoral researcher at University College London, described the results as “unbelievably dangerous” and warned that a false sense of security could cost lives. OpenAI said the study did not reflect real-world usage and that the model is continuously updated.
