Digital rights experts have raised concerns about the security of data recorded by artificial intelligence during psychiatric sessions. A Melbourne psychiatrist has sparked debate by refusing to accept new patients unless they consent to the use of an AI scribe to transcribe their conversations.
AI Scribes in Medical Practice
AI-driven note-taking tools are gaining traction in the healthcare sector, with two in five general practitioners now utilizing such scribes, according to the Royal Australian College of General Practitioners (RACGP). However, this trend has also prompted worries about data security, potential misuse by AI companies, and the accuracy of transcriptions.
In a registration form reviewed by Guardian Australia, psychiatrist Dr. Hemlata Ranga outlined the requirement to prospective patients. The form states: “I consent for use of AI transcription (such as Heidi health AI/ Microsoft) software to assist with notes taking during the appointments, for preparation of clinical letters, reports, and other clinical documents to assist in my clinical care.”
Patients who do not wish to allow AI are informed they will “need to arrange [their] regular doctor/ referring doctor to refer [them] to a different service provider.”
Patient Concerns
The uptake of AI scribes has doubled over the past year, according to the RACGP, coinciding with increased demand for mental health services. One anonymous patient, who was referred to Ranga, expressed objections to the mandatory use of AI. The patient described the wording of the form as “defensive” and speculated that it may reflect numerous previous objections from others. The patient noted the difficulty of finding an alternative psychiatrist, especially given that Ranga came highly recommended. They have not sought another provider.
Clinic and AI Provider Responses
Dr. Ranga, whose practice is located at the Melbourne Clinic in Richmond, did not respond to a request for comment. A spokesperson for the Melbourne Clinic stated that the clinic does not direct its accredited psychiatrists on AI use but acknowledged that many are incorporating it to manage administrative burdens. The spokesperson emphasized that psychiatrists must disclose AI use and obtain patient consent; if consent is not granted, AI should not be used. Practitioners at the clinic are independent from the clinic itself.
Heidi AI, a popular AI scribe, has been used in 115 million sessions over the past 18 months. Co-founder Dr. Tom Kelly stated in March that data is processed locally and not used to train AI or sold to third parties. The company employs third-party testing and auditing to maintain data security and aims for high transcription accuracy, though doctors are still expected to review their notes.
Risks and Regulatory Gaps
Tom Sulston, head of policy for Digital Rights Watch, highlighted that AI transcription is imperfect and often trained on demographics such as male, white, heterosexual, English-speaking users, leading to more errors for other groups. He warned of a “very real risk” of medical data breaches or leaks, and noted that patients wary of AI may self-censor, particularly regarding stigmatized health issues like sexual or mental health. Sulston argued that AI is implemented to reduce administrative costs rather than improve patient outcomes, stating, “In this scenario, patients are not the customer, they’re the product.”
The RACGP reported that consumer groups have raised concerns about inadequate consent discussions and power imbalances between patients and doctors. Sulston called for stronger regulation, noting that AI scribes are currently exempt from Therapeutic Goods Administration regulations because they do not diagnose patients. He urged regulators and legislators to establish a legally protected right for Australians to refuse AI systems without jeopardizing their health.
The patient emphasized that people should have the option to opt out of AI scribes and expressed doubts about the accuracy of AI-generated notes.
