(This story runs with a Special Report: “As AI enters the operating room, reports arise of botched surgeries and misidentified body parts”)
AI apps claim to assist patients but sometimes deliver flawed advice
Doctors express concern over AI’s medical acumen, warn of potential harm
FDA allows AI medical apps to provide ‘patient education,’ but not diagnoses
Dermatology app was bounced from App Store after Apple found it broke guidelines
By Steve Stecklow
Feb 9 (Reuters) - Artificial intelligence is shaking up industries from software and law to entertainment and education. And as physicians like Dr. Cem Aksoy are learning, it’s posing special challenges in medicine as patients tap AI for advice.
Aksoy, a medical resident at a hospital in Ankara, Turkey, says an 18-year-old patient and his family recently panicked after the young man was diagnosed with a cancerous tumor on his left leg. They turned to OpenAI’s ChatGPT. The bot said he might survive only five years.
It was wrong: A plastic surgeon successfully removed the tumor in July. “He was essentially cured after the operation,” Aksoy said.
But a few weeks later, the patient called Aksoy on the verge of tears. “He said, ‘I started coughing recently, and ChatGPT told me it could possibly be metastasis to my lungs,’” meaning the cancer had spread, the doctor recalled. The patient said he needed to write a will. It turned out his lungs were fine. He was coughing because he’d recently started smoking.
“When someone is distressed and unguided,” Aksoy said, an AI chatbot “just drags them into this forest of knowledge without coherent context.”
A spokesperson for OpenAI said its newest models have significantly improved how they handle health questions. ChatGPT isn’t intended as a substitute for a medical professional’s guidance, the company said.
The young Turkish patient’s encounter with AI-dispensed medical wisdom comes as many patients around the world are turning to the technology for advice. In addition to the big ask-me-anything chatbots, consumers are turning to a slew of new, AI-powered consumer medical apps.
‘BECOME YOUR OWN DOCTOR’
A growing number of mobile apps available on the Apple and Google app stores claim to use AI to assist patients with their medical complaints – even though they’re not supposed to offer diagnoses.
Under U.S. Food and Drug Administration guidelines, AI-based medical apps don’t require approval if they “are intended generally for patient education, and are not intended for use in the diagnosis of disease or other conditions.” Many apps have disclaimers that they aren’t a diagnostic tool and shouldn’t be used as a substitute for a physician. Some developers seem to be stretching the limits.
An app called “Eureka Health: AI Doctor” touted itself as “Your all-in-one personal health companion.” It stated on Apple’s App Store that it was “FOR INFORMATIONAL PURPOSES ONLY” and “does not diagnose or treat disease.”
But its developer, Sam Dot Co, also promoted the app on a website, where it stated in big letters: “Become your own doctor.”
“Ask, diagnose, treat,” the site stated. “Our AI doesn’t just diagnose – it connects you to prescriptions, lab orders, and real-world care.”
Apple said that after learning about Eureka Health from Reuters, it removed it from its app store. Apple’s guidelines for developers states that medical apps “must clearly disclose data and methodology to support accuracy claims.”
App developer Sam Dot didn’t respond to a request for comment. But the website changed after Reuters inquired about it. The site no longer mentions the app.
In some cases, apps have given inaccurate and potentially dangerous advice.
“AI Dermatologist: Skin Scanner” says on its website that it has more than 940,000 users and “has the same accuracy as a professional dermatologist.” Users can upload photos of moles and other skin conditions, and AI provides an “instant” risk assessment. “AI Dermatologist can save your life,” the site claims.
Its Lithuania-based developer, Acina, says the app uses “a proprietary neural network” that looks for patterns to make predictions. Acina says it was trained on dermatological images to recognize specific skin conditions.
<h3>DERMATOLOGY APP CLAIMS ‘97% ACCURACY’</h3>
The app claims “over 97% accuracy.” But it has drawn hundreds of one-star reviews on app stores, and many users complain it’s inaccurate.
Daniel Thiberge, a tech-support analyst in New Jersey, told Reuters that he bought the app to interpret seven pictures he snapped of a small growth on his arm. Six results showed there was a “75%-95%” risk it was cancerous, he said. He then went to a dermatologist. The doctor told him the growth didn’t look problematic in any way, and it wasn’t worth doing a biopsy.
“If it’s completely, wildly off, what is the purpose of the app?” Thiberge asked. At best it’s useless, he said. “At worst, it’s dangerous, because you may not go see a dermatologist.”
In another review on the Apple App Store, a user wrote that to test the app, she uploaded photos showing she had melanoma, a serious form of skin cancer, that had been diagnosed and surgically removed. But the app reported that the condition was “benign,” wrote the user. She told Reuters that she fears “some people will trust it and delay doctor visits.”
Reuters didn’t independently confirm the app users’ experiences. Acina said it couldn’t verify them. It told Reuters that AI Dermatologist’s “purpose is not to provide a medical diagnosis, but to offer a preliminary analysis using AI technology to encourage users to consult a professional.”
“Our AI models are built upon dermatological literature and carefully curated datasets that were selected and validated by board-certified dermatologists,” it said, adding that “false positives can happen with any AI system.”
<h3>A DOCTOR WORRIES ABOUT APPS’ ACCURACY</h3>
The company said its AI has received many positive online reviews, including “where users thank us because the app prompted them to check a mole or lesion early – in some cases leading to timely medical attention.”
Apple said it removed the app from its App Store after learning about it from Reuters, in part because of the numerous customer complaints.
Google also removed AI Dermatologist from its Google Play store after Reuters called attention to the app. “Google Play prohibits apps from offering misleading or harmful health functionality and requires regulatory proof or a disclaimer for apps offering medical functionality,” the spokesman said.
But the app is back on the market. Google recently reinstated it after Acina revised it. Google said suspended apps can return if they’re updated with a “compliant version.”
Acina said it “clarified more explicitly that the app is not a medical device,” doesn’t provide diagnoses, and that users should consult healthcare professionals.
Apple also briefly reinstated it, but then removed it again last week. According to Acina, Apple told it that “upon re-evaluation,” it determined this: “The app provides medical related data, health related measurements, diagnoses or treatment advice without the appropriate regulatory clearance.” Acina said it is appealing the removal.
Dr. Rachel Draelos, a physician, computer scientist and consultant in AI healthcare, says AI-powered medical apps are worrisome, particularly in dermatology. “I’m very concerned by it because properly identifying skin things is really hard,” she told Reuters. There are thousands of skin conditions, and “there’s no way that all of these apps actually have a dataset that covers all these things.”
As AI enters the operating room, reports arise of botched surgeries and misidentified body parts here
(By Steve Stecklow. Additional reporting by Hannah Woodworth. Design and graphics by John Emerson. Edited by Michael Williams.)

CS精准医

EHTH

AI-powered apps and bots are barging into medicine. Doctors have questions.