AI not a reliable source of medical advice: UW study

The rise of artificial intelligence (AI) is coming with some risks to health in the form of misinformation.

New research out of the University of Waterloo shows that a commonly used large language model (LLM) AI service would give incorrect answers on open-ended medical questions about two-thirds of the time.

The study took 100 multiple-choice medical licensing questions, changed them to be open-ended, and fed them into ChatGPT-40. Medical students assessing the questions afterwards found that only 37 per cent of the answers were correct.

Troy Zada, a doctoral student at the University of Waterloo, warns that people need to be careful when using AI, as it can be susceptible to misinformation.

“It’s very important for people to be aware of the potential for LLMs to misinform,” said Zada in a press release. “The danger is that people trying to self-diagnose will get reassuring news and dismiss a serious problem or be told something is very bad when it’s really nothing to worry about.”

Zada was under the supervision of Dr. Sirisha Rambhatla, an assistant professor of management science and engineering at the University of Waterloo, for the study. Rambhatla is also the director of the Critical ML Lab at the University of Waterloo.

While the AI did not give incredibly wrong answers, the study concluded that LLMs like ChatGPT are not ready to give medical advice.

“Subtle inaccuracies are especially concerning,” said Rambhatla, in a press release. “Obvious mistakes are easy to identify, but nuances are key for accurate diagnosis.”

Researchers note that going to a medical professional is still the best course of action.

The entire study can be found here.