This story originally appeared in the JC’s Israel Briefing newsletter. You can sign up to the free daily email briefing here
Artificial intelligence chatbots can identify unnecessary medical tests in most cases – but they struggle when doctors themselves disagree, an Israeli study has found.
Researchers from Ben-Gurion University of the Negev and Clalit Health Services evaluated six leading AI models across hundreds of clinical scenarios, analysing thousands of responses.
The scenarios were constructed around recommendations from the international "Choosing Wisely” campaign – a global medical initiative founded in 2012 by the American Board of Internal Medicine to eliminate what it classifies as low-value care.
The models recommended withholding medically unnecessary tests or treatments in 94.6 per cent of cases, while incorrectly advising against necessary interventions in just 0.6 per cent of cases, according to the peer-reviewed study published in US medical journal Annals of Internal Medicine.
However, performance varied in areas where medical guidance is disputed.
The models recommended unnecessary electrocardiograms in 28.3 per cent of responses involving low-risk patients and unnecessary thyroid ultrasounds in 42.8 per cent of cases involving small, incidentally discovered nodules.
Dr Dean Ariel, a researcher at Clalit, said AI systems had largely learnt from medical literature to exercise restraint, but they also reflected the uncertainty sometimes exhibited among clinicians.
"One of the hardest parts of practising medicine is knowing when to hold back," Ariel said.
"Patients find comfort in an extra test, but an unnecessary MRI often catches an asymptomatic disc bulge, setting off unnecessary worry and treatment.
"Language models learnt medicine from human clinical literature – including our less disciplined habits. We wanted to see if they also absorbed the ability to exercise restraint. For the most part, they know when to stop, stumbling primarily where we doctors ourselves hesitate."
Dr Guy Laban of Ben-Gurion University warned that different chatbots could give sharply conflicting advice on identical cases, with unnecessary testing recommendations ranging from below 10 per cent to 80 per cent.
The researchers said further work was needed to establish whether AI could maintain appropriate clinical judgement when patients insist on additional tests.
To get more Israel news, click here to sign up for our free Israel Briefing newsletter.