AI Tool Provides Health Advice On Par with Clinicians

A series of tests carried out on an artificial intelligence (AI) tool has demonstrated its ability to provide heath advice, which is on par with practicing clinicians. The AI tool has been developed by Babylon Health (London, UK), which uses a combination of cutting-edge technology and the best available medical expertise to deliver 24-hours-a-day, 7-days-a-week access to digital health tools (including health assessment, triage and health tools), to people across the world.

The AI technology was tested (including the relevant sections of the MRCGP exam) in a transatlantic collaboration between Babylon and the Royal College of Physicians, Stanford Primary Care and Yale New Haven Health. The MRCGP exam is the final test for trainee General Practitioners (GPs), set by the Royal College of General Practitioners (RCGP). Trainee GPs who pass this assessment have demonstrated their competence and clinical skills to a level that is sufficiently high enough for them to undertake independent practice. A key part of this exam tests a doctor's ability to diagnose.


Babylon took a representative sample-set of questions testing diagnostic skills from publicly available RCGP sources, as well as independently published examination preparation materials, and mapped these to the current RCGP curriculum in order to ensure the questions resembled actual MRCGP questions as closely as possible. The average pass mark over the past five years for real-life doctors was 72%, while Babylon's AI scored 81% in sitting the exam for the first time.

Babylon next collaborated with the Royal College of Physicians, Stanford Primary Care and Yale New Haven Health to further test the AI's capabilities alongside seven highly experienced primary care doctors using 100 independently-devised symptom sets (or 'vignettes'). Babylon's AI scored 80% for accuracy, while the seven doctors achieved an accuracy range of 64-94%. The accuracy of the AI was 98% when assessed against conditions seen most frequently in primary care medicine. In comparison, when Babylon's research team assessed experienced clinicians using the same measure, their accuracy ranged from 52-99%. More importantly, the safety of the AI was 97%, which compares favorably to the average of 93.1% for the doctors.

As the AI is continues to learn and accumulate knowledge, Babylon expects subsequent testing to produce significant improvements in terms of results. Babylon's technology provides health information, rather than a medical diagnosis, for regulatory reasons. The tests carried out relate to the diagnostic exams taken by doctors as a benchmark for accuracy, however, Babylon's AI service remains an information service, rather than a medical diagnosis.

“The World Health Organization estimates that there is a shortage of over 5 million doctors globally, leaving more than half the world's population without access to even the most basic healthcare services. Even in the richest nations, primary care is becoming increasingly unaffordable and inconvenient, often with waiting times that make it not readily accessible. Babylon's latest AI capabilities show that it is possible for anyone, irrespective of their geography, wealth or circumstances, to have free access to health advice that is on-par with top-rated practicing clinicians,” said Dr. Ali Parsa, Babylon's Founder and CEO. “Tonight's results clearly illustrate how AI-augmented health services can reduce the burden on healthcare systems around the world. Our mission is to put accessible and affordable health services into the hands of every person on Earth. These landmark results take humanity a significant step closer to achieving a world where no-one is denied safe and accurate health advice.”

Related Links:
Babylon Health