AI Oncology Tool Uses Medical Imaging and Genetic Information to Support Clinical Decisions

Making accurate clinical decisions in oncology is a complex process that requires the integration of diverse data sources — from imaging and genetic information to patient histories and treatment protocols. To effectively support oncologists, artificial intelligence (AI) systems must be able to process multimodal data and demonstrate human-like reasoning. To address this need, researchers have developed an enhanced AI agent built on GPT-4, equipped with specialized tools and access to authoritative medical knowledge to guide precision medicine in oncology.

This autonomous AI agent was created by researchers at the EKFZ for Digital Health at TU Dresden (Dresden, Germany), in collaboration with partners from Germany, the UK, and the USA, to provide a clinical support system capable of real-time decision-making. To achieve this, they augmented the GPT-4 large language model with a suite of digital tools, including radiology report generation from MRI and CT scans, medical image interpretation, prediction of genetic mutations from histopathology slides, and the ability to search platforms like PubMed, Google, and OncoKB. The model was also given access to approximately 6,800 documents, including official oncology guidelines and clinical reference materials, to ground its recommendations in evidence-based medicine.

The AI agent followed a two-step evaluation protocol in a set of 20 simulated, real-world patient cases. First, it selected the appropriate tools for the task, then used retrieved medical data to guide its reasoning and output. Human medical experts reviewed the system’s conclusions for accuracy, completeness, and proper citation of sources. The AI reached the correct clinical conclusion in 91% of cases and accurately cited oncology guidelines in over 75% of its responses. The inclusion of specialized tools and search functions significantly enhanced the model’s performance, dramatically reducing the frequency of “hallucinations” — inaccurate but plausible-sounding outputs, which is critical in high-stakes clinical settings.

These results serve as proof of concept that AI agents can effectively support oncologists in clinical workflows. While the findings are promising, the researchers noted that the system was only tested on a small sample of cases. Further validation will be needed to confirm its robustness. Next steps include developing “human-in-the-loop” conversational capabilities and deploying the tool on local servers to ensure data privacy. In the long term, the team sees potential for similar AI agents in other areas of medicine, provided they are equipped with domain-specific tools and datasets. This work underscores the significant opportunity for large language models to advance precision oncology through integration with clinical tools and up-to-date medical literature.

“AI tools are designed to support medical professionals, freeing up valuable time for patient care,” said Dyke Ferber, first author of the publication. “They could help in daily decision-making processes and support doctors to stay updated on the latest treatment recommendations, contributing to the identification of optimal personalized care for cancer patients.”

Related Links:
EKFZ for Digital Health