The diagnosis game: A simulated hospital environment to measure AI agents’ diagnostic abilities

There is a genre of social media videos found on platforms like Instagram and TikTok in which medical students challenge their peers to diagnose a theoretical patient in less than a minute. It’s a way for students to put their hard-earned knowledge and diagnostic skills on display in an engaging format.

While these videos are made for entertainment and don’t recreate the exact nature of authentic conversations between physicians and patients, they illustrate an important aspect of real-life clinical interactions. Doctors often come to their conclusions about what might be ailing a patient through a process of informed questioning that repeatedly narrows the universe of potential diagnoses to smaller sets of possibilities.

A new initiative by researchers at MBZUAI recreates this back-and-forth between doctor and patient in a simulated hospital environment, using AI agents powered by large language models (LLMs) to mimic the kinds of conversations that might be heard in a typical exam room. The team’s approach provides insights into the diagnostic abilities of language models and holds the potential to be used as a training tool to help physicians diagnose patients faster and more accurately.

The team’s simulation, MedAgentSim, will be shared in an oral presentation at the 28th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) in Daejeon, South Korea. Out of nearly 3,700 papers submitted to the conference, fewer than 80 were accepted for oral presentations.

The creators of MedAgentSim are MBZUAI doctoral students Mohammad Almansoori and Komal Kumar and Assistant Professor of Computer Vision Hisham Cholakkal.

What brings you in today?

There is growing interest in the potential for AI systems to be used in medicine and the performance of language models has been improving on a variety of benchmarks that are designed to test their diagnostic abilities. That said, these measurements don’t recreate the dynamic nature of conversations between doctor and patient.

When tested on current benchmarks, models are typically given all the information about a patient at once. Sometimes the possible diagnoses are presented in multiple-choice format. This has the potential to inflate performance, making it seem that models are better than they would be if used in a hospital. “One of our motivations was to see if these systems really are good enough to be used in a clinical setting,” Cholakkal says.

MedAgentSim is a simulated hospital environment populated with three kinds of agents: doctors, patients, and evaluators that provide additional patient information like tests and imaging.

The agents are what are known as “non-playable characters” and are powered by a language model or a vision-language model. They can move around the simulated hospital, start conversations with other agents, interact with medical equipment, and make real-time decisions based on what they encounter.

How MedAgentSim works

In a simulation run by the researchers, a patient agent enters the hospital, finds its way to an exam room, and reports symptoms to a doctor agent.

After receiving the initial information, the doctor asks follow-up questions. This is a key differentiator from other evaluation methods because “real patients often don’t necessarily know what information is relevant and only provide it when asked by a doctor,” Kumar says.

The doctor can ask the patient multiple follow-up questions. If the doctor needs to know more, it can refer the patient to the measurement agent for testing or imaging.