Improving patient care with computer vision

Envision a remote village in the future. A villager has pain in his abdomen, but it will take hours, maybe even days, to travel to the nearest hospital.

A neighbor, who has no formal medical training, scans his abdomen with a non-invasive device to analyze the painful area. With this simple scan, the device provides information suggesting that the man is suffering from indigestion, not appendicitis, and there is no need to make the long trek to the hospital.

“This is one practical innovation that artificial intelligence has the potential to bring about in the future,” Mohammad Yaqub, associate professor of computer vision at MBZUAI, said.

Though a device like the one Yaqub describes may be years away, today he and his team are working on more immediate applications of artificial intelligence.

In both theory and practice

Yaqub’s career has spanned industry and academia, but he has always focused his research on how AI can be used to solve problems in health care.

“Because AI is helping tackle many problems, I would love to see AI make healthcare affordable, accessible and sustainable,” Yaqub said.

At MBZUAI, Yaqub runs the BioMedical Image Analysis (BioMedIA) lab, where he and his colleagues use AI to develop solutions in disciplines including cardiology, pulmonology and oncology. The lab often collaborates with hospitals, industry and other academic labs.

During his graduate studies, Yaqub combined AI with image analysis and focused on fetal health. Using ultrasound images, this work analyzed the relationships between the baby’s “material characteristics,” such as bone development, and how these characteristics can be affected by maternal smoking, exercise and nutrition.

Because AI is helping tackle many problems, I would love to see AI make healthcare affordable, accessible and sustainable,

Yaqub’s research led to an insight that the growth of the baby’s bones, the femur in particular, is correlated to the mother’s vitamin D levels, a connection that would not likely have been discovered by human analysis of ultrasound images alone.

“A doctor can analyze a few thousand scans in his or her life. But with AI, it’s like putting the brains of many great doctors in one place and having them assess a scan together, which likely results in a better analysis than having one person read an image. It’s a supportive tool that helps improve clinical practice and decision-making strategy,” Yaqub said.

AI applications in oncology

Now Yaqub is applying his expertise in image analysis to head and neck cancer, which accounts for approximately 900,000 cases and 400,000 deaths every year across the globe, according to the Global Cancer Observatory of the World Health Organization (WHO).

When treating patients with head and neck cancer, physicians use tools called positron emission tomography (PET) and computed tomography (CT) to create images of tumors. In addition to these images, physicians also analyze patient data, such as age, gender and ethnicity. By synthesizing scans and patient information, physicians develop an assessment of the patient’s cancer. These assessments are important because they determine the kind of care a patient will receive, and accurate assessments allow physicians to develop an appropriate treatment plan that can save a patient’s life.

In Yaqub’s recent work, he and his colleagues built a “deep learning method” that simulates the way an oncologist conducts an analysis of PET and CT images and patient information, together known as “multimodal data,” a task that poses challenges for many systems. Yaqub’s model employs a “transformer-based multimodal network,” as transformers have the ability to process and synthesize disparate types of information, in this case, images and patient information.

The application of transformers to image analysis is rather new. Computer scientists have previously developed models to process head and neck cancer, but those approaches typically used separate systems to analyze the images and patient data. Yaqub’s approach crunches the multimodal data all at once.

The researchers trained their model on an established dataset and it was shown to outperform the most advanced methods in the field. The research was presented at the International Conference on Medical Image Computing and Computer-Assisted Intervention, which was held in Singapore in September and builds on earlier research his lab produced on the topic.

Though the field of medical image analysis is progressing, Yaqub noted a significant challenge that does not relate to computer science, but to access: “when it comes to healthcare, the ability for researchers to access medical data is important” and institutions should consider how to best work with researchers to share anonymized data in a responsible way.