Metaverse healthcare in red, green, and blue

In the early months of the COVID-19 pandemic a team of researchers set out to solve a simple, yet urgent problem: how to measure a patient’s health using only a humble webcam. The team set out to measure three vital signs — heart rate, respiratory rate, and blood oxygen level. The goal was to triage people quickly from afar, and refer only those patients showing the telltale signs of COVID-19 infection present in the over 765 million confirmed cases and over 6.9 million deaths as of May, 2023, according to the WHO.

“We wanted to help frontline medical staff triage patients at home using the lowest common sensors found everywhere — the webcams on our computers and smart devices,” Professor and Acting Chair of the MBZUAI Computer Vision Department Abdulmotaleb El Saddik said.

Although the WHO Director-General announced in May 2023 that COVID-19 was no longer a public health emergency of international concern, the need for effective telemedicine and tele-triage has gone undiminished. One of the United Nation’s Sustainable Development Goals (#3), for example, is the establishment of good health and wellbeing. Over 1.6 billion people live in parts of the world that have a “weak national capacity to deliver basic health services,” according to the UN, something El Saddik and his co-authors hope to help address.

In their research, El Saddik et al. turned to a method of gathering vital signs with an unwieldy name – Photoplethysmography or PPG. You might know it as the technology behind the pulse oximeter that a nurse places on your finger, or the technology that your Apple Watch uses to monitor your vital signs.

Digital twins and non-invasive monitoring

El Saddik et al. demonstrated in their paper titled: “Towards a Machine Learning-Based Digital Twin for Non-Invasive Human Bio-Signal Fusion,” that photoplethysmography can also be deployed using a webcam to stream real-time data about the health status of real humans. Their big contribution, as AI researchers, was to build a digital twin (DT) system using this data, and then adapting machine learning approaches to combine multiple vital sign measures into one useful output for triaging and monitoring human health.

DT systems are not new, in fact, they are credited to Alrick B. Hertzman’s research from 1937, which sought to create virtual representations of manufactured items to better undertake lifecycle management. Since Hertzman’s initial research the concept has been used in a wide variety of sectors — in space exploration, car manufacturing, environmental assessment, infrastructure development, and, as is the case with El Saddik and his collaborators, in healthcare as well.

The authors harnessed the ability of the common webcam to detect tiny color intensity changes caused by changes in facial blood flow. The team captured webcam data and then analyzed red, green, and blue channels in such a way as to measure vital signs effectively. The result was a system that displayed real-time vital sign information related to heart rate, breathing rate, and blood oxygen saturation.

The implications of such a system are far reaching. One can imagine an emergency room where patients are triaged as they walk through the door to save vital time and lives. Remote healthcare for the billion or more people around the world that do not have access to care providers is another important option.

El Saddik and his co-authors also propose that the monitoring of a range of patients such as the elderly, the incapacitated, or for people of determination, are all important possible applications.

About Abdulmotaleb El Saddik

Before joining MBZUAI, El Saddik served as a distinguished university professor and university research chair in the School of Electrical Engineering and Computer Science at the University of Ottawa. He was the director of the Ottawa-Carleton Institute for Electrical and Computer Engineering (OCIECE) and the director of the Medical Devices Innovation Institute (MDII) and Director of the Information Technology Cluster, Ontario Research Network on Electronic Commerce (ORNEC).

El Saddik is a fellow of the Royal Society of Canada, and a fellow of IEEE, a fellow of the Canadian Academy of Engineering and a fellow of the Engineering Institute of Canada. He is an ACM distinguished scientist and has received several awards, including the Friedrich Wilhelm Bessel Award from the German Humboldt Foundation, the IEEE Instrumentation and Measurement Society Technical Achievement Award.