Foundations of Multisensory Artificial Intelligence

Building multisensory AI systems that can learn from many sensory inputs such as text, speech, audio, video, real-world sensors, wearable devices, and medical data holds great promise for impact in many scientific areas with practical benefits, such as in supporting human health and well-being, enabling multimedia content processing, and enhancing real-world autonomous agents.

In this talk, I will discuss my research on the machine learning foundations of multisensory intelligence, as well as practical methods in building multisensory foundation models for many modalities and tasks. In the first half, I will present a new theoretical framework formalizing how modalities interact with each other to give rise to new information for a task. These interactions are the basic building blocks in all multimodal problems, and their quantification enables users to understand their multimodal datasets and design principled approaches to learn these interactions. In the second part, I will present cross-modal attention and multimodal transformer architectures which now underpin many of today’s multimodal foundation models. Finally, I will discuss our collaborative efforts in applying multisensory AI for real-world impact: (1) aiding mental health practitioners by predicting daily mood fluctuations in patients using multimodal smartphone data, (2) supporting doctors in cancer prognosis using pathology images and multiomics data, and (3) enabling robust control of physical robots using cameras and touch sensors.