Overcoming the curse of dimensionality

How would a computer, based on satellite data, interpret a given spot in the Middle East? Would it characterize the region based upon the vast, uninhabited Empty Quarter or would it instead focus on densely populated urban areas such as Dubai and Riyadh? The same question could be asked of Ontario, Canada’s most populous province, yet one that is also vast and empty across much of its more than one million square kilometers.Data is quite literally all around us – from the global positioning systems that we use to navigate to the ultra-high definition cameras we use to capture our world. How we grapple with collecting and using this data is central to how we understand our world. And it is how we reduce the data we collect, down into its essential components, that has become a foundational challenge in machine learning known as dimensionality reduction.

Dimensionality reduction is an approach to extracting the parts of a dataset that are useful, while also ensuring that enough data is present to generate meaningful results from it. The set of conditions that occur along the way is often referred to as the “curse of dimensionality,” a phrase coined by the mathematician Richard Bellman to explain at length how growth in a dataset presents a massive challenge to the use of that data.

Elements of dimensionality reduction and manifold learning

Recently, a group of experts worked across the vastly different spaces of the Middle East and Canada to quite literally write the book on dimensionality reduction. MBZUAI Professor of Machine Learning and former Provost Fakhri Karray worked with co-authors Benyamin Ghojogh (former Ph.D. candidate of Karray), Mark Crowley (Professor of Electrical and Computer Engineering at the University of Waterloo), and Ali Ghodsi (Professor of Computer Science at the University of Waterloo), to create “Elements of Dimensionality Reduction and Manifold Learning,” a textbook and resource that is both useful and informative for students, while also supportive of the work that researchers, postdocs, and faculty do in various areas of machine learning and artificial intelligence.

The text, which was published by Springer Nature Switzerland, breaks out into four parts, within which are 21 individual chapters dedicated to elucidating spectral, probabilistic, and neural network-based dimensionality reduction. 

“It has highly technical and powerful material, while also attempting to illustrate core concepts in dimensionality reduction, manifold learning, and more,” Karray added. “Additionally, the book is highly referenced, so there are also many directions of inquiry you can go using our book as a jumping off point — we have gone to great lengths, in fact, to reference seminal books and publications across various areas of machine learning.”

A history of dimensionality reduction research

The book opens with four introductory chapters that address foundational subjects including a history of dimensionality reduction, as well as chapters on linear algebra, kernels, and optimization. The book also includes extensive references that span from 1940 to the present, laying out much of the canon of the discipline.

At the end of the day, we’re talking about how to make data more useful and more usable. These are things that will touch all of our lives.

“With the explosion of interest and advances in machine learning, there has been a corresponding increased need for educational and reference books to explain various aspects of machine learning,” the book’s introduction reads. With this premise in mind, the authors have created a text which “delves into basic concepts and recent developments… providing the reader with a comprehensive understanding,” of dimensionality reduction and manifold learning.

It is an ambitious task in such a rapidly changing field such as AI. But as machine learning progresses, it is still essential, according to Karray, that foundational texts be produced and made available, so that students benefit from the latest in the field, and so that practitioners may have a point of reference, from which to anchor rigorous academic inquiry.

“At the end of the day, we’re talking about how to make data more useful and more usable,” Karray said. “These are things that will touch all of our lives, and producing a seminal reference work in this space has been a labor of great interest and love. It has also been several years of intense work, and for that I have to thank my co-authors and editors, who made the book possible. Without everyone’s dedication, this book doesn’t happen — so I’m extremely grateful and humbled, and I know academics, students, and industry professionals alike will benefit.”