Reaping the full benefits of AI-driven applications

Two scientists at MBZUAI are working to advance an emerging approach to designing neural networks that has the potential to improve their computational power and energy efficiency.

Assistant Professor of Machine Learning Bin Gu and Assistant Professor of Machine Learning Huan Xiong are developing new and innovative concepts to make computational models known as spiking neural networks (SNNs) more practical. The two researchers will present their most recent study on spiking neural networks at the 38th Annual AAAI Conference on Artificial Intelligence, which will be held later this month in Vancouver, Canada.

“We think spiking neural networks will be the future of neural networks, certainly on mobile devices,” Gu said. “As AI-driven applications advance, we will want to run applications like GPTs that are powered by neural networks on our devices in order to reap their full benefits. The energy efficiency of SNNs have the potential to make this possible.”

An eye towards nature

Dr. Bin Gu

Neural networks are computational models that are inspired by the architecture of biological neurons found in living organisms, including humans. Neural networks are made up of interconnected groups of artificial neurons, or nodes, as they are often called. These nodes are organized into layers. An input layer takes in data, one or more so called hidden layers process the data and an output layer that delivers the output, which could be a prediction or classification, depending on the application.

Neural networks drive many artificial intelligence applications and are capable of learning and modeling complex information. They’re used for image and speech recognition, natural language processing, medical image analysis and other computer-vision related tasks like those that are involved in self-driving vehicles. Their ability to learn from the data they’re exposed to and improve over time without being programmed with specific rules makes them extremely useful.

That said, there are ways in which they can be improved.  

Spiking neural networks are an alternative to traditional neural networks. Though the concept of spiking neural networks has been around for decades, they are an area of substantial innovation and interest recently.

Dr. Huan Xiong

Compared to traditional neural networks, SNNs even more closely mimic biology in that they process information in discrete events, or 'spikes,' during which information is transmitted between nodes. This fundamental difference can offer advantages. Perhaps most notable is energy efficiency. Since the nodes of SNNs are triggered at specific moments rather than being always on, they have the potential to use significantly less energy than traditional neural networks. This is particularly appealing for applications where power consumption a concern, such as on mobile devices.

“If we want to deploy a GPT-style model on a mobile device, today the model would use a huge amount of energy, even if it was a medium size model,” Gu explained. “No one would want to use an application like that if it meant that you had to charge your device much more frequently.”

What’s more, SNNs excel in handling time-based information, a task with which traditional neural networks can struggle. They also have the potential to be much faster. Yet, despite these advantages, the adoption and development of SNNs face challenges, primarily due to the complexities involved in developing hardware to run these models and in training and implementing them.

“SNN have significant potential benefits compared to traditional neural networks, but hardware produced by the industry today supports traditional neural networks,” Gu explained.

Even though these benefits haven’t been proven in practice, Gu and Xiong believe that by advancing their innovative approaches to train SNNs, they will be able to build interest in the strengths of SNNs which will lead to greater investment in hardware that can run them.

Next-generation SNNs

A recent study by Gu and Xiong and colleagues that was presented at the 37th Conference on Neural Information Processing Systems (NeurIPS 2023) proposes a new approach to training SNNs. 

“This study is an important step in developing methods that can be used to train SNNs more efficiently,” Xiong said.

Today, there are several ways to train SNNs. One approach is to through supervised training, along with a method called back-propagation, which adjusts the network’s parameters based on the errors it produces in its output.

While back-propagation often works well in training traditional neural networks, there are challenges to this approach on SNNs, due to the on-or-off nature of SNN activation functions, which makes traditional back-propagation ineffective. Their study also mentions the use of “surrogate methods” to overcome limitations, but this approach can be resource-intensive due to the complex and the temporal nature of SNNs.

In response to these challenges, Gu, Xiong and their colleagues proposed a novel training method called Local Zeroth Order (LOCALZO) that applies zeroth-order techniques, which determines the minimum or maximum of a function, at the level of the neuron in an SNN. This approach not only provides consistency to the training process, making it more robust, but also proposes a new way of managing gradients that could lead to more energy-efficient training, they explain.

“We are the first to apply the local zeroth order technique to spiking neural networks,” Gu said. “We believe that we have provided a breakthrough contribution to the community as one of the fundamental challenges for training an SNN is how to solve the non-differentiability of the step function in a spiking neural network.”

Bhaskar Mukhoty, Velibor Bojkovic and William de Vazelhes of MBZUAI, Xiaohan Zhao of School of Artificial Intelligence, Jilin University, and Giulia De Masi of ARRC, Technology Innovation Institute, Nanjing University of Information Science and Technology and BioRobotics Institute, Sant’ Anna School of Advanced Studies contributed to the study presented at NeurIPS.

At the AAAI conference later this month, Gu, Xiong and colleagues will present more research on advancing SNNs. The study to be presented at AAAI proposes an innovative approach called Dynamic Spiking Graph Neural Networks (DySIGN). The researchers write that “the primary insight of proposed DySIGN is to thoroughly explore how to apply SNNs to dynamic graphs, and address the challenges of information loss and memory consumption by using the information compensation mechanism and implicit differentiation on the equilibrium state which is designed for classifying nodes within dynamic graphs.” 

This work not only introduces a new tool for dynamic graph analysis but also opens up new possibilities for the application of SNNs in graph-based data, the researchers write, offering a more efficient and effective way to deal with the challenges of dynamic graph processing. 

Nan Yin of MBZUAI, Mengzhu Wang of Hebei University of Technology, Zhenghan Chen of Peking University and Giulia De Masai of Technology Innovation Institute contributed to the study to be presented at AAAI.