Emulating the energy efficiency of the brain

The brain and its biological structures have long been a source of inspiration for computer scientists. Some early research in the discipline focused on developing machines that were modeled on the way networks of neurons in the brain process information. This resulted in what are known as artificial neural networks, which today are used in deep learning for many different applications, ranging from video processing, to translation, to image generation.

The basic composition of artificial neural networks include nodes — the “neurons” of a neural network —and layers. Nodes receive information and produce outputs. The nodes are organized into series of layers, including an input layer that initially receives information, an output layer that is responsible for generating the network’s response, and hidden layers, which do the bulk of the processing and can number in the thousands for the largest neural networks.

Is it the complexity of these deep learning networks that allows them to model intricate patterns from large amounts of data.

There’s a cost to everything

Deep learning models have had a bit of a moment over the past year. Since the launch of OpenAI’s GPT-4 model in March, the company’s chat bot has been the subject of countless stories in the press. It is powered by deep learning models that process human language with the goal of generating responses to user questions in understandable human language. Yet while ChatGPT often produces impressive results, training and running it consumes a massive amount of energy.

Dr. Bin Gu and Dr. Huan Xiong – both Assistant Professors of Machine Learning at MBZUAI – are working to develop more efficient technologies with an approach called spiking neural networks. These are similar to artificial neural networks but are designed to function even more like the neuronal networks of the human brain. Their approach may lead to significant savings in terms of energy.

“One query to ChatGPT uses the equivalent of 3.96 watts, or a third of the battery capacity of an iPhone 13 Pro,” said Gu. “That is a large consumption of energy for one question.”

While a human brain is of course limited in its capacity to master the huge breadth of knowledge that a large-language model like ChatGPT can cover, millions of years of evolution have produced a biological neural network that is extremely energy efficient.

In a whole day, an adult human brain consumes slightly more than a third of a watt of energy, or approximately 8% the energy of one question posed to a large-language model, Gu explained. “We want to know how the human brain consumes so little energy compared to artificial neural networks and how we can build neural networks that mimic the human brain and are more efficient,” he said.

On and off again

Spiking neural networks, or SNNs, are neural networks that attempt to emulate the behavior of biological neurons more closely than traditional artificial neural networks, with the goal of increasing efficiency.

The concept of spiking neural networks has been around for decades. Researchers in the 1950s first proposed the idea. They are like artificial neural networks, but with key differences. “You will find that the architecture of both spiking neural networks and artificial neural networks are similar, but the inputs and the activation function are different,” Gu said.

Instead of transmitting information continually, as is the case with traditional artificial neural networks, SNNs use discrete events, called spikes, during which information is sent from one neuron to another. This way of processing information is closer to the way the human brain works. As in the brain, there is what’s known as a refractory period, a moment of downtime, after a node has fired during which it doesn’t fire again. Because nodes in SSNs aren’t continuously processing information, they have the potential to be more energy efficient than traditional artificial neural networks.

“Currently we are focused on theoretical study to understand the architecture and algorithms that are used in SNNs and to discover some new algorithms that will make SNNs more efficient,” said Huan Xiong, Gu’s collaborator on the initiative.

Looking to the future, the scientists believe that there are many applications which could benefit from advanced spiking neural networks.

“One application could be a long-range search and rescue drone that is powered by the sun and needs to be extremely energy efficient,” Gu said. There are also potential applications in healthcare as well. Since spiking neural networks process information more like the human brain, they may be better suited in the development of prosthetics that require the use of machine learning.