A new way of seeing: vision transformers for radar data

Yahia Dalbah, a recent master’s degree graduate of MBZUAI, and colleagues from the university, authored a study that was recently presented at the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), which was held this month in Honolulu, Hawaii. The study proposes new ways in which radar could be used to identify objects in the environment and has potential applications in the quickly growing field of autonomous vehicles, among other uses.

Jean Lahoud, a research associate in computer vision at MBZUAI, and Hisham Cholakkal, assistant professor of computer vision at MBZUAI, contributed to the study.

Old meets new

Radar, an acronym for radio detection and ranging, was developed decades ago, and found its first applications in militaries of the early 20th century. As its name suggests, radar employs radio waves and can detect the presence, direction, distance and speed of objects. It works by sending out radio waves and analyzing the reflections that bounce back. And while this technology has been in use for well over a century, it is extremely effective, and is still widely used in many sectors including aviation, shipping and meteorology.

Yahia Dalbah

That said, radar has limitations: “Radar can’t necessarily tell you what something is, but it can tell you that it’s there, which is perfectly fine for most operations,” Dalbah explained. “If you are an airplane or a car on cruise control, you have to stop if an object is in front of you no matter what the object is.”

Dalbah and his colleagues were interested to learn how radar could be used in combination with other technologies to identify objects in the environment. “Fusing radar with other information for object identification hasn’t been widely explored,” Dalbah said. “It’s a new field and it’s interesting because there hasn’t been a lot of work done on this topic.”

Object identification describes the capacity for a machine to identify and classify objects within images or videos. This process involves algorithms that process visual data, identify specific features and categorize these features as distinct objects. It’s a critical component of many applications like self-driving cars and facial recognition systems. Object identification typically employs machine learning, by which systems are trained on large datasets of images in the effort to give them the ability to autonomously recognize and classify objects in a variety of conditions.

Dalbah and his colleagues’ most recent work on the topic, which was shared at WACV, is “TransRadar: Adaptive-Directional Transformer for Real-Time Multi-Vide Radar Semantic Segmentation.”

Dalbah explained that today cameras in combination with deep learning are the best tools for object recognition, but cameras are highly sensitive to weather conditions. Another option is lidar — or laser imaging, detection and ranging — which uses light to gather three-dimensional information about the environment and is used in self-driving cars. Lidar has a short range, however — only about 200 meters.

Dalbah and the team’s approach is a step towards melding data from radar, lidar and cameras. “The idea is that when one of them is offline, say cameras in bad weather, another one steps up, and they can concurrently correct each other and learn together to improve their performance,” he said.

It’s a challenge to make sense of these different kinds of sensor information, a technique known as sensor fusion. Cameras may shoot at 60 frames per second, while lidar samples the environment 150 times per second and radar 1,000 times per second. These sensors also may view an environment from different angles, or from different aspect ratios, which describes the relationship between the height and the width of an image. These variables need to be standardized so that the information from the different kinds of sensors can be complementary instead of confounding.

Transforming radar

The team’s key innovation relates to applying a technique known as a transformer to process the radar data. A transformer is a type of neural network that uses a self-attention mechanism and has the capacity focus on important components of an image while also making sense of the image as a whole. “Self-attention is a way of correlating what is happening in different parts of the image with each other,” Dalbah said. This makes it possible for transformer-driven applications to identify relationships between objects that might be physically distant from each other and might not be identified by other approaches.

Previous studies using attention mechanisms proposed that the data be sampled along specific axes, such as width or height. Dalbah and the team’s approach, which they describe as “adaptive-directional attention,” allows the attention mechanism more flexibility, to analyze across axes, providing the model a more comprehensive view of what is represented in the data, Dalbah explained.

Dalbah and his colleagues also developed a new way to optimize the model’s interpretation of the radar data, a difficult task because radar has a lot of information that is noise, or irrelevant, and a small percentage of radar data actually relates to the object of interest. They note in the paper that the more than 99% of the total pixels they processed in their data set were background, meaning less than 1% related to the objects their model was trying to detect.

Their approach proved to be successful. Using two standard radar data sets, their model achieved “state-of-the-art performance” in semantic segmentation and improved performance in object detection, they wrote. They hope to continue their efforts and fuse radar with color images from cameras to “produce more robust predictions,” which may set the new standard for how vehicles gather information about the world around them, they wrote.

Dalbah himself intends to continue his studies in a doctoral program and is interested to explore how data fusion, including data from radars, could be used in applications such as mapping or indoor navigation.