Peering into humanity through music

A murmuration of starlings dart, twist, chirp, ascend, and accelerate together, in what looks, from afar, like an organism at war with itself. It is both together, and yet, not in sync. Schools of fish are much the same, and for good reason—dazzling displays confuse and bewilder predators, and they help communities of species thrive. Working together, organisms can create safety, choose ideal nesting spaces, and quite a bit else.Humans can also exhibit some of these behaviors. In ensembles, musicians follow and lead simultaneously. They adjust to each other’s tempo, volume, and mood changes. And collectively they create a singular experience for listeners that is unique every time it is performed.

My dream is to let Beethoven develop my Arabic motif, in a moonlight sonata style, and then let Mr. Miles Davis interpret that using a flute.

The same could be said for instances of human-computer interaction—a field of study that delves into the sometimes mundane, and often unique experiences involving humans and computers. Computers bring order and volumes of information. Humans bring intuition, unpredictability, and creativity.

So, whether the interaction is asking Google for directions, or querying an expanse of data on a supercomputer, there is a uniqueness to the interaction between man and machine each time it is undertaken.

Human-computer interaction is different than human-human interaction, particularly when it comes to abstract activities such as music. Understanding why these interactions are different is the subject of Gus Xia’s research. Xia is a visiting assistant professor of machine learning at MBZUAI and a global network assistant professor in computer science at NYU, Shanghai.

Xia studies music, but in truth, his work is about much more. Xia uses his love and appreciation for the complexity of music to study how AI might be made to act in more natural human ways when presented with high context activities, music being just one example.

“Music is highly abstract,” Xia said. “You don’t have to very rigidly associate your notes with the physical world. It is purely about leveraging your imagination and that is why I think music is the perfect way to study creative intelligence.”

Xia’s work involves analyzing and creating music with computers, a frustrating and often comical way to probe the inner workings of both human and computer “minds.”

“Machines don’t perceive music exactly, I would say they have a shadow perception,” Xia said. “A machine can produce a spectrogram, and from there it can extract high-level concepts. By contrast, human ears, perceive music in something similar to a spectrogram, but as humans we also perceive many other high-level concepts simultaneously or nearly simultaneously.”

 

Machines, according to Xia, stay at a low level when it comes to musical concepts. And this, of course, relates to many things that humans and computers do. At the signal level, computers have sharper ears than humans, but they don’t have the “musical mind,” according to Xia, to extract all of the higher-level concepts needed to produce beautiful, layered musical compositions.

Xia is himself a professional musician and an innovator in how humans can harness the unique capabilities of computers to learn music faster, and to compose and perform pieces of music.

“I am a professional performer, but thankfully, I am a novice at composition,” Xia said. And it’s this novice’s approach to composition that he credits for helping fuel his interest in teaching computers how to compose at a high level of proficiency and taste.

“My dream is to let Beethoven develop my Arabic motif, in a moonlight sonata style, and then let Mr. Miles Davis interpret that using a flute.”

 

 

About Gus Xia

Xia’s research is very interdisciplinary. He is broadly interested in the design of interactive intelligent systems to extend human musical creation and expression. This research lies in the intersection of machine learning, HCI, robotics, and computer music. Some representative works include interactive composition via style transfer, human-computer interactive performances, autonomous dancing robots, large-scale content-based music retrieval, haptic guidance for flute tutoring, and bio-music computing using slime mold.

Xia is a Global Network Assistant Professor in Computer Science at New York University, Shanghai. He also holds affiliations at Tandon, CILVR at the Center for Data Science, and MARL at Steinhardt. He received his Ph.D. in the machine learning department at Carnegie Mellon University (CMU) in 2016, and he was a Neukom Fellow at Dartmouth from 2016 to 2017.

Xia is also a professional Di and Xiao (Chinese flute and vertical flute) player. He plays as a soloist in the NYU Shanghai Jazz ensemble, Pitt Carpathian Ensemble, and Chinese Music Institute of Peking University.