Tools of the trade: teaching robots to learn manual skills

The success of today’s large language models such as ChatGPT has been built on decades–worth of accumulated internet–based data and knowledge – equipping these models with vast stores of data with which to learn complex linguistic patterns and generate coherent and convincing responses. 

But while artificial intelligence has made huge strides in language – and increasingly, image – generation, machines have been left behind in many ways. With no comparable databases of human motor skills to learn from, robots working in the physical world have struggled to learn basic physical tasks that most humans master in early childhood, such as turning a door handle, or tightening a bolt. 

A new framework developed by Sami Haddadin, Vice President of Research and Professor of Robotics at MBZUAI, aims to close this gap – teaching robots to learn through touch, trial and error, and transforming their potential as truly hands–on partners in industry, manual labor, and everyday life. 

His approach, called Tactile Skills, was developed by Haddadin alongside his Ph.D. students Lars Johannsmeier and Samuel Schneider, as well as Etienne Burdet from Imperial College London, and Yanan Li from the University of Sussex. Their work was recently published in Nature Machine Intelligence. 

“This is an important step towards giving robots the ability to rapidly learn at extremely low energy and compute demands to execute manipulation skills at a performance matching industrial levels,” says Haddadin, who explains that the broader vision is to create a scalable, robotic skill platform – a ‘robotic handyman’ capable of performing household maintenance, industrial tasks, and more.  

“Over time, this foundation can evolve to support more advanced roles such as medical or rehabilitation assistants,” he adds. “This scalability could help solve labor shortages in trades and manufacturing, especially in regions such as the U.S. where skilled labor is in decline.” 

Prioritizing practicality 

According to Haddadin, current robotics research often focuses on advanced, acrobatic demonstrations which, while visually impressive, do not solve the basic manipulation problems that define many human jobs–tasks such as grasping, using tools, or adjusting to unexpected changes in the environment.  

This is where Tactile Skills seeks to make a difference, using documented skill knowledge to learn, understand and respond to physical interactions in real time.  

“Many physical skills cannot be learned through observation alone,” Haddadin says. “Watching someone play the guitar is not the same as learning how to play it. You have to do it to understand it. 

“The challenge for us was to take this type of human learning and apply it to machines. True human-level physical interaction involves nuanced control, adaptability, and feedback from the environment. To replicate this, we used knowledge from centuries of vocational training – manual skills that have been documented and refined over time in standardized curricula, particularly in industrial contexts such as German engineering.” 

By giving robots the information that they need, they can learn the theory of a particular task. But the real learning comes in the ‘doing’. The team’s framework addresses this by incorporating feedback from the environment – force, energy, motion – into the way robots learn. They learn through real-time interaction, adjusting the grip of a handshake, for example, based on what they physically experience. This mirrors the way humans often learn, using trial and error during direct engagement. 

Extensive testing on 28 distinct industrial tasks including plug insertion and precision cutting yielded an almost 100% success rate. 

Another key component of the research is an algorithm called GGTwReP, which Haddadin says significantly outperforms standard deep reinforcement learning systems. Unlike algorithms that depend on high–powered GPUs and long training times, GGTwReP runs on standard CPUs, trains in minutes, and consumes far less energy. 

Tests have shown it to be faster, more reliable, and up to 1,000 times more energy–efficient than common reinforcement learning methods such as state of the art algorithms. This makes it not only more efficient, but also more scalable for real–world applications where energy usage and compute infrastructure are limiting factors. 

Assisting, not replacing 

Beyond the technical performance of Tactile Skills, Haddadin argues that the framework could also help preserve and transfer human vocational skills at a time when many trades are facing labor shortages and generational decline. 

“Many skills are disappearing as older workers retire and fewer young people enter trades,” he says. “By capturing how skilled professionals perform these tasks and encoding them into robotic systems, we have a way to retain and share this knowledge.” 

He sees particular relevance in contexts such as industrial maintenance, household repair, and medical assistance, where robots could eventually support professionals or take on tasks where human workers are in short supply.  

Key to this is the intention for machines to assist rather than replace humans. 

“Even if you wanted to, replacing a human worker in the real world would involve far more than making a robot that can move,” he says. “You would need to build the entire infrastructure around it, provide power, maintenance, and training. And even then, the robot’s performance is nowhere near that of a skilled human in most practical scenarios. 

“We see robots as potential assistants, not replacements – tools that can help humans become more productive, or take on tasks that are dangerous, repetitive, or undesirable.” 

The Tactile Skills framework has already entered pilot use in industrial environments in Europe and China, and further development is planned in order to expand the range of tasks it can support and to evaluate its long–term reliability in production settings. 

If widely adopted, the approach could lead to a new category of robots – machines that can work with humans in everyday environments, guided by the same kinds of hands––on experience that underpin human expertise. 

“The aim is not just to teach robots to perform,” concludes Haddadin, “but to give them the ability to learn from doing.”