Supervillains bent on world domination with the help of robot hordes must be feeling pretty impatient these days. Robots that do backflips, jumps, and choreographed dances under carefully controlled conditions are not going to force the people of Earth into submission, but that is about as good as modern robot tech gets. Where are the robots science fiction has been promising us for decades, like the 6502 CPU-powered T-800s from The Terminator? Now that would get us silly humans waving our white flags.
World domination aside, there are good reasons to develop more capable robots. They might, for example, take care of our chores around the home in the future so we can spend more time doing things that we don’t hate. But that is easier said than done. Robots have a very difficult time navigating, and interacting with, the types of unstructured environments that are found in the real world. In order to become more useful in the world of humans, robots will need to become more like humans.
An overview of the approach (📷: N. Gu et al.)
A good starting point for building such a robot would be to give it more human-like abilities in sensing its environment. Robots commonly rely on computer vision alone to capture information about the world around them, but that leaves out the very rich information humans gather from their other senses, like touch. In an effort to close this gap, a group of researchers at Tohoku University and the University of Hong Kong has developed a control system that leverages both sight and touch.
The system, named TactileAloha, is an extension of ALOHA (A Low-cost Open-source Hardware System for Bimanual Teleoperation), a dual-arm robotic platform developed by Stanford University. While ALOHA gave researchers an open-source playground for robotic teleoperation and imitation learning, it relied primarily on cameras. TactileAloha adds an extra dimension: a tactile sensor mounted on the gripper. This upgrade gives the robot the ability to recognize textures, distinguish the orientation of objects, and adjust its manipulation strategies accordingly.
The researchers used a pre-trained ResNet model to process the tactile signals and then merged them with visual and proprioceptive data. The combined sensory stream was fed into a transformer-based network that predicted future actions in small chunks. To make execution smoother, the team introduced weighted loss functions during training and a temporal ensembling strategy during deployment.
The team put the system to the test with two challenging tasks: fastening Velcro and inserting zip ties. Both require fine-grained tactile sensing to succeed. Compared to state-of-the-art systems that also included some tactile input, TactileAloha improved performance by about 11%. Moreover, it adapted its actions dynamically based on what it felt, not just what it saw, which is a crucial step toward human-like dexterity.
While we are still a long way from robots that can fold laundry without making a mess, or whip up dinner without burning the house down, adding touch to their toolkit is a meaningful step. By combining vision and tactile sensing, robots gain a deeper understanding of the physical world and can handle tasks that confuse purely vision-based systems.
Supervillains may need to wait a bit longer for their robot armies, but for the rest of us, this research points toward a future where robots could finally lend a helping hand.
