You've likely met someone who identifies as a visual or auditory learner, but others absorb knowledge through a different modality: touch. Being able to understand tactile interactions is especially important for tasks such as learning delicate surgeries and playing musical instruments, but unlike video and audio, touch is difficult to record and transfer.

To tap into this challenge, researchers from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) and elsewhere developed an embroidered smart glove that can capture, reproduce, and relay touch-based instructions.

To complement the wearable device, the team also developed a simple machine-learning agent that adapts to how different users react to tactile feedback, optimizing their experience. The new system could potentially help teach people physical skills, improve responsive robot teleoperation, and assist with training in virtual reality.

An open-access paper describing the work was published in Nature Communications on Jan. 29.

Will I be able to play the piano?

To create their smart glove, the researchers used a digital embroidery machine to seamlessly embed tactile sensors and haptic actuators (a device that provides touch-based feedback) into textiles. This technology is present in smartphones, where haptic responses are triggered by tapping on the touch screen.

For example, if you press down on an iPhone app, you'll feel a slight vibration coming from that specific part of your screen. In the same way, the new adaptive wearable sends feedback to different parts of your hand to indicate optimal motions to execute different skills.

The smart glove could teach users how to play the piano, for instance. In a demonstration, an expert was tasked with recording a simple tune over a section of keys, using the smart glove to capture the sequence by which they pressed their fingers to the keyboard. Then, a machine-learning agent converted that sequence into haptic feedback, which was then fed into the students' gloves to follow instructions.

With their hands hovering over that same section, actuators vibrated on the fingers corresponding to the keys below. The pipeline optimizes these directions for each user, accounting for the subjective nature of touch interactions.

"Humans engage in a wide variety of tasks by constantly interacting with the world around them," says Yiyue Luo MS '20, lead author of the paper, Ph.D. student in MIT's Department of Electrical Engineering and Computer Science (EECS), and CSAIL affiliate. "We don't usually share these physical interactions with others. Instead, we often learn by observing their movements, like with piano-playing and dance routines."

"The main challenge in relaying tactile interactions is that everyone perceives haptic feedback differently," adds Luo. "This roadblock inspired us to develop a machine-learning agent that learns to generate adaptive haptics for individuals' gloves, introducing them to a more hands-on approach to learning optimal motion."

The wearable system is customized to fit the specifications of a user's hand via a digital fabrication method. A computer produces a cutout based on individuals' hand measurements; then, an embroidery machine stitches the sensors and haptics in. Within 10 minutes, the soft, fabric-based wearable is ready to wear. Initially trained on 12 users' haptic responses, its adaptive machine-learning model only needs 15 seconds of new user data to personalize feedback.

More information: Yiyue Luo et al, Adaptive tactile interaction transfer via digitally embroidered smart gloves, Nature Communications (2024). DOI: 10.1038/s41467-024-45059-8

This story is republished courtesy of MIT News (web.mit.edu/newsoffice/), a popular site that covers news about MIT research, innovation and teaching.