Researchers unveil text-guided 3D motion generation for hand-object interaction

Researchers unveil text-guided 3D motion generation for hand-object interaction

2024-06-27 01:57:17

A technology that enables precise 3D motion with simple text input, without the need for complex initial settings, has been developed by Professor Seungryul Baek and his research team in Artificial Intelligence Graduate School at UNIST.

Researchers unveil text-guided 3D motion generation for hand-object interaction
Figure 1. Given a text and a canonical object mesh as prompts, the research team generated 3D motion for hand-object interaction without requiring object trajectory and initial hand pose. They represented the right hand with a light skin color and the left hand with a dark skin color. The articulation of a box in the first row is controlled by estimating an angle for the pre-defined axis of the box. Credit: arXiv (2024). DOI: 10.48550/arxiv.2404.00562

A technology that enables precise 3D motion with simple text input, without the need for complex initial settings, has been developed by Professor Seungryul Baek and his research team in Artificial Intelligence Graduate School at UNIST.

Known as Text2HOI, this allows for interaction between hands and objects based on text input into the prompt window, paving the way for advancements in the commercialization of the 3D field.

Text2HOI is capable of executing actions such as holding and interacting with objects through text commands. Its applications are vast, spanning across various industries, including virtual reality (VR), robotics, and , offering easy usability due to its streamlined setup process.

By analyzing the user-input text, Text2HOI predicts the contact points related to hand motions in response to the command object. For instance, a command to "Hand over an apple with both hands" will probabilistically calculate the likely contact point between the hands and the apple.

This enables precise when picking up the apple, adjusting hand position and angle based on the apple's size and shape.

The versatility of this technology allows for its integration into diverse sectors, from simulating to controlling character behavior in games and virtual reality, as well as facilitating complex scientific experiments virtually. In the realm of robotics, Text2HOI opens up possibilities for natural interactions with robots through accurate hand motion control.

Researchers unveil text-guided 3D motion generation for hand-object interaction
Figure 2. The research team demonstrated the generated hand-object motions and the predicted contact map results. It shows the results with objects seen during training. Credit: arXiv (2024). DOI: 10.48550/arxiv.2404.00562

Professor Baek expressed his optimism about the broad for Text2HOI, noting its potential for advancing virtual reality (VR/AR), robotics, and medical fields. He emphasized a commitment to continuing research that benefits society.

First author Junuk Cha highlighted the potential for Text2HOI to serve as a fundamental tool in linking text prompts with interactive hand and object motions, thereby promoting further research in this area.

The findings of this research will be published in the online version of the Conference on Computer Vision and Pattern Recognition and are currently available on the arXiv preprint server.

More information: Junuk Cha et al, Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction, arXiv (2024). DOI: 10.48550/arxiv.2404.00562

Journal information: arXiv

Citation: Researchers unveil text-guided 3D motion generation for hand-object interaction (2024, June 26) retrieved 26 June 2024 from https://techxplore.com/news/2024-06-unveil-text-3d-motion-generation.html

This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.