Researchers have created an innovative system for generating personalized storybooks. This system utilizes generative artificial intelligence and home IoT technology to assist children in language learning. The findings are published in the journal Proceedings of the CHI Conference on Human Factors in Computing Systems.

Their research was showcased at the ACM CHI (ACM SIGCHI Conference on Human Factors in Computing Systems), where it earned an "Honorable Mention Award," recognizing it as one of the top 5% of submissions.

Children's language development is crucial as it impacts their cognitive and academic growth, their interactions with peers, and overall social development. It is essential to regularly evaluate language progress and provide timely language interventions to support language acquisition.

The issue is that children grow up in diverse environments, leading to variations in their exposure to vocabulary. However, traditional approaches often rely on standardized vocabulary lists and pre-made storybooks or toys for language skill assessments and interventions, lacking the diversity diverse support.

Recognizing the shortcomings of conventional, one-size-fits-all approaches that fail to address the diverse backgrounds of children, the team created an innovative educational system tailored to each child's unique environment. They began by employing home IoT devices to capture and monitor the language children hear and speak in their daily lives.

Through speaker separation and morphological analysis techniques, the researchers examined the vocabulary children were exposed to, the words they spoke, and those they heard but did not vocalize. They then assessed each word by calculating scores for each word based on key factors relevant to speech pathology.

To create personalized educational materials, the team utilized advanced generative AI technologies, including GPT-4 and Stable Diffusion. This enabled them to produce custom children's books that seamlessly integrate the target vocabulary for each individual child. By combining speech pathology theory with practical expertise, the researchers developed an effective and personalized language learning system.

The researchers designed the system to accommodate variations in children's language development by allowing for individualized weighting of factors and flexible vocabulary selection criteria. The system can automate both the extraction of target vocabulary for each child and the creation of personalized storybooks, ensuring that both the vocabulary and the storybooks can be continuously updated in response to changes in the child's language development and environment.

After testing the system in 9 families over a four-week period, the results showed that children effectively learned the target vocabulary, demonstrating the system's applicability in everyday settings beyond the therapy room.

Inseok Hwang from the Department of Computer Science and Engineering, along with students Jungeun Lee, Suwon Yoon, and Kyoosik Lee from the Department of Computer Science and Engineering at POSTECH in collaboration with Professor Dongsun Yim from Ewha Womans University's Department of Communication Disorders collaborated on the project.

Jungeun Lee from POSTECH, the lead author of the paper, said, "We effectively addressed the limitations of traditional, one-size-fits-all approaches to child language assessment and intervention by using generative AI. Our goal is to leverage AI to create customized guides tailored to different individuals' levels and needs."

Professor Hwang from POSTECH, the corresponding author, remarked, "Through interdisciplinary research, we have successfully developed a personalized language stimulation and development system that integrates generative AI technology with speech pathology theory. We hope our findings will encourage educators to respect and incorporate the diverse environments and learning goals of children."

Co-author Professor Yim from Ewha Womans University said, "Our work demonstrates the potential for non-traditional, personalized language support services. The system showcases the ability to tailor target vocabulary extraction and linguistic stimuli delivery for children exposed to varied environments and languages."

More information: Jungeun Lee et al, Open Sesame? Open Salami! Personalizing Vocabulary Assessment-Intervention for Children via Pervasive Profiling and Bespoke Storybook Generation, Proceedings of the CHI Conference on Human Factors in Computing Systems (2024). DOI: 10.1145/3613904.3642580