Virtual reality (VR) video games that combine screen time with exercise are a great way to get fit, but game designers face a major challenge—like with regular exercise, adherence to "exergames" is low, with most users dropping out once they start to feel uncomfortable or bored.

Computer scientists at the University of Bath believe they've found a solution: Create exergames that use sensors to continuously measure a person's emotional state while they exercise, then tweak the game—for instance, making it easier or harder—to keep the user engaged.

Dr. Dominic Potts, lead author of a new study into harnessing cutting-edge sensor technology to keep exercisers motivated, said, "When it comes to physical exercise in all forms, motivation and exercise adherence are huge problems. With exergaming, we can address this issue and maximize a person's enjoyment and performance by adapting the challenge level to match a user's abilities and mood.

"Exercise games that are completely adaptive will sense a person's emotions and give them more 'rewards' when they're struggling and more obstacles when they're ready for a new challenge."

The findings are published in Proceedings of the CHI Conference on Human Factors in Computing Systems.

Game designers have long aspired to develop more personalized exergames—i.e., programs that tune in to a person as they work out, adapting to their struggles and ambitions of the game unfolds. However, finding a trustworthy method for measuring a user's evolving emotional state has proven elusive.

The Bath team has made a breakthrough by employing a novel range of sensors—which could be embedded in VR headsets and wearable devices such as smartwatches—to track physical changes experienced by an exerciser as they work out.

The team's paper received an honorable mention award at the CHI Conference on Human Factors in Computing Systems.

The researchers hope their findings will be adopted by game designers to create immersive programs that can keep a person pedaling, running or weightlifting long after they'd normally choose to quit.

Background noise

Until now, sensors—which can be extremely effective at tracking emotional states when a person is sedentary—have proven unreliable at measuring the emotional landscape of a person involved in physical activity, making it hard to recognize whether a person is experiencing happiness, stress or boredom.

"Traditionally, these sensing devices have been put into VR headsets to track blinks and pupil dilation, but generally they are highly susceptible to physiological and background noise," explained Dr. Potts.

Unwanted 'noise' is generated in two ways: by a person moving in unpredictable ways—as they often do when exercising—and by the exerciser responding to the virtual environment (VE) in which they're immersed. So, for instance, a sensor that shows a person's pupils dilating as they exercise might reflect luminosity changes in the VE rather than the user's evolving emotional state.

Reliable tracking

In the new study, 72 participants were involved in a VR static bike race while the Bath scientists used a specific combination of sensors to measure pupil size, facial expressions, heart rate, levels of sweating, skin inflammation and electrodermal activity (which measures the skin's ability to conduct electricity, reflecting stress levels).

Data was collected through the sensors while racers worked out in four distinct VEs, where each environment was designed to induce a specific emotion (happiness, sadness, stress, and calmness). Participants moved through these VEs at three different exercise intensities (low, medium, and high).

For each workout, the researchers were able to paint an accurate picture of a user's emotional state, matching the game's level of difficulty and the nature of the VE with the physiological changes experienced by the user, as picked up by their sensors.

Drawing from this research, eight guidelines have been formulated for creators of VR exergames, aimed at enhancing the emotional engagement of users. Among these recommendations are the following:

• Pupil-detecting sensors should be designed to correct for luminosity changes in the virtual environment.
• Preexisting sweat levels of a user should be taken into account to predict stress and arousal of the nervous system.
• Sensor data should be cleaned before and during an exergame (i.e. removing signals unrelated to emotional changes) to measure interpersonal differences and enable the game to be tailored to each user.
• Multiple physiological sensors should be used to improve predictions of a user's emotional state.

Dr. Christof Lutteroth, who leads exergame research as director of the REVEAL research center and is co-investigator at CAMERA, both based at the University of Bath, said, "In the long run, our objective is to make VR exercise emotionally intelligent.

"We fully expect VR physical activities to explode in popularity in the years ahead—school children are already using them as part of their exercise programs and they are also being used in rehab and sports science—so it's important to focus on making technology that's emotionally intelligent and adaptive to differences between users."

The University of Bath research team for this study included Dr. Dominic Potts, Masters student Zoe Broad, Psychology undergraduate Tarini Sehgal, Joseph Hartley, Professor Eamonn O'Neill, Dr. Crescent Jicol, Dr. Christopher Clarke and Dr. Christof Lutteroth.

More information: Dominic Potts et al, Sweating the Details: Emotion Recognition and the Influence of Physical Exertion in Virtual Reality Exergaming, Proceedings of the CHI Conference on Human Factors in Computing Systems (2024). DOI: 10.1145/3613904.3642611