The estimates of calories burned made by smartphones, smartwatches and other wearable devices vary wildly. That's because these devices lack the sensors required to gather all the information they need to make accurate estimates.

But new work by researchers at Carnegie Mellon University and the Indian Institute of Technology (IIT) Gandhinagar shows that adding an inexpensive thermal camera to these devices could substantially improve accuracy.

Using the thermal camera to monitor a person's breathing rate and body temperature could reduce inaccuracies in energy expenditure estimates from nearly 40% with a current smartwatch to just under 6%, said Mayank Goel, an associate professor in the School of Computer Science's Software and Societal Systems Department (S3D) and Human-Computer Interaction Institute (HCII).

"When people see these numbers, they make changes in their behavior and that can be troublesome if the numbers are wrong," Goel said. Someone who thinks they just burned 400 calories on the treadmill, for instance, may eat more calories throughout the day, even though their actual expenditure was closer to 200 calories.

"That is a huge problem," he added.

Monitoring respiration has been a longtime interest of Goel and his Smart Sensing for Humans (SMASH) Lab, which develops technologies for such applications as health sensing and activity recognition. For instance, he previously developed ways to measure breathing using several methods ranging from wireless router data to custom wearables that analyze chest movements.

While pursuing a different project, Maite Sadeh, a Cornell information sciences major who was a SMASH Lab summer intern, found reports on how respiration could be measured using a thermal camera to detect exhalations of hot air. Goel's group then realized that inhalation leads to evaporation around the lips and nostrils. Both of these signals can be captured by a thermal camera.

Rishiraj Adhikary, a Ph.D. student in computer science at IIT Gandhinagar who was also a lab intern via a Fulbright Scholarship, then found studies showing that respiration combined with heart rate could be used to measure energy expenditure.

But respiratory and heart rates aren't sufficient because they fail to take individual physical and contextual differences into account, Goel said.

The gold standard for clinically measuring energy expenditure is a calorimeter, which uses heart rate, respiration and the concentration of carbon dioxide in exhaled breaths to determine calories burned. Wearables already do a reasonable job at measuring heart rate and adding a thermal camera would provide a means for measuring respiratory rate. No solution exists for measuring CO₂ concentrations with a wearable device, but the thermal camera could measure body temperature.

"We lose the ability to measure the concentration of oxygen and CO₂, but we gained temperature measurements," Goel said.

The combination of those three data points, with help from machine learning, enabled the researchers to develop a system, called JoulesEye, for estimating energy expenditure. They recruited 54 participants who either cycled or ran for 15 minutes. Their tests showed that JoulesEye could estimate burned calories with an error rate of just 5.8% when compared to a clinical calorimeter.

In addition to helping fitness buffs, JoulesEye could be used in sports training, as well as for monitoring people with chronic diabetes or cardiovascular disease.

A report on JoulesEye, co-authored by Goel, Adhikary, Sadeh and Nipun Batra, an assistant professor of computer science at IIT Gandhinagar, has been published in the Proceedings of the ACM Mobile, Wearable and Ubiquitous Technologies and will be presented in October at the UbiComp 2024 conference in Melbourne, Australia.

The cost of incorporating a low-resolution thermal camera into wearable devices should be feasible, as these cameras are already available for $45 or less. But Goel says the team is working to incorporate an even lower-resolution thermal camera into the system, which could lower the price of the sensor. It would also reduce privacy concerns about a camera being routinely pointed at the user's face.

The team also hopes to reduce the amount of time the thermal camera must be aimed at the user's face. It now takes about 40 seconds to make the necessary measurements.

"Our goal is that the time it takes to check your watch should be enough time to get the information we need," Goel said.

More information: Rishiraj Adhikary et al, JoulesEye, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (2024). DOI: 10.1145/3631422