Billions of dollars are being spent worldwide to modernize electricity grids with smart meters. These meters promise to save households money by making it easier for us to understand and manage our energy use. However, our new research suggests these promises might not be fully delivered due to a lack of access to high-resolution, real-time energy data.

Smart meters are the enabling technology of modern smart electricity grids. Smart grids can use digital technology to fine-tune the management of electricity supply and demand. This ensures the grid can deliver low-cost and reliable power.

Countries like Australia are racing to install smart meters extensively. Last year the Australian Energy Market Commission recommended a goal of 100% uptake among small customers by 2030. In response, an Australian Energy Council article suggested these meters aren't living up to their potential.

This isn't just an Australian problem—it's a global challenge. Our research offers a solution to unleash the promised benefits of smart meters at least cost. From improving data transmission to protecting our privacy, there's a lot we can do to make our energy systems smarter and fairer for everyone.

Why do we need a truly smart grid?

Our demand for electricity is set to soar as the push to electrify everything gains momentum. The Victorian government, for instance, has banned gas in new homes from 2024.

The International Energy Agency forecasts a 3.4% annual rise in electricity consumption from 2024 to 2026. As transportation electrifies, electricity's share could increase from 1% in 2018 to 49% by 2050.

To meet this growing demand while cutting carbon emissions, we must ramp up renewable energy production. However, the unpredictable nature of wind and solar power presents challenges for the grid.

To manage highly variable supply and demand, we need to digitize our grid. Advanced technologies such as sensors, machine-learning algorithms and cloud computing will enable us to optimize electricity generation, distribution and consumption.

Smart meters are the cornerstone of such a system. They can provide the detailed, real-time data needed for smart grid applications.

Smart meter deployment has surged globally. The smart meter market is forecast to grow from US$17.5 billion ($A26.6 billion) in 2024 to US$31.8 billion by 2028.

Our research sheds light on this global deployment and its significant challenges.

What will better data allow us to do?

Grid modernization and smart meters came with big promises of saving money for consumers. This hasn't happened. The reason is that many direct benefits to consumers require high-resolution data—and the required level of fine detail in real time isn't being provided.

For example, as a direct benefit to consumers, some machine-learning techniques can help households optimize their energy use by providing insights into exactly how much electricity each appliance is using and when. This information could enable them to lower their electricity bill. These tools can also detect abnormal usage patterns, allowing timely intervention and maintenance of faulty appliances.

However, these applications and other smart grid benefits for consumers all require high-resolution data.

What obstacles must be overcome?

We found three major reasons for the current limitations of smart-metering infrastructure.

Data transmission is the first big challenge. High-resolution and more frequent data means a higher volume of numbers, which leads to more delays or disruptions to data transmission.

The second challenge is the data warehousing needed for huge volumes of data. It's expensive, too.

Building and running a data warehouse costs US$19,000–$25,000 per terabyte each year. Upgrading from hourly data to every two seconds requires 1,800 times the storage, at an extra cost of US$36 million! And that's not counting maintenance, backups, or sharing the data.

The third major issue is data privacy. The data can also be exploited by attackers. They could figure out what appliances you have, your home setup, or even your habits.

This can lead to criminal activities or serious invasion of privacy. For example, people could be tracked based on their vehicle-charging patterns.

Even law enforcement uses electricity data in court cases. One case involved the detection of indoor marijuana growing.

A way forward at the cheapest cost

Ideally, we need a solution that tackles all the issues using the smart meters we already have. Our solution is based on discovering repeated patterns within electricity usage data, then dividing these data into two parts.

It's like a book divided into piles of papers and page numbers, with each then handed to different parties. Neither the page alone nor the page numbers make sense until they are combined.

Similarly, we suggest dividing detailed data into smaller patterns called codewords and their daily representations. We'd send only representations to the data center, letting users keep their codewords to ensure their privacy.

Patterns of energy use often repeat. By using a single codeword to represent multiple days of similar consumption, we can greatly reduce the amount of data that needs to be transmitted. This would cut data communication and warehousing costs.

Continuous research on software, hardware and regulations is needed to refine the proposed framework for the stages of data collection, transmission, storage and analysis.

It's important for modern energy consumers to be aware that as well as consuming and generating energy (from rooftop solar systems), they also generate data through their smart meters. This data asset is becoming increasingly valuable in the transition to the net-zero era.

This article is republished from The Conversation under a Creative Commons license. Read the original article.