When we think about embedded systems and mobile devices, we immediately think of such industries and devices as communications and cell phones and smartphones, banking and ATMs, and medicine and pacemakers. We think of devices in our home for our entertainment, such as TVs and video game consoles and security systems. But would we ever think that the electric utilities industry would employ such technology? Welcome to the era of the smart grid.
At a high level, a smart grid provides electricity based on a two-way communication between the home and the utility. The utility part of the system can provide analysis and move electricity between locations in real time. An important feature of smart grid technology is advanced metering infrastructure (AMI), the devices through which communication happens. AMI can be employed, not only at the utility company but also in the home. It can provide customers with live, real-time information regarding their electricity consumption, and it may provide them with opportunities to reduce consumption and save money.
So, how do you test it?
I had the opportunity to test smart grid technology for a major North American electric utility company. The project was a pilot program. Its purpose was to find out whether customers would change their behavior if they knew how much electricity they were using on a real-time basis. My project involved testing not only the components of the AMI, but also its integration with the utility's customer information and billing applications.
The devices included in the home area network (HAN) included a smart meter, a transport device, a load control device, one or more programmable thermostats, and an in-home display. The smart meter transmitted readings in 15-minute intervals through the transport device to the programmable thermostat and the in-home display and to a Web portal.
These devices also provided a means by which the utility could communicate with customers. The electric utility could call an event, a request to reduce consumption. For example, customers could be requested to increase the temperature on their thermostat in order to reduce their use of air conditioning. The customers could opt in or opt out of the event via any of the devices. For their participation in the events, customers would receive either reduced rates or rebates.
Testing the electric utility devices in the HAN involved many of the same issues in any embedded testing project: The hardware was developed concurrently with the software, fixes to both hardware and software were made in different release schedules, performance requirements were tight, and computations needed to be exact.
In order to test the various configurations of HANs and billing rates, we designed a test lab that simulated 18 homes. Each home was a box mounted on a wall that contained the smart meter and all the devices assigned to that home. Now we needed to generate load on the meters to measure electricity consumption. We tried hair dryers, lights, microwave ovens and even a little refrigerator, but the only effective way to generate load was to use space heaters. And so, our test lab would be over 100 degrees during the events.
The test strategy involved testing both the devices and application by calling the event, and observing the results. Through the events, fundamentally all the features on all the devices were exercised. I performed some destructive testing during the events; one programmable thermostat model actually fell apart in my hands.
The system integration test approach was to track the data generated during the events from the meters in the homes through the meter data application, the customer information system and the billing system, to the customer's bills, verifying at each integration point.
The majority of the defects were in three areas. First, some hardware models, particularly the early ones, were easily damaged and not physically user friendly. For instance, when the power source of one device was disconnected and reconnected, it failed to rejoin the HAN. Second, there were defects in the application code on the devices. For example, a customer could opt out of an event on the In-Home Display, but not opt back in on the Thermostat. Third, there were many defects in the code that would estimate the correct usage when a meter lost communication with the WAN or HAN.
The overarching lesson is that testing devices is very different from testing desktop or Web applications. It involves an appreciation that the application and device are intertwined and that it's difficult to tell where the hardware ends and the software begins. Defects often involve elements of both. Testers working with embedded systems need to understand both applications and devices in order to properly evaluate quality.