Software development in general, and embedded development in particular, is hard work. In addition to the normal...
sorts of issues that plague the development task, embedded software is often used in places where software updates are difficult to accomplish, making the need for quality that much more important. The first part of this two-part tip spoke to the motivation for using a continuous integration server, and spoke to the sorts of things that this CI server can do for us. In this second part, we learn more about how and why CI can help us develop embedded systems better, faster and cheaper.
Lowering your risk profile through continuous integration
The classic Boehm Cost of Change Model says that the longer an item remains as an unfound defect, the more expensive it is to fix it, and that the cost of that remediation rises exponentially over time. Think about it. Imagine you have a defect in your engine control system that you’re building, and you don’t find out that there is a defect until two million units are out driving around, with five deaths associated to the defect. The cost of getting all of the units updated could seriously erode your company’s bottom line -- and that’s not even considering the costs associated with the deaths. But if the defect was found when the first three cars were hand-assembled for engineering testing, the cost might be 100 times less. If the defect was found during QA testing of the release code candidate, the cost might be 100 times less still. And if we found this defect in a nightly regression test, the cost to fix the defect might be close to zero. Which outcome do you think your company desires? In other words, we can reduce risks in relationships, reputation, and profitability simultaneously by a simple application of CI to our software development process. Quality baked in from the start is always cheaper than fixing a mess due to a lapse in quality later.
Let’s consider a different risk -- a risk of not completing the code in a timeframe needed to match the release plans for the car. How do you reduce that risk? If you have a giant project plan and ask each responsible person what their percent complete is, you’re likely to get an answer of “I don’t know. Yesterday I was 83% complete, so I guess I’m 86% complete today.” Even if they knew those numbers, is that helpful, especially since “the last 10% takes 90% of the time?”
A better way might be to have all of the functional tests in the CI server, failing at first, and then passing as the code gets developed. We then have a solid metric that we can count on -- the percent of features that are implemented and ready to ship. Additionally, if we are following an Agile process, we can easily derive an estimate of when everything will be done, by dividing the total story points remaining in the product backlog by the team’s velocity. And unlike a static project plan, this completion estimate just gets better and better as time goes by.
This brings us to the final benefit that CI brings to the embedded software development process -- that of visibility.
How a continuous integration server can aid establishing the right communication
Learning to fly a plane is harder than learning to drive a car. Why? Because of the information overload that a pilot is faced with compared to a driver. Most cars can get by with one “engine check light,” a speedometer, and a gas gauge. Beginning drivers have issues with steering, braking and reading even these rudimentary gauges, while avoiding hitting things with their cars. But on even a basic Cessna, you have multiple radios to monitor, airspeed gauges, direction (and rates) for all three axes of motion, multiple engine monitoring measurements, all the while controlling the three flight surfaces. Oh, and by the way, pilot to a destination and don’t crash into anything in the process. The actual flying part isn’t harder than the driving part. But the sensory overload is much harder for a beginning pilot than a novice driver.
Same sort of thing for an embedded software developer. They have to navigate through multiple levels of hardware, hardware abstraction, stacks of software, etc., trying to get to the goal of implementing some functionality, and trying very hard not to crash the system. When a system being developed does crash, identifying the cause of the crash to find the code in error can be extremely challenging. The overload of finding the causal chain that produces your defect is extremely time-consuming. Even just reading through a compilation log to see why your code doesn’t compile can be a huge challenge.
Luckily, use of a CI system, especially one with test support tools in use, makes the workload a whole lot easier. Changed repository state, errors in code compilation, tests that don’t work, metrics generation, and so on, are all automatically generated, filtered, and formatted in an easy to read manner (just like that “check engine” light.) You can now drill down through the error and rapidly fix problems, rather than scratching your head as to what the problem is and where the problematic code is located. Less time sifting through tons of meaningless log files gives you more time to do what you really have to do -- work on implementing new functionality.
Sending artifacts such as eMails about the CI server build and test status to only those who want to know helps, too. Less time sifting through meaningless email subject lines gives you more time to implement new functionality as well.
eXtreme Feedback Devices take all of this to the next level. When most managers mosey by the development area, they are really just looking to answer a simple question: “How are things going?” If they can answer that question by way of a simple CI server status, with number of unit tests passing and maybe some simple metrics, they are happy and informed to the level they need. And if you can do that with a bubbling lava lamp, even better. There’s another benefit. Building eXtreme Feedback Devices, such as the lava lamp, are a lot more fun than putting together a detailed status report that no one really wants to labor through, anyway.
Using the wrong (or no) set of development tools can consume so much time that instead of owning the development process, it ends up owning you. The benefits of continuous integration are numerous, and the use of an appropriate set of tools can easily justify the upfront costs in acquiring the tools and setting them up.