A major software rollout under his belt, release manager Ken Vane breathed a huge sigh of relief. "Every single VP was looking our way," said the IT change and configuration manager for the Navy Federal Credit Union, headquartered in Virginia. "The guy from mortgage, the guy from lending, the guy from car loans, they were all there."
The VPs watched as the release manager and his team rolled out a new version of the credit union's application for managing member accounts. "It was go or no go," said Vane. "And all we kept hearing was, 'Go, go, go, go …'" To the team's great relief, the new software worked.
If anything had gone wrong, the VPs would have weighed in with a "no go." They were responsible for signing off on each set of code changes that was released. If an error occurred -- an incorrect value that updated a general ledger account, for example -- Vane and his team would have brought the rollout, which affected hundreds of thousands of transactions, to a halt.
The successful software rollout, which involved the participation of 250 employees, with a 50/50 split between business and IT, was no accident, said Vane. The release manager and his team spent 18 months preparing for November 2012 upgrade and credit their success to the following factors:
- Advanced planning that put in place proper controls for major upgrades, as well as routine updates going forward;
- Thorough test runs to preproduction environments;
- Close alignment between business executives and release managers;
- Better management of code changes, tying them to business goals; and
- A rollback plan, which the team rehearsed before the release went live.
Vane shares his advice on rolling out new software with other release managers.
The problem: Too much custom code
The release management project was a major upgrade to the mainframe application, Fidelity Services, which handles more than four million member accounts in the credit union's 220 branch offices around the world. Nonprofit Navy Federal, which provides banking and financial services to members of the U.S. armed forces, had relied on the packaged application since the 1990s. But over the years, the software was customized so extensively, it was difficult to keep track of all the interlocking parts and manage updates efficiently. "Over time, the body of changes cascades, affecting not just one service, but also many other services that inherit those changes," said Vane. The bank made a decision to move away from customization, essentially starting over with a clean slate. "We said, Let's document how the interfaces work, how we are talking to each other." The credit union replaced its aging customized version of the application with the most recent off-the-shelf version of the Fidelity mainframe software.
Tip #1: Put controls in place to manage code changes
Eighteen months before the November 2012 release, Vane's team sat down and asked, "What procedures do we need in place to be successful?" In the past, the process for making code changes was pretty informal. "You committed the changes to the [software change and configuration management] tool and said, Okay, we are doing a new release on Friday." But Vane knew that the planned mainframe software upgrade could not succeed without strict controls. So the team came up with a structured process: Before work could begin, an employee fills out a change management ticket, specifying exactly what the change entailed. For example, this update includes 12 enhanced business requirements and eight new bug fixes, said Vane. "In the past, changes were made at will. A business unit would ask for a change, and we would do it. We were good handymen, but we were not good at asking what the impact was."
To prepare for the upgrade, Navy Federal relied on products from Serena Software, including Serena Release Manager, which automates to process of rolling out new software and updates Prior to using this tool, Vane and his team managed the releases manually, tracking information in an Excel spreadsheet. "Now, with a centralized repository, the process is much easier," said Vane. Navy Federal also uses three other Serena products: Business Manager (for automating business processes), Dimensions CM (for software change and configuration management) and ChangeMan (for mainframe release management).
Tip # 2: Develop a business case for code changes
Specifying the details about requirements and bug fixes was the first step of the new process. Step two required the line-of-business manager requesting application changes to document the reason for the request. "If this change is implemented, what impact will it have on the business? And what's the impact if the change is not implemented?" said Vane, explaining what he and his team wanted to know. One line-of-business manager was especially quick to comply. A marketing manager, in charge of campaigns that rely on Navy Federal's inbound marketing application, said this: "The new business rules [implemented through changes to the application code] will allow us to launch a new marketing campaign on Friday. If you don't do this, we are going to lose $300,000," recalled Vane. "He knew to how to make his case."
Tip #3: Allow for concurrent development
As plans for the Fidelity upgrade got underway, the release manager's team initially believed it would make sense to put a moratorium on changes to other applications while the new Fidelity software rolled out. They quickly learned that didn't make business sense. A manager who relied on Navy Federal's internal application for managing conversations with customers said "No way," Vane recalled. "He needed the ability to make changes whenever the offered customers a new product or promotion." Vane offered an example. Running a short-term promotion, such as a new Certificate of Deposit (CD) offer with a 3% interest rate and $500 minimum deposit, involves tweaking application code to support that promotion. "Business decisions were being made," said Vane, and his team members understood that those decisions would drive code changes.
Tip #4: Do user-acceptance testing in a preproduction environment
All release management projects should involve some sort of trial run. But Vane and his team treated the trial run seriously -- completing the entire process of rolling out the software upgrade to a preproduction environment. They ran through all that "logistical management stuff" as if it were the real thing. "We did the conference call, where everyone gets on, including the line-of-business VPs," he said. The team included 70 employees, split evenly between IT and business.
Tip #5: Plan for a rollback -- and pray it doesn't happen
Rollback, pulling back the new release because the code is not stable, is every release manager's worst nightmare. "A rollback involves extraordinary measures. You may have made changes to millions of database tables, and now you have to move that back to the previous version," said Vane. To accommodate this possibility, you take an image of what was before, and restore it back, he said. It's not easy to accomplish. "But we planned for rollback. We rehearsed it. It's insane not to, and our CIO insisted on it."