CIOs and software quality assurance: Five hurdles for QA managers

Director of QA and instructor John Scarpino has provided tips for overcoming five hurdles associated with CIO expectations.

Throughout my experience in software testing and software quality assurance across such industries as finance, telecommunications, healthcare, ecommerce, eMarketing, insurance and online education, it has become clear to me that not all CIOs expect the same output from quality assurance managers. Much of this depends on how knowledgeable the CIO is about quality assurance, the size and culture of the corporation, and whether it is a research-based institution, traditional Fortune-500 corporation or a start-up organization.

There are many hurdles that QA managers must overcome in order to meet deliverables and exceed a CIO’s expectations. The purpose of this article is to shed light on some lessons I’ve learned from some of the best and brightest in the IT industry, particularly regarding CIO expectations about quality assurance -- including what you should, and should not, do.

Hurdle #1: Defining software quality vs. software testing

Some CIOs, especially at large corporations, do not understand the difference between software quality assurance, which ensures the usability of software quality processes and practices, and software testing, which carries out the actual testing activities for the application. But for some CIOs, the two are identical. This is one of the first hurdles that a quality manager must face; demonstrating what QA is and what it isn’t, and making the case for why QA (in addition to software testing) is imperative for success.

Quality assurance must be supported by executives at the top (namely, the CIO), to avoid the tunnel vision of “I don’t care how you do it, but just get the end-product out the door today!” CIOs who understand the importance of QA and can actually explain his or her expectations of “good enough” will consequently have a good technical understanding of the resources needed to maintain an IT organization and make it successful. I have seen many CIOs say that QA can be outsourced. They think from this perspective because to them, software testers are bodies and not assets. In reality, QA is the last department you would want to outsource because it means giving up control of your product to another company.

Hurdle #2: Removing dollar signs from the “quality equation”

A CIO’s level of risk-acceptance depends on how public-facing the application and its systems are and how many bugs the application can handle before they start to have an adverse effect on the product’s profit margin. Much to the chagrin of QA managers, one of a CIO’s top priorities is to push a product to go “live” as quickly as possible, for both business and personal reasons. Obviously, the sooner the product goes live, the more revenue the company will make; the more likely it is that the company will be viewed as a success in the public eye and the CIO will be championed as one of the key people who got the company there. When quality is judged simply on the basis of revenue, it isn’t any wonder why some CIOs are willing to overlook risks for the sake of making a client happy with an ahead-of-schedule launch. Nevertheless, financial success is not always an indicator that the company has a quality product. CIOs’ number-one priority should be to prevent risk -- it is in their personal best interest and in the best interest of the company for them to do so. If not, then a reputation/corporate image disaster can result, which no dollar amount can save.

Hurdle #3: Mitigating software risk with available QA resources

Achieving a completely bug-free system before product launch is no picnic, and quality assurance managers often struggle to adequately communicate the implications of risk at the executive level. What’s worse, even if a QA manager does reach the CIO, their concerns may fall on deaf ears. As mentioned in Hurdle #1, CIOs tend to think of software testing and software quality assurance in the same light. CIOs instruct their QA managers to quickly test the product until it is just good enough to pass customer inspection and get out the door. So when QA managers voice their grievances about quality, the CIO may refute it by saying, “The customer hasn’t had a problem yet, and as long as the customer is happy, that’s all that matters. I’m not going to waste precious time and resources on fixing something that isn’t broken.”

But consider this: just because a boat can stay afloat in calm seas does not mean it is structurally sound enough to withstand a hurricane, tsunami or rogue waves. And if the boat sinks, the crew (i.e., QA team) goes down with the ship … including the captain (i.e., the CIO). Quality affects the entire organization, and it is not just a departmental issue. Efficiency is built with the resources and processes that are available within the company. In other words, QA must be ingrained throughout the internal aspects of the organization before it can be appropriately demonstrated through an external-facing product. In successful QA organizations, quality assurance activities are not only managed by the CIO, but also by the COO/Office of the Comptroller, and sometimes even the CFO/Office of Finance. This setup requires the QA manager to report to the CIO on matters concerning IT activity and the COO/CFO on the matter of cost. This assures an unbiased environment removed from the confines of software development, so that quality can be achieved both internally and externally. The point is to have QA infiltrate the organization and persuade the Board of Directors to accept it as an imperative for success.

Hurdle #4: Heightening security and performance

We, as information technology and information systems experts, understand that security and performance play a huge part in how the software will impact the customer. Unfortunately, we usually don’t hear about it unless the result is a negative one. There has been a massive increase of security threats and performance issues in software applications, systems and networks in recent years. Given the current landscape, most companies feel that maintaining the integrity of sensitive data is an important part of ensuring the quality of their product or service. The financial and healthcare sectors and third-party vendor applications are particularly wary of security issues; Sarbanes Oxley (SOX), Office of the Comptroller of the Currency (OCC), Health Insurance Portability and Accountability Act (HIPAA), and the Statement on Auditing Standards (SAS 70) are great examples of standards that ensure that data is managed appropriately.

Daily testing, monitoring and controlling of these environments are needed to prevent threats and ensure that new services, products and solutions are secure and well-balanced when used publicly. CIOs must be aware that security and performance testing, quality assurance and quality control are all unending processes. The investment of resources to find solutions is worth it. The moment that you stop investing in your QA is when hackers and other threats to your product become not only an internal issue but also a corporate reputation issue to external constituents. For instance, if your customer is a bank whose financial records or accounts are accessed illegally (hacked) because your software product failed, it will likely turn into negative publicity for the bank, but the repercussions will be even more negative for your company. Negative media coverage can severely damage or even bankrupt a company, so using software testing to prevent security and performance risks is a no-brainer if for no other reason than to avoid bad public relations.

Hurdle #5: Product usability and reliability

Product “usability” is rarely taken under consideration. Always ask: “How easy is our system to use?” Your product should be measured against the competition to ensure that it is the best, fastest and least complicated system available. Software test usability is often overtaken by its tremendous number of system functionalities. Just because a software system or tool works does not mean it is the best. If it takes a long time for the user to figure out all of the functionalities and actually make it work, how practical is the tool? Walking the line between “cutting-edge” and “lean-stability” is key for customer satisfaction, overall success and ROI. Think about the Nintendo Wii; senior citizens are using it for exercise because the controller has simple buttons and uses natural body movement to perform the act. If you can use a fly-swatter, you can play the Wii. Now think of a traditional Super Nintendo controller and related games which required a combination of up, down, left, right and pressing two or more buttons at once to perform an action. Which is easier?

CIOs find it difficult to understand how something as intangible as software quality assurance can assist with improving the bottom line; how it can move from being an idea to actually proving ROI. If a CIO runs the software quality department loosely without control or understanding of how to make it come to fruition, then SQA’s growth, success and niche in the company will not be a success.


About the author: Dr. John Scarpino is director of quality assurance and a university instructor in Pittsburgh.

Next Steps

Learn how to improve QA tester productivity with time management


Dig Deeper on Topics Archive