What challenges one software tester might be a no-brainer for another. Lloyd Roden of Grove Consultants noted this fact as he laid out his seven top software testing challenges and advised on how to handle them in a session today at the 2011 Software Testing, Analysis and Review (STAREAST) Conference in Orlando. Some of the challenges he chose, recounted below, and his methods for fixing them may be considered controversial -- depending on a software tester’s experience and context.
Challenges, by definition, require special skill and effort to face, Roden said. Challenges are different for different people. They can be a positive thing when they help you improve or open your mind. Or they can be negative if they are harmful or have undesirable consequences.
Challenge #1: Ban the use of “best practice”
In our industry, we often hear of “best practices” for testing techniques; but what may work well in one context often doesn’t work well in another, Roden said. He stepped through the Dreyfus Model for skills acquisition, which include novice, advanced beginner, competent, proficient and expert. The competent practitioner will use defined practices. However, the more advanced testers, those who are proficient and expert practitioners, will take context into account, using experience and instinct rather than just stepping through defined processes.
Best practices are useful for helping people learn, but following step-by-step test scripts without question will stifle creativity and frustrate the best testers. Proficient and expert testers will know when to use prescribed techniques and when techniques need to be tailored or adapted to the situation.
Roden got laughs from the crowd when he asked what would happen if a Jedi needed to stop and consult the rule book to figure out how to use his light saber. “We need Jedi testers,” he said.
Challenge #2: Focus on quality rather than quantity of test cases
Often managers judge the productivity of testers based the number of test cases they execute. This is a meaningless number if we don’t understand the background of the test case, Roden said. Some people may count every click of a button as a test case. Others may be counting transactions as test cases. Someone else may count a complete installation as a test case.
Rather than measuring quantity of test cases, Roden suggests that quality of the tests cases should be assessed. What are the test cases doing? Are they adding value? What is the test case coverage?
High quality test cases are those that will help the project team gain confidence by demonstrating a depth and high degree of coverage, Roden explained. High quality test cases are those that find and remove the defects. The objective of the test effort is to provide quality information about the application under test so that management is able to make informed decisions.
Challenge #3: Don’t lie with metrics
Sometimes managers are so engrossed in the numbers and metrics, they aren’t looking at the meaning behind them. Roden has found that metrics can be very misleading. Before management makes any judgments or decisions they need to have a good understanding of how the metrics are obtained.
Roden gave several examples of how graphs, metrics or pie-charts can be misleading unless some questions are asked. If there isn’t a statistically significant amount of data, the results are inconclusive. Another graph showed an increase in bugs found over time; however, what it didn’t show is that it was the effort to find the bugs was increasing over time. In other words, it was more difficult to find bugs, an important factor to consider when assessing quality.
Challenge #4: Using the resources we have
The current economy causes companies to demand that workers do more with less. That assignment calls for assessment of the strengths of the team as a first step and figuring out how to capitalize on them comes next. Roden advised test and project managers to be honest with vendors about their budget, so that there is trust built on both sides. Internal training and workshops, rather than outside training, might help with cost savings. Also, he suggested investigating the use of free or open source utilities and tools.
“Ascertain what you do have rather than what you don't have,” advised Roden.
Challenge #5: Tailoring dashboard information to the recipient
Different people on a team need different types of information about the project. For instance, said Roden, the project manager will be interested in different information than the business, the development manager or the CEO. When designing a dashboard, it’s important to understand who the recipients are, what kind of information they want to see, and what will be available with the tools being used. You have to talk the language of the people who are receiving the reports.
Roden gave the example of the manager who said he did nothing with reports because he didn’t understand them. When asked what he wanted, one senior manager said, “All I want is a red or green. I don't understand yellow. It's the yellow light that causes the most accidents.”
Roden advises people to “learn the tools. Don't just touch a button and get a lot of reports that you don't understand. "
Challenge #6: Challenge complexity at every opportunity
Sometimes simplicity can be seen as weak, while complexity is seen as sexy and cool; but is it?
Complexity should always be questioned. Roden took out the instructions to his watch and said he needed to bring it with him at all times in order to set the alarm. After reading about pushing at least four different buttons, Roden’s point was clear: The instructions were so complicated they were funny! He also described the very expensive pen that was developed so that it could write on the moon, upside down and in many different conditions. Roden then delivered the punch line: “The Russians did the same thing. They used a pencil.”
When complexity is encountered -- whether in requirements, design, or any part of the lifecycle – always ask: Can the same thing be accomplished in a simpler way?
Roden recalled a 2002 Standish Group study that determined that 64% of the features and functions of software applications are rarely or never used. He concluded: “We drive up the complexity by adding unneeded features to the system.”
Challenge #7: Test managers and leads should test
Roden has heard various excuses from managers and leaders about why they don’t participate in hands-on testing. They don’t have enough time or they expect their team to do the testing. Roden is passionate about this challenge: “Get your sleeves rolled up and you've got to do some testing, even if it's one hour a week. Seventy-five percent of those meetings are useless,” he says emphatically.
In order to really understand the frustrations the team is dealing with, the manager and lead needs to actively participate in the testing efforts, in Roden’s opinion. This will help the leader gain more credibility and trust within the organization.
Roden suggested that software test teams adopt Friday afternoon as an exploratory testing or bug hunting afternoon, and that the test lead and manager participate.
Software testing action items
Roden ended his presentation with a final challenge to the group: “I've given you seven of my top challenges. You may have others that are more important to you. Set yourself an action. Tackle one at a time.” Roden warned against apathy. Don’t hesitate, he said, to tackle those challenges that most affect your organization.