|
Code reviews can reveal paths and conditions that the application code has been written to address. Functional testing, even with the addition of many types of input and user acceptance testing, is not the same as determining code coverage. You'll have to decide what goal you are trying to accomplish -- are you trying to build functional testing and/or user acceptance testing? Do you need code reviews? Do you need to determine code coverage? One does not replace the other.
Let's use an example to discuss the difference between functional testing and code coverage. We'll use a Web application with shopping cart functionality and think about code conditions that may have been built. Then we can look at functional test scenarios versus code coverage to consider an example where the gap between testing and code coverage can be understood.
Very specifically, to keep this example manageable, let's think about code that may be built to address a user whose credit card on file may include the following conditions:
- Credit card is valid and not expired
- Credit card is valid and expired
- Credit card is valid and will expire this month
In functional testing, a tester may consider conditions 1 and 2 but might not consider or realize condition 3. Let's suppose that the code has been written to show a warning message to the user that their credit card will expire this month. The user can shop with the card until the first day of the next month but perhaps throughout the month, the application will show a warning message to the user that the card on file will expire at the end of the month. If a tester isn't aware of this condition, then it might not be a scenario that would be addressed through either functional or user acceptance testing.
"Sufficient" is an interesting word. Is it sufficient to test just conditions 1 and 2 listed above? If no change occurs in processing the order, does the testing team feel the need to test condition 3? What's sufficient? An understanding of the code would reveal there was a branch in the code that checked for condition 3. What if on condition 3, the warning message shown to the user was incorrect or had an embarrassing typo? The perspective of "sufficient" seems to shift when a defect is missed in production.
I'd prefer to think in terms of "realistic" versus sufficient. It seems there will never be enough time to test, so many conditions, so many variables, so many browsers and so many operating systems can add up to more testing than there is ever time for. What are likely or realistic conditions to test for? And what conditions are there to test that I might not be aware of without insight into how the code was built?
This is why I spend time with the developers and system architects to understand the conditions that have been coded. Unless I'm involved in a code review, it's not likely I will be mindful of all the conditions that have been coded for. I try to gather all the possible test conditions I can think of and solicit ideas from others on the team. Then I pare that entire list of testing down to what is realistic, thinking of risk and likelihood all the while.
Can you gain coverage by performing other forms of testing -- other forms such as load and stress testing as well as testing with different data? Absolutely. Let's consider that you were able to get access to a production copy of data and to test with that data -- now you increase the likelihood of finding a user whose credit card will expire this month. In the case of stress testing, you might find a defect such that the credit card authorization system can't provide authorization rapidly enough for "reasonable" performance when a number of users submit their order at the same time. (I put the word "reasonable" in quotes because I recognize this is a vague reference and reasonable has to be determined, or at least discussed within your project team. And it may be that "reasonable" is never agreed upon throughout the team.)
A final comment on trying to make the testing realistic: It might be that you identify more tests than can ever be executed. I've been in that situation numerous times. In one environment, I agreed with my project stakeholders what would be tested for every release. Additionally, we had some test conditions -- especially stress testing -- that we only executed every few releases, with an awareness across the team that this was the case. No one on the team thought we were covering everything; we had all accepted that covering everything wasn't possible. Instead we had agreed we would focus on conditions that were highly likely with a strong understanding of where the risks were. We didn't focus on sufficient, we focused on realistic.
|
