The other weekend, my son's team played in the championship game for his (American) football league. I arrived at the sports complex a few minutes after him to find him and his teammates more or less randomly throwing footballs, chasing, tackling and the like in an open field with no coaches to be seen. I remember thinking, "What is the coach thinking?!? These kids are going to wear themselves out before the game even starts! Or worse, they're going to end up getting hurt!"
As I watched them play, I quickly noticed the smiles. I immediately stopped thinking like the parent who wants his son's team to win as I realized that these children were simply in a state of 8-year-old bliss. This was in stark contrast to a few minutes earlier, when I was talking to Nicholas on the phone as he was on the way to the game from home and I was heading in to the game from the airport. He told me then that he was "kinda nervous." You'll have to take my word for this, but if my boy said "kinda nervous," he must have been really nervous; nervous isn't his style. But there was no sign of nerves now, not in him or any of his teammates. I also noticed that the members of the opposing team were standing together in a huddled mass on the next field over, appearing anxious and pensive, also with no coaches in sight, watching my son's team run around and play. Now I was thinking that Nicholas's coach was pretty smart to let the boys play and relax for a bit.
After a while, the coaches brought the boys together to do their pre-game warm-ups. By this point in the season, the boys went through the stretches, drills and walk-throughs on virtual autopilot. The routine of the warm-ups seemed to focus the boys without bring back their nerves.
 |
|
|
|
|
In that touchdown run, "scripted" and "unscripted" came together, through the kids' observation, to find the "defect" in the opposing team's defensive scheme.
|
|
|
|
|
|
|
|
|
|
Eventually, the game kicked off. For the first 38 (of 40) minutes, it was a hard-fought, evenly matched, one-score game. With two minutes to go, Nicholas scored the touchdown that put the game out of reach for the opposing team. I jumped and cheered for the first time all season, nearly causing Taylor (my other son, age 4) to fall off of my shoulders. I won't even try to tell you that I wasn't cheering in part because I knew that the boys had just sealed the championship for their team, but that wasn't the only reason. I had made a point all season not to make a big deal about whether the team won or lost, but rather to help Nicholas focus on the more important lessons of teamwork, being "coachable," having a positive attitude, playing with heart, trying his best and having fun.
But during that 6-yard, championship-game-sealing pitch to the right, Nicholas did something I hadn't noticed him do before that made me exceptionally proud. He improvised, just a little. Even more impressive was the fact that his teammates adjusted to his improvisation to enable him to score.
If you've never watched 8-year-old boys play organized tackle football, that might not sound overly impressive to you. But after watching practices and games for an entire season, let me assure you it is. You see, these boys learn to execute the plays as they are scripted. The team made it to the championship game by learning early in the season that the plays usually worked, when they executed them the way that coach taught them, and they usually resulted in disaster when individuals tried to do something different. Disaster occurred mostly because they wouldn't try something a little different; they would try something so seemingly random and illogical as to confuse not just the other team, but the rest of their teammates as well.
This time it was different. It was a small, logical deviation from the "script," based on the experience gained from paying attention to the defensive scheme for the first 38 minutes of the game, in the form of a two-step cut from the inside of the lead blocker to the outside at the last moment. That small deviation was made successful because the lead blocker saw the cut and adjusted appropriately to support the maneuver and avoid a potential penalty.
It turns out that the lead blocker on the play was the boy Nicholas was playing with before the game. The two of them had been taking turns, one running with the ball, the other trying to keep the rest of the boys from getting the ball as they ran through their teammates on the field. That game of "keep away" that took place before warm-ups was probably where those two devised, whether consciously or not, whatever it was that enabled each of them to improvise in a coordinated way. That was all it took. Touchdown.
What does that have to with exploratory and scripted testing? Everything.
All season, the team had learned and executed their "scripts." In fact, the opposing team had even sent parents to videotape our practices so their coaches could counter our "scripted" plays. But even though Nicholas' coach taught the team to perform these "scripted" plays on the field, he always gave them time to play together "unscripted" (like before warm-ups the day of the championship game). In that touchdown run, "scripted" and "unscripted" came together, through the kids' observation, to find the "defect" in the opposing team's defensive scheme. The opposing team saw a scripted play forming that they had seen several times already that day, and they committed to defending against that play. By the time they realized that the play had morphed into something else, it was too late to adjust; the defect had already been exploited.
Isn't that what happens when we create and execute static, automated regression tests? They run. They occasionally find a defect or two in the beginning, but eventually the developers learn what the tests do and write their code to ensure that it passes those tests. Then one day a tester gets an idea to try something just a little bit different, and a critical defect is exposed. Sound familiar?
So then, why is it that during the course of a five-month football season, a bunch of 8-year-olds can figure out that "precisely scripted and executed plays" stop working after a while, that "randomly running around with a football" occasionally results in big plays, but usually results in big losses, AND that intelligently and deliberately modifying the play slightly in reaction to the defense can reveal the "critical defect" that ensures victory, when so many experienced, educated, adult software testers are still engaging in a debate that has been ongoing for 20-plus years about whether testing should be scripted or exploratory? The reason this debate has been going on for so long is because it's the wrong debate! It's not an either/or question. It's a "for what purpose" and/or an "under what conditions" and/or "how much of which will be most valuable this time" question.
I suspect that most people who test applications, or at least most people who test applications well, start by exploring the application to gain an understanding of what it does and how it works, without a whole lot of rules or boundaries being put on that activity. Then, armed with that knowledge, they start deliberately testing for certain things in a more or less scripted manner ("more scripted" might be by following exactly documented steps; "less scripted" might involve a list of user tasks to be accomplished without explicit steps to follow, or a short charter describing only the type of defect to look for). When we stop finding defects that way, we either quit or increasingly deviate from the scripts until we find new defects.
All this flooded into my head while I was cheering for Nicholas' improvisation and touchdown. Before I even stopped cheering, it occurred to me that maybe I hadn't been giving that coach enough credit.
(If you missed it, you can read my earlier column inspired by Nicholas' football team. Nicholas and Taylor also inspired me to write a column that touches on learning through exploratory play.)
----------------------------------------
About the author: Scott Barber is the chief technologist of PerfTestPlus, vice president of operations and executive director of the Association for Software Testing and co-founder of the Workshop on Performance and Reliability.
