EXPERT RESPONSE
There are several IT-related definitions of the term factory. Wikipedia defines a factory as the term often used to refer to any method whose main purpose is creation of objects (see Factory method pattern). There is a testing utility called Test Factory (testfactory.com), as well. But most people, when referring to a test factor model, are referring to an outsourcing model wherein all testing is outsourced in an "over the wall" or black box manner. As you weren't specific in your question about references to any other test factory, I'll assume you mean the test factory outsourcing model.
In my experience, there are two major models for outsourced testing: The collaborative or extension model, and the factory model. In the collaborative testing model, a company engages a testing partner, leveraging the partner's key strengths while retaining significant insight and responsibility for the testing effort. For instance, in my role as Test Manager at Circuit City Stores, we engaged in a collaborative test effort with IBM India. We had several test engineers working in India, executing test cases. They were connecting to the IT network via VPN (virtual private network) and they were running cases as we directed. We met daily (sometimes more frequently) to track and manage process on the test effort. There was frequent communication, and the testing vendor received significant direction from me and my team, as well as from on-site IBM staff. This high degree of interaction exemplifies the collaborative model.
In a test factory model, the solution is much more turn-key. In general, a product is built in-house and then "thrown over the fence" to a factory-model testing vendor. Rarely does this happen in V1 products, and it also doesn't frequently happen without development also being handed to the vendor. The vendor sets a price, and the client and vendor agree on a completion date. Finally, the client establishes a clear bar for acceptance -- i.e., guidance for saying when the vendor is done. The vendor takes the released code, acceptance tests to prove they can thoroughly test it, and then they carry out the pre-determined testing. This effort is characterized by being independent from the customer -- very little interaction between vendor and customer takes place. On the project completion date, the vendor returns to the client with feedback -- generally in the form of bug reports and so forth.
Each model has its benefits and its drawbacks. The collaborative model provides the customer with a great deal of insight into the project -- its status, challenges and risks. It also costs a lot in terms of management overhead. I've been engaged in collaborative projects where my test leads have spent up to 50% of their time assisting vendors. However, for a high-risk project, or a project with significant change, this is often the best model. The collaborative model is a must for companies evaluating vendors for potential long-term relationships because it exposes to the customer how the vendor approaches IT challenges, decision making and so forth. In a low-trust environment, like a vendor testing a new technology, it's also very beneficial.
On the other hand, the factory model is fantastic in that it's turn-key. It's like leaving your home for the weekend and returning to a new kitchen without experiencing the dust, noise, or pain caused by the work. In a high-trust situation, this solution works out well for each party. The customer can leave it and forget it. The vendor can approach testing with their own methodologies. It moves efficiently. This model is fantastic for testing projects such as compatibility or regression testing. It also works well for benchmark testing or for independent validation testing. The drawbacks, however, are many. Because the testing methodologies carried out by the test factory vendor are black box, as a manager I always have that nagging feeling at the back of my mind. If the project is complex or difficult to understand, it could ultimately cost significantly more than expected -- both in terms of time as well as (unplanned) resources the customer has to dedicate. Even with frequent status updates, there is a risk that the vendor is behind schedule yet hides it until the very last minute.
|
|
 |
You have to pick the correct model based on the situation you're in -- what are your timelines, what are the characteristics of the project, what kind of historical relationship do you have with the vendor, etc. If you evaluate that input carefully, you should be able to reach the right decision and end up with the most effective and efficient approach to your outsourcing relationship.
|
