Offshore resources are never a quick fix, especially for testing. A 2005 report from AMR Research found that it took between fourteen months and three years before the offshore testers had sufficient familiarity to be effective in finding the root cause of problems. Others have found that the lack of domain knowledge resulted in spurious defects being reported that actually increased development overhead due to review and response times. In fact, one organization identified that as many as 33 percent of reported issues were traceable to tester error.There are many reasons to involve external testing services. Sometimes companies need the expertise of an interoperability lab, other times they are attempting to reduce testing staff costs. Whatever the reason, one cost that should be calculated based on a known process is the Find/Fix Cycle Cost for defects. It's important to evaluate these costs not just based on the hourly rate of the offshore/outsource testing service provider, but on the total cost involved in finding and fixing defects. Ultimately, this cost involves all QA and developer resource costs.
The find/fix cycle represents the time required to find and fix one defect and you should know what that cost is to your company. In order to calculate this cost, you need to collect information along the way. I found a nice source for this calculation, as well as a number of other interesting testing topics, at Software Quality Consulting, Inc. Food For Thought. Steve Rakitin does a very nice job in describing this cost in the table below.
Estimate the find/fix cycle Cost
|T1 is time required to run a test that finds a potential defect||T1|
|T2 is time required to report a defect||T2|
|T3 is time required to determine that this is a defect||T3|
|T4 is time required to determine if the defect should be fixed||T4|
|T5 is time required to fix the defect||T5|
|T6 is time required to include fix in a build||T6|
|T7 is time required to run regression tests to determine if defect is really fixed and close out defect report||T7|
|Total time required…||TT = (T1 + … + T7)|
|LC represents fully loaded labor cost ($/hour) for engineering time (development and QA)||LC |
|C represents cost to find and fix one defect||C = LC * TT|
|TC represents the total cost to find and fix the defects found and fixed in the last release (F1)||TC = C * F1|
As you can see from this table, the "round trip" cost of a defect incorporates 7 different time elements that add up pretty quickly, especially if you have any inherent time delays such as dealing with remote test teams, external test staff turnovers, re-requests for defect characterization information, etc. And the final cost is not just that of the test team, but also incorporates the dev team costs.
My guess would be that T1 and T5 would be the largest time elements, while times T3 and T4, which require more verbal communication, could very well match T1 and T5, depending on the teams involved. It would be interesting, and most likely revealing, to measure these times and factor them into the costs associated with each project's QA and dev team composition.