Is It Possible To Achieve 99% Cumulative Defect Removal Efficiency?

I'm finding that many articles on testing continue to be pessimistic about outcomes. Furthermore, they contribute to the idea that "you can't get there from here" - you'll never find all the bugs so quit trying to use engineering rigor and metrics. In our recent article, How Important Is Coverage When Testing Software?, the experts that were interviewed, in answering the question "When is testing done?", gave answers such as "never", "two months after development is done", and "when producers decide". I get what they are all saying and actually agree with them on some levels. But I'd rather see conversations like this address continuous process improvement and not tend towards capitulation. Indeed, the practical should win over the theoretical, but without discussions in the theoretical realm, we really can't advance that which we do as testers.

And there's research that supports this pessimistic view. In fact, most forms of testing are not very efficient, and find only about 25% to 40% of the bugs that are actually present[1]. No wonder the experts from the article mentioned above, when asked when testing is done, answered never. But what we're missing here is the investigation of "most forms of testing." What is that? Well it must be what we're doing if we all somehow agree that you can't find all the bugs in a product. The important thing is, that no matter how you attempt to answer this question, it's not something we want to reinforce! The question to ask (and answer) is "What other forms of testing must we do in order to find ALL the bugs in a product?" As it turns out, there are folks that spend time trying to answer that question.

Defect removal efficiency originated in IBM in the early 1970’s as a method for evaluating testing effectiveness. Because this metric provided information that could be used for inspections and static analysis, as well as testing, it has become a general way of measuring the relative effectiveness of all forms of defect removal activity. Recent findings by companies such as IBM, Coverity, and Dovél Technologies show removal efficiency levels that often top 96%.

96% removal efficiency? Really? Yes, and there are expectations that defect removal efficiency can approach 99%. Now we're talking quality. How is this being done, i.e. what other forms of testing are being done to achieve this? As it turns out, there's no silver bullet. But there is an engineering process that contributes to this - coverage is expanded through different defect prevention and detection tests types. Here's the list of what those are:[2]

1. unit test
2. new function test
3. regression test
4. performance test
5. security test
6. usability test
7. system test
8. some form of external test with customers or clients, such as Beta test or acceptance test

The reason I say there's no silver bullet is because these are not new test types. The question "What other forms of testing are being done to achieve this?" I posed earlier was not about new forms of testing, but about combinations of testing types. The above tells me that, if you're not performing all of these tests, you have holes in your coverage and can expect to find less defects. What, then, keeps us from doing all these tests?

If you want to find out more, check out the article Predicting Test Cases and Test Team Size, by Capers Jones, President, Capers Jones & Associates LLC.

[1]  Predicting Test Cases and Test Team Size, by Capers Jones
[2] ibid