Thursday, October 15, 2009

Is All Testing Essentially Exploratory In Nature?

Having just been to STARWEST 2009, and actually seeing James Whittaker in person, exploratory testing is naturally on my mind. And today, as I was browsing my favorite testing blogs, I ran across this quote, supposedly from Boris Beizer:
All testing is essentially exploratory in nature.
Makes a great title for this article, and also calls into question of why exploratory testing is a separate test type. I mean, after all, if all testing is essentially exploratory, then should there really be an exploratory test type?

In my quest to shed some light on this subject and find an answer, I dug up some articles from well known testing experts that have shared their opinions on exploratory testing, e.g. Kaner, Bach, Robinson, Bj Rollison, and others. But first stop is Wikipedia:

Exploratory testing is an approach to software testing that is concisely described as simultaneous learning, test design and test execution. Cem Kaner, who coined the term in 1983, now defines exploratory testing as "a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project."... Exploratory testing

So naturally, we can't go any further until we hear what Caner has to say:

I coined the phrase "exploratory testing" 24 years ago, to describe a style of skilled work that was common in Silicon Valley. Naturally, the concept
has evolved since then, primarily as a way of focusing on how testers , p y y glearn about the product under test and its risks throughout the product's
lifecycle....
A Tutorial in Exploratory Testing

At the risk of starting an argument, I would think that James Bach may want to take some credit in popularizing exploratory testing:

Exploratory software testing is a powerful approach, yet widely misunderstood. In some situations, it can be orders of magnitude more productive than scripted testing. All testers practice some form of exploratory testing, unless they simply don’t create tests at all. Yet few of us study this approach, and it doesn't get much respect in our field. This attitude is beginning to change as companies seek ever more agile and cost effective methods of developing software....Exploratory Testing Explained

Now many of you know Harry Robinson to be an expert on model-based testing and probably have attended one of his presentations at STARTWEST or STAREAST. But did you know he has a fond appreciation for exploratory testing?

The mission of the Lewis and Clark expedition was to explore and map the territory of the Pacific Northwest. William Clark was the chief mapmaker for the expedition. He spent months perfecting his mapmaking skills before the expedition left St. Louis. As it turned out, later explorers and settlers used Clark’s maps for the next thirty years until newer, more detailed maps were drawn up. The mapmaking was an essential part of the trip. It would have made little sense for Lewis and Clark to return home with collections of trinkets and souvenirs of the trip but no map of the territory that would tell others about the land and how to best navigate through it. On the other hand, it would have been equally foolish to send scores of settlers into that unexplored territory -- much better to send a handful of scouts to survey the territory and report back. Think about exploratory and automated testing from this same perspective.... Exploratory Modeling

We certainly can't leave out James Whittaker, who has just released a book on exploratory testing. You can find plenty about this subject in his book, but here's a blog post that revealed a little more about what James thinks:

I just got finished talking (actually the conversation was more like a debate) to a colleague, exploratory testing critic and a charter member of the plan-first-or-don’t-bother-testing-at-all society....
explaining exploratory testing

I'm not sure that Bj Rollinson is a big advocate of exploratory testing. I've seen him take some digs at this subject before, but this article clearly shows an internalization of exploratory testing that transcends its most common use:

Much of the information about exploratory testing focuses on testing from an end-user perspective. Pundits of exploratory testing claim the approach is also useful from a white box test design approach, but I have yet to see any practical discussion or examples. But, professional testers use exploratory testing approaches all the time from a white box perspective to explore the code for untested paths. Professional testers learn about areas of the code that are at risk, and reactively design effective tests to evaluate previously untested or under-tested areas of the code.... Exploratory testing inside the box

So what do you think - does exploratory testing warrant all this attention as a separate test type, or, as Beizer says, is all testing exploratory?

6 comments. Add Comment.:

Joe said...

"Is All Testing Essentially Exploratory In Nature?"

I would have to say "No".

I've seen some testers execute scripted tests that clearly were doing absolutely no exploration at all (no matter how the term is defined).

Perhaps a more interesting question would be "Should all testing contain elements of exploratory testing?"

Ray Vizzone said...

Joe - that is a more interesting question. Do you think there needs to be some sort of "permission factor" that encourages testers to test outside the test case?

Joe Harter said...

Ray,

I prefer the way Michael Bolton and James Bach talk about Exploratory Testing in their Rapid Software Testing slides. They think of it as an approach to testing and it sits at the other end of a continuum from scripted testing.

This matches up very closely with what I've seen as a tester and test manager. Even if I'm given scripted test cases, I'm probably exploring the application with my eyes. I have not seen a script tell me everything that I'm supposed to or not supposed to see.

If you are doing zero exploration at then you might be checking instead.

Well done on tracking down all of those sources, btw. I'll have to refer back to that in the future.

- Joe Harter

Michael said...

Some good and useful work that people call "testing" is emphatically not exploratory in nature.

I call that work checking. It's an important thing to do, and many people call it testing (which they have every right to do). As long as they persist, there are forms of testing that are not exploratory. For an explanation, please see the Testing vs. Checking series of blog posts.

There was an minor argument (that I witnessed) between Kaner and Bach as to who coined the term: for several years, apparently, Bach had credited Kaner, and Kaner had credited Bach. Bach won the argument, resolving it upon re-reading the first edition of Testing Computer Software, definitively giving the nod to Kaner.

If anyone can dig up an actual reference to the Beizer quote, I'd be grateful to see it. It seems incredible, as Beizer has consistently expressed a near-anaphylactic reaction to those who tout exploratory approaches. Neither Google Web search, nor Google Groups search, nor Google Books (which indexes Beizer's books), returns a result on "+Beizer +exploratory" that looks anything like him saying "All testing is exploratory in nature." The sole exceptions are when B.J. Rollison is making the citation, "adamently" agreeing with Beizer. Try it!

Cheers,

---Michael B.

Ray Vizzone said...

Michael - You're right about the Beizer quote. Only references I find are attributed to B.J. Maybe he'll join in and let us know if this something he heard from Boris or read in one of his books. And thank you for sharing your expert insight with our readers.

For those of you that kindly withheld pointing out my error in not including a link to Michael Bolton's insights on exploratory testing, here's a link to DevelopSense where you can read about Rapid Software Testing, written by James Bach and Michael Bolton.

Cem Kaner said...

Boris had harsh things to say about "exploratory testing" because he believed that the advocates of exploration were advocating undisciplined, unsystematic work.

The problem that he was reacting to is still plenty visible today. Many people say that you should "explore" as if that is a magical method in itself. Much of what I read in blogs today treats "exploration" as an excuse to do little preparation for testing, little assessment of what has been done (to what level of completion/coverage), and little prioritization of the work to come. Boris saw that as sloppy and ineffective. I agree.

However, the essence of exploratory testing is learning. Boris was an advocate of skilled work, rooted in deep knowledge about how systems are built, designed to root out problems. So even though he had some harsh words about "exploratory testing", I think it is fair to say that he advocated much of what we would call exploratory today. It is hard for me to believe that he ever said "all testing is essentially exploratory" but easy for me to believe that he would agree with the statement that "all good testing is essentially exploratory if the exploration is skillfully and carefully done."

I'm going to skip debating whether "ALL" testing is exploratory. Clearly, the kind of regression testing that routinely repeats previous tests is not exploratory. Clearly, preplanned / prescripted testing done to demonstrate the product (e.g. testing focused on a standard or specification, that will ultimately be used to impress the customer or the regulator) is not exploratory. Are these "testing"? That's a vocabulary debate and I'll choose to skip it. (I think that Bolton's distinction between "testing" and "checking" is very useful, but whether it is "true" is a point that I don't need to debate here.)

The more interesting question is whether MOST testing should be exploratory (or whether it is permissible / professionally-reasonable) for ANY testing to be exploratory in style.

If you want to look for enemies of exploration, don't look to people like Beizer, who advocated application of high levels of skill and knowledge to testing.

If you want to look for enemies of exploration, look to the trivializers of the field. Look to the people who think that anyone can learn enough about testing in a week to be "certified" in the field or to people who think that we can organize the testing effort by having smart test designers writing detailed tests for cheap labor ot execute. Or to the people who make their money or fame today by paying lipservice to "exploration" while pushing shallow approaches that might expose a few bugs quickly but don't drive the tester to learn deeply enough to get to the serious underlying problems that show up after the surface problems have been scraped off.

-- Cem