Automated Web Services Testing - An Interview With Matt Krapivner of SmartPilot Software

As more and more companies face decisions about automating their testing, it's important to understand some of the pitfalls of automation and explore methods that avoid those pitfalls. Likewise, we need to find mature open source testing tools that help bend the test automation curve in the right direction. In previous articles, we've written about model-based and keyword-driven testing, each promising to reduce test automation script maintenance costs while increasing effectiveness (coverage). As I've searched for answers in this areas, I've sought examples of how others have approached test automation as a way of benchmarking good ideas. Recently, I've had the good fortune to meet a practitioner who has pursued new and more effective approaches to test automation. His name is Matt Krapivner and his consultancy business has provided a very impressive test automation framework to a local Web 2.0 company.

We interviewed Matt recently and talked to him about the new approach he's using with his current client and how it's a step up from traditional test automation practices.

Q: Matt, you started out with a Computer Science academic background and since have been working in the area of Quality Engineering. What got you interested in that area versus software development and programming?

Ray, my first quality engineering engagement was actually as a developer on a quality engineering team. I believe some companies call this Software Engineer in Testing, but in a nutshell the team was developing testing software. I started as an individual contributor, but ended up managing the team of 7 engineers locally plus another 7 offshore. I saw QE as an oft-undervalued field with an opportunity to make a great impact on the perception of the company’s products in the marketplace. It is also quite fascinating because at different times you have to wear multiple hats, such as a developer at one time, a tough customer advocate at another, sometimes a product marketing/management hat, and so on. The opportunities for improving the product and making an impact are there, it’s just that many companies often overlook QE and do not get the deserved output from their teams, which in turn impacts their perception of QE overall.

There are two different approaches to testing: Quality Assurance and Quality Engineering. In my opinion, those are very different. One will give you an assessment of the state of the system AFTER that system is put together. The other will help you make sure that your system is ENGINEERED with the highest-quality possible. Let me give your readers an analogy. Say you are building a retaining wall and 2 years after it’s built the wall comes down in a mudslide. Using the traditional QA approach, you would state “the wall fell down because the pressure was too large”. If I was using QE methodologies, I would hire an engineer to build the wall according to the building code to begin with, to make sure that mudslides will not affect it. That’s a fundamental difference, because the further you get in the product lifecycle, the more expensive it will get to fix the mistakes. In my retaining wall example (which I actually just finished building in the backyard), I made a mistake of not putting in the drainage behind it. It cost me more than the initial total construction cost to make an opening in the wall, add the pipe, and close the wall. If I did what was necessary from the beginning, it would only add 5-10% to the construction cost. Instead, by trying to save a few dollars and rushing the construction crew I ended up spending double the amount allocated initially. The same can be said about software development – the longer you wait to find and address an issue, the more you will end up paying to correct the mistakes.

Q: Can you tell us about your first test automation project and what you learned from it?

My first automation engagement was at Palmsource, where I managed a team which was responsible for design and development of an automated test harness for the Palm OS API. In a way, our team played a role of 3rd-party developers early on in the software development lifecyle (SDLC). This was quite fascinating as we got to develop software using a cutting-edge OS before everyone else did. We also received a lot of visibility in the company since as “early adopters” we were able to make informed suggestions for features, and avert a few blunders along the way before they hit the market. As we drove internal adoption, our team started receiving praises from other groups which used our automated tests to validate their changes, small and large. Very quickly, the automated tests were used as a means of certifying customers’ devices for deployment field. This meant that no device could ship with Palm OS or bear the Palm OS logo until it successfully passed our automated tests.

The interesting observation I have made since then in most companies I have worked with, is that very often the testing software ends up being more complex than the actual application under test. There are multiple reasons for that, from the need to cover and verify various edge cases, to just being able to confirm the results. In some cases, especially when you have a hardware/software interaction, this is not trivial at all. One also has to learn where to draw the “line of trust”. Let me explain. Assume that you are testing a PDA application and you need to verify that certain text appeared in a certain position on the screen. While it sounds trivial, verifying the correct result often is anything but. There are typically several solutions. We can call an internal API to query it for the text at certain coordinates, or a text in a specific field. But can you “trust” this API? What if it doesn’t exist to begin with? What if the API itself is buggy? In that case, your test will be meaningless. Another option is to ask your developers to give you a custom “hook” into the display buffer, but most likely this request will not be on top of their priority list. Also, there is a chance it may introduce another point of failure. Yet another option is using a software simulator, if one exists. We actually had to write our own software simulator for testing purposes at one of the companies where I ran the QE group. The point is that there are often many ways of verifying the information in your application under test, and the “easiest” choice may not necessarily be the right one. The challenge then becomes to persuade upper management to invest time and money into building a test system, where an immediate ROI may not necessarily be realized, but rather will result in intangible benefits such as improved product quality, customer satisfaction, etc.

Q: I understand that you have recently provided test automation development services for several local companies. What tools have you used and what was your assessment of the long term maintenance of the frameworks and test scripts you've delivered?

It’s true that there are many test automation systems, and there is no “perfect choice” that would work anywhere. Over the course of the last few years, I have designed a house-grown test automation system; used a hybrid house-grown and commercial tool (Automated QA’s TestComplete); used a commercial tool in combination with open source ones (Squish and Selenium); and finally developed what I believe to be the future of automated testing – a system based on FitNesse.

I will not dive deeply into the benefits and challenges of using commercial tools, there are countless articles and books written on the topic. However, I would like to mention a few points from my experience as a manager/director of QE teams, and as a consultant working on implementing these systems. When selecting an automation tool, one of the main things a company should look at is whether or not a specific tool will work with their application. If it will not, all other benefits become a moot point. Another significant factor is price. In some cases, it will run into tens of thousands of dollars for the tool alone. As a QE director, I had to restrain my teams quite often because the price of the tool was just prohibitive given our budget. Most often though, there is no technical reason to select those tools anyway as there often is a less-expensive alternative. With any commercial tool though, a person responsible for test automation will have to know at least a scripting language, such as Perl or Python. Ideally, that person will be able to program in a higher-level language as well, such as C++ or Java. As a result, your automation system will most likely be built fairly well. The major limiting factor to the tool’s adoption then becomes its scalability, or how quickly can new tests be developed to ensure adequate test coverage. It is, of course, less of a factor for large companies with deep pockets as these companies can most likely just organically grow the test automation team. For startups and smaller companies though, this becomes a major hurdle and automation efforts sometimes may not succeed, or may not provide the desired ROI.

At my last consulting engagement, my client approached me with a similar problem and suggested that I implement a system based on FitNesse. FitNesse is a Wiki-based system where tests are written in natural English language in the form of a Wiki table. Underneath, there is a “fixture” (in Fit-speak) which does the “translation” between English and the underlying tool. That’s right, we can still plug in any test automation tool with FitNesse (well, as long as it exposes some API). With this system, the automated tests now can be written by anybody in the company, including manual testers, product management folks, even a VP of Engineering. It is also easy to answer questions such as “what’s our testing coverage for feature XYZ”, “why did the latest ABC test failed”, and so on. Granted, there still needs to be someone to integrate FitNesse with a testing tool of choice, but this approach provides companies with the much-desired scalability of their testing efforts and a tool with no price tag and ample documentation.

I have integrated FitNesse with Selenium Grid for my client’s web application, and with CruiseControl for continuous build integration. We can now simultaneously test over 30 various targets (different OS/browser combinations), and get results within 15-20 minutes for a test run that spans the entire product in 10 languages. It is easy for anyone now to add to the automated test collection.

Q: For those of us that have tried to bring more automation to testing, we've each run into the problem of "brittle" automated test cases, as well as the need to generate more test cases early on in the development cycle. How has FitNesse solved this problem?

FitNesse is actually a great tool for this, since it allows non-programmers to write test cases which are immediately automated by definition. Most likely, the product managers in this case will not even have to change their logic since they will write the test cases in the form of a “use case”, using more or less the same English language to define a feature. So when the feature is defined, it is put into FItNesse. When the feature is implemented, the test case can be executed to provide immediate response to the team. This becomes even less labor-intensive when a continuous build integration system is utilized. In fact, it is so easy for developers to utilize it to achieve the ever-elusive target of “test-driven development”, that there are no reasons for them NOT to use it…

Q: Given what you've said about FitNesse, particularly the ability for non-programmers to write test cases using a Wiki, how easy is it for someone to produce a test case?

As easy as it is for someone to edit a Wiki page. The tests literally look like:

|user clicks button|Submit|
|page reloads in less than|30|seconds|
|page contains text|Hi Ray|
…
If a person can write a manual test case, they can write a test case in Wiki. If a person can write a use case in Word, they can write a test case in Wiki. You get the idea.

Q: What would you say is the most compelling reason to adopt FitNesse and what challenges are there for organizations to take this path?

The most compelling reason for companies to adopt FitNesse is its ability to provide so much valuable output with so little effort. Granted, it is a relatively new tool and sometimes you will stumble but in my experience the hurdles one will face with FitNesse are not any greater than with any other commercial test automation tool. In fact, the major challenge will likely be organizational in nature as it will require a slight change in the way people operate and contribute to the testing efforts. A FitNesse endeavor is much more likely to be successful if folks from multiple departments contribute.

Q: How extensible is FitNesse, i.e. what if my web service uses a lot of AJAX or components that don't work well with Selenium?

As long as Selenium can handle it, so can FitNesse. Remember that FitNesse operates with “fixtures”, which provide translation between English (the language of FitNesse) and Selenium API. 

I chose to use Java API for Selenium, because FitNesse is written in Java and it is technically easier to use Java for the fixture. I was comfortable with Java, so I chose not to change something I did not have to. Any language can be used though, and if your automation tool of choice can only speak Python, for example, then you simply write your fixture in Python.

AJAX is an interesting case though, and the limitations one will face will have nothing to do with FitNesse, but rather with Selenium’s ability to handle AJAX. In my case, we handle AJAX by a combination of standard Selenium APIs, custom Selenium APIs that I wrote specifically for my client’s application, and custom hooks that developers have provided on the server-side. However, all of this is hidden from the tester since their tests are written in plain English. That’s the beauty of FitNesse – it really hides implementation details from a tester who is only concerned with, for example, whether or not a text appeared in the right spot on the page. We provide a custom action for the testers, and they simply do not and should not care how we implement this action behind the scenes.

Q: What would you recommend to those that want to adopt FitNesse for their test automation projects and where can they go for help in getting a FitNesse framework up and running at their company?

First of all, find a test automation tool that you would use without FitNesse. Then, find out if this tool has an API that you can call from FitNesse. Some tools do, and some do not. Some tool companies will be willing to open up their API for you, and some may even be willing to write a sample fixture for you that you would be able to extend. Some may prohibit you from calling their tools from anything besides their own proprietary interface. When in doubt, just ask the tool vendor.

As for FitNesse, there is plenty of easily-digestible documentation that comes along with the distribution of FitNesse. If you search online, you may even find ready-to-use “fixtures” for the most popular open source automation tools, such as Selenium for example. Those fixtures will give you a very good start.

You can reach Matt Krapivner by email at matt@smartpilotsoft.com.