Tuesday, November 24, 2009

Testing Android Apps - The Compatibility Testing Matrix Continues To Grow


Android, an open source operating system, must avoid the fate of J2ME, an open source mobile applications platform. Open source is great, until everyone splinters off into their own world. That’s what happened to J2ME, and a number of frustrated Android developers are now saying that there is a risk Android will follow the same path.
-Michael Arrington, A Chink In Android’s Armor, 10/11/2009
Android developers continue to voice their concerns, and in a more recent article, "Android's rapid growth has some developers worried", there's even more evidence that development and testing compatibility issues have a potential to grow out of control. Developers identify a slew of compatibility concerns:
  • 3 versions of the OS (Android 1.5, 1.6 and 2.0)
  • Differences across handsets
  • Custom user interfaces, e.g. HTC Sense, Motorola Blur, etc.
  • Phones with and without camera flash
  • Keyboard differences (physical vs. GUI)
  • Different carrier ROMs
  • Non-standard commands for triggering standard functions, e.g. GPS on Hero vs. other Android devices
  • Uncooperative graphics drivers
These differences have created significant compatibility issues, so much so that developers are creating separate builds and maintaining separate code for different Android devices. The best analogy for the problems faced by Android developers can be found in Michael Arrington's article:
Imagine if Windows developers had to build different versions of their applications for different PC manufacturers. Or even different versions for various models by a single manufacturer. That’s what some Android developers are saying they are facing now.
I'm not so sure Android development and testing is any different than with most software development - witness the issues developer face with supporting their web apps across different browsers. But I do think that their expectations are reasonable: an OS developer (in this case Google) needs to do all they can to standardize how the application layer will behave. Certainly Apple's iPhone has set the bar high in this case and ultimately developers will decide how much they are willing to put up with.

Monday, November 23, 2009

Intelligent Automated Testing - A Look At FitNesse

For those of you that read our interview with Matt Krapivner, you probably gained additional insight on the FitNesse automated testing framework. Having met with Matt recently, I've seen just how useful FitNesse can be, and so I wanted to share some videos I've found about FitNesse. This first one is a quick tutorial showing how to download and run FitNesse. This includes a demo of the FitNesse acceptance tests. The steps are straightforward, and when you're done, you'll have a fully functional FitNesse setup - ready for hacking and learning.

Friday, November 20, 2009

Model-Based Testing For Web Applications With TestOptimal - (Part 4) Reports



Previous articles on Model-Based Testing For Web Applications With TestOptimal:
Part 1 - Preparing The Model
Part 2 - mScript
Part 3 - Generating Sequences
Today we conclude this series of articles on using TestOptimal for web applications testing by looking at the reporting features. TestOptimal provides detailed information on all aspects of each model-based testing project. The main reporting elements are:
  1. Execution Status
  2. Execution Progress
  3. Testing Coverage: includes State/Transition traversals and # of each that are covered or left un-covered
  4. Transition Sequences: a description of each state-to-state transition
  5. Execution Details: includes exceptions and transition response times in milliseconds, as well as statistical analysis of the 3-sigma range for each response time on state/transitions based on historical MBT executions
  6. Debugging: this information is provided in several panes - Console, mScript Log, Server Log and Selenium Log
Below is a report-by-report look at the information TestOptimal provides, along with a brief description for each report.

This is the Monitor Tab, which provides information on execution and coverage:



The Sequence Log tab shows the test sequence (state and transition traversal sequence) generated from the model with the selected sequencer. Use this transition log to find when and where the MBT exception occurred. You may also use this transition log to capture the test sequence and feed it to your testing frame work:



The Stats Tab shows MBT execution stats collected by TestOptimal, both in-progress MBT execution or historical executions:



The client log messages are written to this console log window. You can adjust the debug level using the drop-down list on top of the console:



The mScript Log window displays the log messages generated by your mScript and errors and exceptions triggered by the mScript:



The Server Log dislays the log generatd by TestOptimal sever. Ths is the place where you can find the errors and exception stack traces:



If you are using the bundled Seleninum RC server, this log window will let you access Selenium RC sever log file:


This concludes our 4 part series on using TestOptimal for model-based testing of web applications.  Although we covered the main steps that relate to model-based testing - create the model, customize the mScript, generate and execute test sequences, and analysis results - there's a lot more to be discovered with TestOptimal.  For more information on other features (e.g. mCase sequences, java support), different editions, and how to extend TestOptimal, visit their website here.  I highly recommend that you look at their various tutorials and examples to see just how extensive this tool is for model-based testing.

Thursday, November 19, 2009

Windows 7 In The News - A Month After Launch

Windows 7 has been out for a month now and most everyone agrees that it's an improvement over Vista.  As for testing with Windows 7, it looks like there's more to this new version of Windows than simply a better implementation of Vista.  Aside from the security issues described in the links below, and the fact that the federal government cannot use Windows 7 yet, there are some pretty interesting unknowns.  The two that impressed me most were the hidden tool for examining energy usage (might be useful during testing) and the way it handles crashes - Windows 7 actually modifies how it allocates memory after several crashes, which would affect how reproducible a bug might be.  See if you agree:

In an inauspicious beginning to the week, the first zero-day bug for Windows 7 has emerged. The bug touches on Microsoft's Server Message Block (SMB) program -- specifically, SMBv1 and SMBv2 on Windows 7 and Windows Server 2008 R2. Microsoft has issued a security advisory describing workarounds, but says most users would be protected from attacks by blocking two ports at the firewall. Windows 7 Gets Its First Bug

Windows 7 has a hidden tool that will help you by examining your energy use and suggesting what you can do in order to improve it. To get a power efficiency report, Computerworld’s Preston Gralla provided the following steps...
Tips and Tricks for Windows 7

If an application crashes a few times, Windows will automatically apply a shim that intercepts its memory operations. FTH will over-allocate memory, and keep a copy of freed memory so that attempts to re-read it will succeed. Windows 7's dirty secrets revealed

More than a dozen federal agencies, including the White House and all branches of the military, are testing Windows 7, according to Microsoft. But it may be another six months before agencies can move ahead with Windows 7 deployment because a government-mandated security standard hasn't been finalized. Government Use Of Windows 7 Hinges On Security Spec

Spotty support for networked all-in-one printers in particular may be a testing, QA, and certification issue; some notebook drivers aren't yet available either. Windows 7's Spotty Driver Support

Last week, security vendor Sophos published a blog post in which it said that Windows 7 was vulnerable to 8 our of 10 of the most common viruses. Microsoft has responded to these test results, which are a classic case of "scare 'm and they'll fall in line". Microsoft: Sophos Windows 7 Malware Test 'Sensationalised'

Wednesday, November 18, 2009

Why Would Anyone Want To Implement Automated Testing?

Moreover, why does anyone even want to test their software at all? They just want it to work. But most know better, and so they test. But does management care if testing is manual or automated? Should testers care if their testing is manual or automated? My guess is that management cares about automated testing if they think testing is going to cost less and provide faster results. Tester's will consider automated testing if it increases coverage while making them more efficient.

The reason I'm writing about this subject is because of articles posted on Bj Rollison's blog about calculating ROI for automated testing. In Rollison's first article on the subject, Measuring Test Automation ROI, he says this about ROI:
I have 2 essential problems with ROI calculations within the context of test automation. First, if the business manager doesn’t understand the value of automation within a complex software project (especially one which will have multiple iterations) they should read a book on managing software development projects. I really think most managers understand that test automation would benefit their business (in most cases). I suspect many managers have experienced less than successful automation projects but don’t understand how to establish a more successful automation effort. I also suspect really bright business managers are not overly impressed with magic beans.
Harsh comments but probably a realistic view of things. But it makes things difficult for anyone wanting to bring automated testing into their QA processes, given that there's a need to explain benefits versus cost. The follow-on Part 2, further condemns ROI calculations to being a silly waste of time. What Rollison does offer is why he automates tests, as well as points to a set of questions useful in deciding when to automate. Rollison identifies 4 reasons he chooses to automate tests:
  1. Free up my time,
  2. Re-verify baseline assessments (BVT/BAT, regression, acceptance test suites)
  3. Increase test coverage (via increased breadth of data or state variability),
  4. Accomplish tasks that can’t easily be achieved via manual testing approaches.
In addition to his list, he praises Brian Marick's paper on When Should a Test Be Automated. Rollison lists Marick's 3 key questions one should ask before automating:
  1. Automating this test and running it once will cost more than simply running it manually once. How much more?
  2. An automated test has a finite lifetime, during which it must recoup that additional cost. Is this test likely to die sooner or later? What events are likely to end it?
  3. During its lifetime, how likely is this test to find additional bugs (beyond whatever bugs it found the first time it ran)? How does this uncertain benefit balance against the cost of automation?
I think these questions and the process by which one answers them, is an effective way to inform one's decision as to automate or not. Neither Rollison or Marick offer answers, nor should they. As Rollison puts it: "Unfortunately, there is no single cookie-cutter approach in deciding what tests to automate. Different projects have different requirements and expectations, and, of course, not all tests are equal."

What I take away from these questions is a process for making logical decisions about automated tests with the ability to explain why the automation effort will benefit the organization's testing goals - if it improves testing it should be worth the investment, so long as that improvement fits within your goals.

Tuesday, November 17, 2009

USB Interoperability And Compatibility Testing Must Have - P55

If you conduct USB interoperability testing, you'll want to add systems that use the Intel P55 Express chipset.  We're finding out that the P55's USB controller can be problematic with certain devices.  Here are a number of reports worth looking into:

Intel's quality control team who seem to have somehow missed an issue between Apple's app carrier deluxe and the P55 Express chipset's USB controller. iPhone and Windows 7 don't play nice, Intel P55 chipset to blame.

The Intel P55 Express chipset snafu that caused iPhones to lose their syncing minds has now been remedied -- at least by one motherboard maker. Gigabyte fixes iPhone sync issue with BIOS update


Asus and MSI, which also use the P55 express on their motherboards, haven’t issued any updates yet, but hopefully will soon. Busted Windows 7 + Intel P55 Express iPhone Sync Fixed for Gigabyte Motherboards — Asus and MSI Pending

"Our leading theory is a BIOS or system configuration issue, but we are still investigating," Intel said Monday. Intel comments on iPhone sync glitch

iTunes 9 on Windows 7 x64 Has Problems with iPhone

Monday, November 16, 2009

Getting Started With TestOptimal

For the last few weeks we've been writing about model-based testing using TestOptimal. Today we have a video tutorial on getting started with TestOptimal. The video covers:
  • Installation
  • Screen Navigation
  • Main Features, and
  • Working with Demo Apps.

Friday, November 13, 2009

Model-Based Testing For Web Applications With TestOptimal - (Part 3) Generating Sequences


So far we've prepared our TestOptimal project with the model we created in Part 1 (Model-Based Testing For Web Applications With TestOptimal - (Part 1) Preparing The Model) and the mScript we crafted in Part 2 (Model-Based Testing For Web Applications With TestOptimal - (Part 2) mScript). Now it's time to see how this all comes together by generating our test sequences.

TestOptimal offers 5 sequence modes. They are:
  • Optimal - uses the proven optimization algorithms to generate the optimal test sequence to achieve 100% coverage of your application. Because only the minimum number of test steps have to be repeated, the test execution completes in least amount of time.
  • Random - generates the test sequence by randomly selecting a transition from the current state. You can set conditions when to stop the test execution
  • Greedy - works the same way as Random except that it prefers un-traversed (un-tested) transitions over already traversed transitions. This helps to achieve the desired coverage with less number of test steps.
  • mCase Optimal - generates the optimal test sequences to cover all states and transitions contained in mCases. Use mCaseSequencer if you just want to test certain scenarios or parts of your application. You may also use mCaseOptimal with virtual users to simulate concurrent users running certain test scenarios.
  • mCase Serial - is a variation from mCaseOptimal. It differs from mCaseOptimal by preserving the order of transitions/states in mCase(s) in the test sequences it generates.
Each of these sequence types describe how TestOptimal will traverse our graph, and consequently how much coverage we will achieve. I tried out the first three sequence types (mCase Serial and Optimal won't be covered in this article) and took them for a "dry run" to see what was produced.

For each sequence type I selected, I received a report on Testing Coverage from the Monitor tab:

Greedy





Random





Optimal





As you can see, the Monitor tab shows how many traversals were done for each state and transition, as well as how many states/transitions were covered or not.  Since I set up the sequencer to seek 100% coverage, there were no un-covered states/transitions.  I was surprised to see the Random sequence achieve a greater efficiency over the Greedy sequence.  And I was very impressed to see how efficient the Optimal sequencer was.

In addition to the information above provided in the Monitor tab, TestOptimal provides several other views for each run.  There's a Sequence Log, which shows the last 1000 traversals and lists each state and transition traversed:



And a Stats tab, which provides state and transition coverage information with performance times:



After trying out these sequences in "dry-run" mode, I decided to run the test sequences against the actual target - maps.google.com.  It worked flawlessly.  After pressing the Run button, TestOptimal launched a browser window and began running each of the several hundred test sequences.  It was fun to watch and, more importantly, demonstrated the power of model-based testing.  Next week we'll take a look at some of the details within TestOptimal reports and analyze our results.

Thursday, November 12, 2009

Learning How To Test Mobile Device Applications (Part 4) - iPhone

Learning How To Test Mobile Device Applications (Part 1) - Palm webOS
Learning How To Test Mobile Device Applications (Part 2) - Android
Learning How To Test Mobile Device Applications (Part 3) - BlackBerry
Today we conclude this HOWTO series with the iPhone. Like the other mobile device platform companies, Apple provides everything we need to create a simple app so that we will have an understanding of how to test those apps. And it meets our cost goal - the SDK is free (if you know where to look) and you can create and test your iPhone app without having to buy an iPhone.

If you visit the iPhone Developer Program page, you'll end up being asked to join the program for a fee.  And that makes sense if you want to be able to create and publish your iPhone app, but we just want to learn what it takes to develop and test an app by downloading the SDK for free.  What you want to do is sign up as a Registered iPhone Developer. And if you have already have an iTunes, Apple Online Store or MobileMe account, you can use your existing Apple ID to sign in and become a registered iPhone developer. It's easy and takes a few seconds to do. Once you're registered, then you can access the iPhone Dev Center to download the SDK.

After you register and then log in to the iPhone Dev Center, you can download the SDK here. Once downloaded and installed, then you're ready to create your app and run it on the iPhone simulator. And luckily, there's a tutorial for that - Tutorial: Hello, World! This tutorial takes through these steps:
  1. Create the Project
  2. Write the Code
  3. Run the Application
Seems simple enough, and fortunately the tutorial has pictures and clear descriptions for each step.  When you're done, I recommend you setup Apple's iPhone Configuration Tool and run your app with the console window open.  There you'll be able to log application and system level warnings.  By knowing what's being reported when interacting with an app can help you refine the steps you need to reproduce tricky (and memory related) problems.

Wednesday, November 11, 2009

Technology Advisory: Safari 4.0.4

Apple released Safari 4.0.4 today.

This update is recommended for all Safari users and includes improvements to performance, stability, and security including:
  • Improved JavaScript performance
  • Improved Full History Search performance for users with a large number of history items
  • Stability improvements for 3rd-party plug-ins, the search field and Yahoo! Mail
For detailed information on the security content of this update, please visit this site: http://support.apple.com/kb/HT1222.

Safari 4.0.4 is available via Software Update and Apple's Safari download page. Versions for Mac OS X Snow Leopard, Leopard, and Tiger, Windows 7, Vista, and XP are available.

Technology Advisory: Mac OS 10.6.2 Released

Apple released Mac OS X 10.6.2, the second maintenance update for Snow Leopard. The update is available via Software Update and the company's software downloads page.

The update contains several bug fixes, including a fix for an issues that caused data to be deleted when using a guest account.

The 10.6.2 Update is recommended for all users running Mac OS X Snow Leopard and includes general operating system fixes that enhance the stability, compatibility, and security of your Mac, including fixes for:
  • an issue that might cause your system to logout unexpectedly
  • a graphics distortion in Safari Top Sites
  • Spotlight search results not showing Exchange contacts
  • a problem that prevented authenticating as an administrative user
  • issues when using NTFS and WebDAV file servers
  • the reliability of menu extras
  • an issue with the 4-finger swipe gesture
  • an issue that causes Mail to quit unexpectedly when setting up an Exchange server
  • Address Book becoming unresponsive when editing
  • a problem adding images to contacts in Address Book
  • an issue that prevented opening files downloaded from the Internet
  • Safari plug-in reliability
  • general reliability improvements for iWork, iLife, Aperture, Final Cut Studio, MobileMe, and iDisk
  • an issue that caused data to be deleted when using a guest account
For detailed information on this update, please visit this website:http://support.apple.com/kb/HT3874.

Monday, November 9, 2009

Software Testing and Quality Assurance Rules of Thumb

New this week: THE NUMBER OF WHITE-BOX TESTS REQUIRED TO OBTAIN SUFFICIENT COVERAGE OF A MODULE IS ADEQUATELY DETERMINED BY ITS CYCLOMATIC COMPLEXITY.
Rule of thumb - The earliest citation comes from Sir William Hope’s The Compleat Fencing-Master, second edition, 1692, page 157: "What he doth, he doth by rule of thumb, and not by art." The term is thought to originate with wood workers who used the length of their thumbs rather than rulers for measuring things, cementing its modern use as an inaccurate, but reliable and convenient standard.
As testers, we often use rules of thumb throughout a project. For example, we sometimes use total number of expected defects during test planning and then compare actual defects per hour found versus what we would expect, during test execution. Each of these rules of thumb aids us in managing the information we deal with as testers and QA managers.

It would be nice (and useful) to have a collection of these rules of thumb in one place, each documented with examples. And that's my goal: to list as many software testing rules of thumbs in this article. I'll update it each week and re-post it. I look forward to reader comments and additions to this list and hope to make it a reference you each bookmark and refer to often.

THE NUMBER OF WHITE-BOX TESTS REQUIRED TO OBTAIN SUFFICIENT COVERAGE OF A MODULE IS ADEQUATELY DETERMINED BY ITS CYCLOMATIC COMPLEXITY. One common testing strategy, espoused for example by the NIST Structured Testing methodology, is to use the cyclomatic complexity of a module to determine the number of white-box tests that are required to obtain sufficient coverage of the module. In almost all cases, according to such a methodology, a module should have at least as many tests as its cyclomatic complexity; in most cases, this number of tests is adequate to exercise all the relevant paths of the function. [ NIST Special Publication 500-235: Structured Testing: A Testing Methodology Using the Cyclomatic Complexity Metric .]

MORE THAN 40% OF ALL SOFTWARE APPLICATIONS ARE RELEASED WITH BETWEEN ONE AND 10 CRITICAL DEFECTS. And managers are fully aware. Three-quarters of deployed apps have between one and 10 critical errors, and again, bosses know about it. [Software Is Deployed, Bugs and All, SD Times.]

80% OF DEVELOPMENT COSTS ARE CONSUMED BY SOFTWARE DEVELOPERS IDENTIFYING AND CORRECTING DEFECTS. A study commissioned by the United States Department of Commerce's National Institute of Standards and Technology (NIST) found that software defects cost the U.S. economy almost $60 billion annually. The study also found that about 80 percent of development costs are consumed by software developers identifying and correcting defects. [The business value of software quality.]

ABOUT 7% OF DEFECT REPAIRS WILL THEMSELVES ACCIDENTALLY INJECT A NEW DEFECT. A bad fix is a secondary defect accidentally injected in a bug repair. In other words, a bad fix is a failed attempt to repair a prior bug that accidentally contains a new bug. On average, about 7 percent of defect repairs will themselves accidentally inject a new defect, although the range is from less than 1 percent to more than 20 percent bad fix injections. [Capers Jones. "Software Cost Estimating Methods for Large Projects©"]

DEVELOPERS INJECT FEWER DEFECTS WHEN DESIGNING (2.0 DEFECTS PER HOUR) THAN WHEN THEY DESIGN WHILE CODING (4.6 DEFECTS PER HOUR). The practice of designing while coding is error prone. From data on 3,240 programs written in Personal Software Process (PSP)SM courses, the SEI has found that experienced developers inject fewer defects when designing (2.0 defects per hour) than when they design while coding (4.6 defects per hour). If you want low-defect designs, you must produce those designs, instead of just creating them while coding. [WATTS S. HUMPHREY. "A Personal Quality Strategy", Stage 2 – Detailed-Design Quality.]

NONREPRODUCIBLE OR AD HOC TESTING IS OF LITTLE OR NO USE. One of several basic rules of testing compiled from Myers (1976), Priest (1988), and Pullum and Doyle (1998). [W.R. Blischke, D.N.P. Murthy. "Reliability", Section 9.5.1, pp. 311.]

CHAOS WITH NO MEASURES TO TELL THE ORGANIZATION WHAT'S GOING ON LEADS TO … 3 IN 10 PROJECTS BEING CANCELED, 5 IN 10 OVERRUNNING SCHEDULE AND/OR BUDGET BY NEARLY DOUBLE, AND ONLY 1.6 IN 10 MAKING THEIR DEADLINES AND BUDGETS. Many organizations struggling with process chaos claim that they don't have time for software measures, which are sometimes perceived as a bureaucratic Dilbert exercise. But getting metrics in the face of deadlines, and using them to manage that pressure and reduce the chaos, is key to successful software development. [Johnson, Jim. "Chaos: The Dollar Drain of IT Project Failures." Application Development Trends (January 1995), pp. 41-47.]

THE AVERAGE TIME TO FIND AND FIX DEFECTS IS 10 TO 20 HOURS. "One way to think about quality is to consider how the process would change as a function of defect fix times. For example, programmers generally think that it takes only a few minutes to fix defects in test. They base this on their experience with most of the defects they find in unit testing. In system test, however, the time to find and fix defects typically extends to many hours or even days. While most of these defects are fixed rather quickly, some take much longer. The average time to find and fix each defect is generally 10 to 20 or more hours." [Watts Humphrey, Learning from Hardware: Design and Quality, p.107]

ABOUT 80 PERCENT OF AVOIDABLE REWORK COMES FROM 20 PERCENT OF THE DEFECTS. That 80 percent value may be lower for smaller systems and higher for very large ones. Two major sources of avoidable rework involve hastily specified requirements and nominal-case design and development, in which late accommodation of off-nominal requirements causes major architecture, design, and code breakage. [Software Defect Reduction Top 10 List]

EXPECT BETWEEN 6.5 AND 9 DEFECTS PER KLOC DURING SYSTEM TEST. Based on data from the Jet Propulsion Laboratory (JPL) from an internal report that listed all the defects found in the system testing of three spacecraft. These data were taken after the programs were developed, compiled, unit tested, and integration tested. [Watts Humpherey]

THE RELATIVE COST OF FIXING DEFECTS DURING TESTING IS 15 TIMES GREATER THAN DURING DESIGN. As reported by the Systems Sciences Institute at IBM. Additionally, the cost to fix an error found after product release was four to five times as much as one uncovered during design, and up to 100 times more than one identified in the maintenance phase. [IBM Systems Sciences Institute]

REQUIREMENTS, DESIGN, AND CODE REWORK COSTS 40 TO 50 PERCENT OF THE TOTAL COST OF SOFTWARE DEVELOPMENT. "If you can prevent defects or detect and remove them early, you can realize a significant schedule benefit. Studies have found that reworking defective requirements, design, and code typically consumes 40 to 50 percent of the total cost of software development (Jones 1986). As a rule of thumb, every hour you spend on defect prevention will reduce your repair time from three to ten hours. In the worst case, reworking a software requirements problem once the software is in operation typically costs 50 to 200 times what it would take to rework the problem in the requirements stage (Boehm and Papaccio 1988). "[Steve McConell]

TESTERS WILL FIND, ON AVERAGE, 1 DEFECT EVERY 4 HOURS DURING BLACK BOX TESTING. Based on average defect detection rates for 6 different JPL projects as reported in "An Analysis of Defect Densities Found During Software Inspections," Proceedings of the Fifteenth Annual Software Engineering Workshop. More information can be found in Grady's Practical Software Metrics For Project Management And Process Improvement. [Grady]

THE NUMBER ONE CAUSE OF DEFECTS IS SPECIFICATIONS. "Software Testing", R .Patton, p.17. [Patton]

TESTING DONE WITHOUT MEASURING CODE COVERAGE TYPICALLY EXERCISES ONLY ABOUT 55 PERCENT OF THE CODE. Robert Grady of Hewlett-Packard reports that testing done without measuring code coverage typically exercises only about 55 percent of the code (1993). See page 615 of Code Complete. [McConnell]

Friday, November 6, 2009

Model-Based Testing For Web Applications With TestOptimal - (Part 2) mScript

This week we will create the scripting component of our TestOptimal model-based testing project. But before we do, I need to refer back to last week's article and point out that my initial model contained a "bad state" - one that was inaccessible from any other state. I found this out by importing my model into TestOptimal and having it check out the many states and transitions of my model. You can do a dry run using TestOptimal, which will generate a test sequence that covers all transitions in your model. If you have any isolated states or states that are not reachable (as was the case for me), an error popup will be displayed, which indicates you have missed some transitions thus making some states not reachable. TestOptimal found my "bad state" and I quickly fixed it. It turns out that a typo contributed to this error. The new (and correct) model has been posted to last week's article (Model-Based Testing For Web Applications With TestOptimal - (Part 1) Preparing The Model).

TestOptimal uses a scripting format called mScript. mScript is a simple, XML based scripting to drive your web applications. It has only 14 tags and is very easy to learn and use for writing your testing scripts.  It's also very powerful.  Here's a list of the 14 tags:
  1. action - executes the action specified in the code attribute, e.g. to click the button named "Order", you would enter code="click('name=Order')". See mScript function. You may also create your own user defined functions.
  2. assert - asserts the condition specified. if condition fails throw the MBTException.
  3. dataset - defines a dataset. Its body contains tags that populates the dataset.
  4. db - defines a jdbc data source to be used by sql tag.
  5. field - defines a field in a row and sets its value.
  6. if - performs condition check and if evaluates to true then execute the command tags in the body.
  7. log - logs a message to TestOptimal server log file.
  8. mbt - container tag to hold one or many scripts to control MBT execution.
  9. rowdata - adds a row to the data set. Must be within a dataset tag and contain at least one field tag.
  10. script - defines a script method to be executed at state entry/exit or transition prep, action or verify. The body of the script tag is the command tags. This tag is equivalent to the method declaration in a java class.
  11. sql - declares the sql statement to populate dataset.
  12. state - container tag to hold scripts for a state including transition.
  13. transition - container tag to hold scripts for a transition.
  14. while - executes the script in its body if the condition specified is true. 
To write your mScript, you'll need an XML editor.  I use Notepad++ (Windows) or TextWrangler (Mac OS). Here are the steps I used to create my mScript:

1. Generate a skeleton mScript.  To do so, just click on mScriptTab, the mScript displayed is the mScript we will use. Right-mouse click on the script content and select "View Source" option, copy and paste the code into your XML editor.

2. Gather UI identifiers from the target web interface (google maps).   I used the FireFox Selenium IDE to do this.  By recording each action from my model, e.g. click Search maps, the Selenium IDE captured the identifiers I needed.

3. Add the specific script actions that pertain to each transition in the model. There were 11 transitions in my model, so I had 11 scripts to write. Within the skeleton script generated in Step #1 above, there are 2 sets of function stubs under each state and 3 sets of function stubs under each transition. To keep things simple, I did not use any of the state functions, instead concentrating on the transition function actions. The transitions for my model involved only two actions: clicking on objects (e.g. buttons) or selecting from a pop-down menu.  Hence I needed to write two action scripts:
  <action code="$click(element_locator)"/>
  <action code="$selectOption(element_locator,label)"/>
"element_locator" is a string to locate the link on the page. Again, I used the Selenium IDE FireFox plug-in to find the identifiers for each element locator.  To refresh our memory on what we are testing, here's a screenshot of the limited UI we selected for these articles:

Referring to the above graphic, here are the mScript action codes I used for the transitions in my model:

Click Show options
<action code="$click('id=showss')"/>
Click Remove options
<action code="$click('id=hidess')"/>
Select option 'All Results'
<action code="$selectOption('id=mrtbox','label=All results')"/>
Select option 'Locations'
<action code="$selectOption('id=mrtbox','label=Locations')"/>
Select option 'Businesses'
<action code="$selectOption('id=mrtbox','label=Businesses')"/>
Select option 'User-created content'
<action code="$selectOption('id=mrtbox','label=User-created content')"/>
Select option 'Related maps'
<action code="$selectOption('id=mrtbox','label=Related maps')"/>
Select option 'Mapped web pages'
<action code="$selectOption('id=mrtbox','label=Mapped web pages')"/>
Select option 'Real estate'
<action code="$selectOption('id=mrtbox','label=Real estate')"/>
Click Search Maps
<action code="$click('id=q-sub')"/>
Click Google maps logo
<action code="$click('id=logo')"/>

4. Modify the skeleton mScript code with the action codes from above (and remove any unneeded tags as desired).

5. Deploy your mScript code. To do so, copy or save to the /script/ folder in the TestOptimal home directory using the same name as the model name, e.g.  "google_maps.xml".

We're now ready to exercise our model using the mScript that has been tailored to our test target (I've included the script below).  Next week we'll generate test sequences and start testing.

<xml version="1.0">
 <mscript>
  <state id="n1">
   <transition event="show_options">
    <script type="action">
     <action code="$click('id=showss')" />
    </script>
   </transition>
  </state>
  <state id="n2">
   <transition event="remove_options">
    <script type="action">
     <action code="$click('id=hidess')" />
    </script>
   </transition>
   <transition event="select_all">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=All results')" />
    </script>
   </transition>
   <transition event="select_locations">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Locations')" />
    </script>
   </transition>
   <transition event="select_bus">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Businesses')" />
    </script>
   </transition>
   <transition event="select_user">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=User-created content')" />
    </script>
   </transition>
   <transition event="select_related">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Related maps')" />
    </script>
   </transition>
   <transition event="select_mapped">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Mapped web pages')" />
    </script>
   </transition>
   <transition event="select_real">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Real estate')" />
    </script>
   </transition>
   <transition event="search_maps">
    <script type="action">
     <action code="$click('id=q-sub')" />
    </script>
   </transition>
   <transition event="click_google_maps">
    <script type="action">
     <action code="$click('id=logo')" />
    </script>
   </transition>
  </state>
  <state id="n3">
   <transition event="remove_options">
    <script type="action">
     <action code="$click('id=hidess')" />
    </script>
   </transition>
   <transition event="select_all">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=All results')" />
    </script>
   </transition>
   <transition event="select_locations">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Locations')" />
    </script>
   </transition>
   <transition event="select_bus">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Businesses')" />
    </script>
   </transition>
   <transition event="select_user">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=User-created content')" />
    </script>
   </transition>
   <transition event="select_related">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Related maps')" />
    </script>
   </transition>
   <transition event="select_mapped">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Mapped web pages')" />
    </script>
   </transition>
   <transition event="select_real">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Real estate')" />
    </script>
   </transition>
   <transition event="search_maps">
    <script type="action">
     <action code="$click('id=q-sub')" />
    </script>
   </transition>
   <transition event="click_google_maps">
    <script type="action">
     <action code="$click('id=logo')" />
    </script>
   </transition>
  </state>
  <state id="n4">
   <transition event="remove_options">
    <script type="action">
     <action code="$click('id=hidess')" />
    </script>
   </transition>
   <transition event="select_all">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=All results')" />
    </script>
   </transition>
   <transition event="select_locations">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Locations')" />
    </script>
   </transition>
   <transition event="select_bus">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Businesses')" />
    </script>
   </transition>
   <transition event="select_user">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=User-created content')" />
    </script>
   </transition>
   <transition event="select_related">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Related maps')" />
    </script>
   </transition>
   <transition event="select_mapped">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Mapped web pages')" />
    </script>
   </transition>
   <transition event="select_real">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Real estate')" />
    </script>
   </transition>
   <transition event="search_maps">
    <script type="action">
     <action code="$click('id=q-sub')" />
    </script>
   </transition>
   <transition event="click_google_maps">
    <script type="action">
     <action code="$click('id=logo')" />
    </script>
   </transition>
  </state>
  <state id="n5">
   <transition event="remove_options">
    <script type="action">
     <action code="$click('id=hidess')" />
    </script>
   </transition>
   <transition event="select_all">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=All results')" />
    </script>
   </transition>
   <transition event="select_locations">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Locations')" />
    </script>
   </transition>
   <transition event="select_bus">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Businesses')" />
    </script>
   </transition>
   <transition event="select_user">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=User-created content')" />
    </script>
   </transition>
   <transition event="select_related">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Related maps')" />
    </script>
   </transition>
   <transition event="select_mapped">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Mapped web pages')" />
    </script>
   </transition>
   <transition event="select_real">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Real estate')" />
    </script>
   </transition>
   <transition event="search_maps">
    <script type="action">
     <action code="$click('id=q-sub')" />
    </script>
   </transition>
   <transition event="click_google_maps">
    <script type="action">
     <action code="$click('id=logo')" />
    </script>
   </transition>
  </state>
  <state id="n6">
   <transition event="remove_options">
    <script type="action">
     <action code="$click('id=hidess')" />
    </script>
   </transition>
   <transition event="select_all">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=All results')" />
    </script>
   </transition>
   <transition event="select_locations">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Locations')" />
    </script>
   </transition>
   <transition event="select_bus">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Businesses')" />
    </script>
   </transition>
   <transition event="select_user">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=User-created content')" />
    </script>
   </transition>
   <transition event="select_related">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Related maps')" />
    </script>
   </transition>
   <transition event="select_mapped">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Mapped web pages')" />
    </script>
   </transition>
   <transition event="select_real">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Real estate')" />
    </script>
   </transition>
   <transition event="search_maps">
    <script type="action">
     <action code="$click('id=q-sub')" />
    </script>
   </transition>
   <transition event="click_google_maps">
    <script type="action">
     <action code="$click('id=logo')" />
    </script>
   </transition>
  </state>
  <state id="n7">
   <transition event="remove_options">
    <script type="action">
     <action code="$click('id=hidess')" />
    </script>
   </transition>
   <transition event="select_all">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=All results')" />
    </script>
   </transition>
   <transition event="select_locations">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Locations')" />
    </script>
   </transition>
   <transition event="select_bus">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Businesses')" />
    </script>
   </transition>
   <transition event="select_user">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=User-created content')" />
    </script>
   </transition>
   <transition event="select_related">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Related maps')" />
    </script>
   </transition>
   <transition event="select_mapped">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Mapped web pages')" />
    </script>
   </transition>
   <transition event="select_real">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Real estate')" />
    </script>
   </transition>
   <transition event="search_maps">
    <script type="action">
     <action code="$click('id=q-sub')" />
    </script>
   </transition>
   <transition event="click_google_maps">
    <script type="action">
     <action code="$click('id=logo')" />
    </script>
   </transition>
  </state>
  <state id="n8">
   <transition event="remove_options">
    <script type="action">
     <action code="$click('id=hidess')" />
    </script>
   </transition>
   <transition event="select_all">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=All results')" />
    </script>
   </transition>
   <transition event="select_locations">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Locations')" />
    </script>
   </transition>
   <transition event="select_bus">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Businesses')" />
    </script>
   </transition>
   <transition event="select_user">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=User-created content')" />
    </script>
   </transition>
   <transition event="select_related">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Related maps')" />
    </script>
   </transition>
   <transition event="select_mapped">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Mapped web pages')" />
    </script>
   </transition>
   <transition event="select_real">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Real estate')" />
    </script>
   </transition>
   <transition event="search_maps">
    <script type="action">
     <action code="$click('id=q-sub')" />
    </script>
   </transition>
   <transition event="click_google_maps">
    <script type="action">
     <action code="$click('id=logo')" />
    </script>
   </transition>
  </state>
 </mscript>
</xml>

Thursday, November 5, 2009

Technology Advisory: FireFox 3.5.5 Released

Mozilla has released Firefox 3.5.5, which includes the following changes:

• Fixed several stability issues.

For more information please see the Firefox 3.5.5 release notes.

Learning How To Test Mobile Device Applications (Part 3) - BlackBerry

In this HOWTO series, we've covered two popular mobile device platforms, webOS and Android, in these articles:
Learning How To Test Mobile Device Applications (Part 1) - Palm webOS
Learning How To Test Mobile Device Applications (Part 2) - Android
Today we'll look at BlackBerry. As we've stated in previous articles, our goal is learn enough from developing simple apps on each platform so that we have a better understanding of the platform as testers. And we want to accomplish this without having to buy each mobile device or any development software. Fortunately, developer offerings from RIM meet these requirements, albeit with less hipness than webOS and Android.

Their "Getting Started" info is somewhat buried on their BlackBerry Java Development Environment page, and most of the info is provided as PDFs.  Which is OK, but not as convenient as having online reference web pages, particularly since some of these PDFs load slowly.  Basically, here's what you'll need to get started:
  1. Intel® Pentium® 4 Processor (minimum 3 GHz)
  2. Microsoft® Windows VistaTM, or Windows® XP 
  3. Java® SE Development Kit (JDK) version 5 or version 6 Version 6 is required if you
    are using BlackBerry MDS-CS for debugging
  4. Eclipse IDE for Java Developers version 3.4.0
  5. Blackberry tools (Java Application Development tools)
They do say you can develop on Linux or Mac OS X, but the "easiest and best supported is the Windows 32bit platform."  Interesting, especially since the minimum Windows systems needs to be 3GHz.

Here are the step-by-step instructions for setup:
  1. Download the JDK at http://java.sun.com/javase/downloads index.jsp. The current version is JDK 6 Update 12 which includes the JRE
  2. Select required platform and language, agree to the licence agreement, and proceed with the download
  3. Download the Eclipse IDE at http://www.eclipse.org/downloads/
  4. Select Eclipse IDE for Java Developers and the version you require
  5. Save the file on your computer
  6. Download the BlackBerry tools at http://na.blackberry.com/eng/developers/resources/devtools.jsp. Here you will need to download the BlackBerry Plug-in for Eclipse, as well as the BlackBerry JDEs
  7. Download all available JDEs (currently 4.3 to 4.7) so that you can test your applications on most current BlackBerry devices. You will need the plug- in and at least one JDE.
  8. Once you click on the link you want to download you will have to sign in (or register if you are using this service for the first time) 
  9. Use this video to learn how to install and configure the BlackBerry plug-in for Eclipse.
The above will get you to the point where you can create your first app.  RIM provides a detailed, step-by-step tutorial on creating a "Hello World" app.  This includes instructions on how to run it on their BlackBerry simulator.  Follow their instructions and you'll have your first app up and running and ready to test. And you'll find a wealth of testing and debugging information in their How to Debug and Optimize guide.

Wednesday, November 4, 2009

iPhone App Project Managers - How Did You Test Your App?

iPhone application development is exploding (How Hot Is The iPhone Application Development Market?). And users are demanding better quality - no longer is it acceptable for an iPhone app to crash unexpectedly with the explanation that apps crash every so often because of the iPhone OS. Moreover, with more-and-more iPhone and iPod touch models to choose from, developers face a compatibility testing challenge. So how are project managers testing their iPhone apps?

We'd like to hear from iPhone application developers and managers who have navigated their way onto the App Store and can share with us the problems they faced in testing their apps, as well as tips and recommendations.

If you are interested in helping out, please leave a comment or email us. We will send you questions by email and publish your interview here.

Tuesday, November 3, 2009

Top 10 Software Engineering Risk List

This list is from an interesting article in SoftwareTech News and represents project assessments over many years. The author, Donald J. Reifer, has had the benefit of conducting risk assessments over several decades and he put together a list that identified root causes and impacts. Here are the Top 10 Risks he identified:
  1. Impossible Schedules
  2. Inadequate Budgets
  3. Poor Requirements
  4. Staffing Shortfalls
  5. Lack of Operational Concept Document
  6. Failure to Emphasize Testing Early Enough in the Process
  7. Poor Measurement
  8. Subcontract management focuses on contract issues
  9. Over-Reliance on Commercial Off-The-Shelf (COTS)
  10. Licensing Headaches
On the testing side of things, I can definitely relate to several in the list, particularly about not testing early enough and poor requirements. QA and testing always suffers from those problems, and yet they continue to be at the top of the list. The impact listed for failure to test early emphasized "Not enough time and resources available to pull together a meaningful test program...".

For more on this article, visit Common Risk Patterns

Monday, November 2, 2009

Testing Applications In Window 7's XP Mode (HOWTO)

A lot has been said about XP Mode for Windows 7. And I thought it was included in all versions of Windows 7. Turns out it's not. You have to install XP Mode and it only works on Ultimate, Professional and Enterprise. This video covers the key features with XP Mode as well as a brief instruction on how to obtain and install it. At least Microsoft provides a fully functional copy of Windows XP for free with the download.