Developing applications for Apple's iPhone and iPod touch is an exciting and interesting experience. Last year RTL established an iPhone Apps Test Lab and 
set about to develop its own app (Phone Alarm - available at the iTunes App Store) to understand what developers go through in order to better serve their testing needs. In the middle of creating our iPhone Apps Test Lab, we were fortunate enough to work with the developers at NewToy, the creators of Chess With Friends*. These guys were great to work with and their app was fun to test.Here's how we planned and executed the testing.
The first thing we did was outline a compatibility test matrix. You would think that the compatibility list would simply be an iPhone and iPod, but it ended up growing much larger - not unlike most compatibility matrices. We had to take into account both the first generation iPhone and the iPhone 3G, as well as the iPod touch (and now the iPod touch 2G). On top of that, we needed to apply the connectivity variables: Edge, 3G, 802.11, as well as no connection. This made for a rather large coverage matrix.
Once we had the compatibility matrix laid out, we composed a Basic Acceptance Test procedure (BAT) that covered all the main features as well as some corner cases. Then it was just a matter of running a BAT for each compatibility configuration in the test matrix.
Along the way, we did run into challenges and issues unique to testing on the iPhone platform. The following 3 areas are worth sharing:
- Installation Logistics
- Defect Characterization
- Debugging Information
Installation Logistics
Because of the way Apple controls the installation of applications on the iPhone, you can't simply receive an app and install it locally to your iPhone. You have to provide the developer with the unique IDs for each device. This is described on the developer portal and requires that you connect your device to a computer and use iTunes to find the ID. Here are the steps for doing this:
1. Launch iTunes.
2. Connect your device to your computer.
3. Select the device in the Devices list.
4. In the Summary pane, click the Serial Number label. It changes to Identifier.
5.Choose Edit > Copy.
6. Submit your device identifier to the developer.
Once we collected all the device IDs, we sent them to the developers and they in turn had to create a certificate that was installed on each test device - and that required connecting the device to a computer and using iTunes for the installation.
Defect Characterization
For the most part, the defect characterization process for testing an iPhone App is no different than what you would use for any other testing. The challenge with iPhone Apps is in the number of environmental combinations you'll encounter. Not only are you testing across multi-generational devices, but each device can have multiple connectivity options. And when you're testing an app like Chess With Friends, which requires internet connectivity, verifying defects on each option adds to the overall defect characterization process and tester time.
Debugging Information
Providing debugger info when testing iPhone Apps can be tricky. In fact, unless you use the SDK, there are no simple options for providing standard debug logs. However, depending on the nature of the app, there are a number of things you can do. First and foremost, learning how to take screenshots is imperative. It's easy enough to do on the iPhone platform (hold down the Home button while momentarily pressing the power button). Secondly, knowing if an app is storing data elsewhere, e.g. a back-end server, allows you to use that data as debugger info so long as you can isolate it. If the app you're testing uses online account info, then you can help the developers isolate needed debug info by reporting user account info and time stamp data with each defect.
Overall, the process for testing an iPhone application is no different than for any other product. However, learning how iPhone apps are developed and reading the posts at the developers forum will better inform you on what testing issues may arise.
*Chess with Friends was the winner of the 2008 Best Free or Ad Supported Game
