Top 10 Tips For Testing iPhone Applications

This article started out as a Top 5 list but quickly grew as we researched all the tips and techniques we use at RTL for iPhone application testing. Most of these tips came from tracking down memory-related problems, which often resulted in defects that were very difficult to capture.

1. Accurately report available memory.  Many of the non-reproducible bugs you run into when testing iPhone apps are related to memory problems.  It's critical that you know and report available free memory before launching an application.  In all likelihood, the reproducibility of a crashing iPhone app bug is related to low memory conditions.  Consequently, a crashing defect may disappear when there's plenty of free memory.  In a previous article (Using Memory Sweep for iPhone App Testing) we described a tool that can be used to determine free memory.

2. Provide 'crash reporter' logs with your defect reports.  Each time an iPhone application crashes, a .crash file is created on your iPhone.  You can retrieve this file when you synch your iPhone with iTunes.  Here's a link that describes where those files are stored: iPhone Crash Logs

3. Spy on the app from the console.  iPhone apps will report application and system level warnings to the console.  You can view these warnings in real-time using Apple's iPhone Configuration Tool.  By knowing what's being reported when interacting with an app can help you refine the steps you need to reproduce tricky (and memory related) problems.

4. Test under low memory conditions.  This relates to #1 above.  You'll be able to tease additional crashing bugs if you force free memory to a very low level, e.g. < 2MB, before proceeding with your tests.  One way to do this is to open several Safari windows before you start your testing.

5. Screenshots, screenshots, screenshots. Nothing makes a UI bug stand out for a developer than when you send screenshots.  And with the iPhone's built-in screen capture, there's no excuse not to do this.

6. Provide useful defect characterization information.  Developers always like to have help in their debugging process, and useful defect characterization helps them narrow down the root cause of a bug.  If a crash happens under low memory conditions (see #1 and #4 above), then try it under conditions where there's lots of memory available, e.g. >40MB.  If a problem occurs under iPhone OS 2.2.x, then try it under 3.x.

7. Create connectivity problems. If you're testing an iPhone app that depends on internet connectivity, then test for degraded or unavailable connectivity. It's easy to make connectivity unavailable by simply turning on Airplane Mode.  To degrade connectivity, especially on Edge or 3G, employ some sort of metallic "shield" on top of your iPhone.

8. Boundary test data input. Wherever an iPhone app asks for text input, you have an opportunity to find a bug.  My favorite technique for this is to copy a huge amount of text, then paste it into each text field.  You'd be surprised at how this trips up some apps.

Additionally, we’ve been finding that application errors are generating when entering the following characters into text fields: !@#$%^&*()_. (Note: Holding down letters (A, E, I, O, etc) or symbols ($, !, &, etc) on the onscreen keyboard generates a keyboard popup that includes localized and 2-byte characters. These should also be entered into text fields.)

9. Gather up UDIDs (unique device identifiers) early.  This is a simple logistics task but seems to be one that becomes critical as the first build approaches.  And it's a hassle for the dev team to add new UDIDs and create new provisioning files as each new person wants to install an application during development.  Get the UDIDs of all know devices that will be used during testing and set a cut-off date for the addition of any new devices.  Check out Find UDID with a single click. You can also connect all your iPhones and iPods touches to your computer and use the iPhone Configuration Tool to collect UDIDs.

10. Employ background applications.  But the iPhone can only have one application running at a time, right? That's true for those of us that develop and test applications, but not for Apple.  Applications that continue running in the background on the iPhone are Safari, iPod and Mail.  And what about reminders and push notifications?  These "interrupters" can affect the behavior of an application under test.

Also, since iPhones and iPod touches are devices that users buy primarily to use as a phone or a music player, it’s important to test that an app can gracefully handle situations where the user receives a call or plays music from their music library (iTunes) while the app is running. We’ve seen issues here where apps aren’t smoothly multi-functioning in these areas.