Top 10 Tips For Testing iPhone Applications

This article started out as a Top 5 list but quickly grew as we researched all the tips and techniques we use at RTL for iPhone application testing. Most of these tips came from tracking down memory-related problems, which often resulted in defects that were very difficult to capture.

1. Accurately report available memory.  Many of the non-reproducible bugs you run into when testing iPhone apps are related to memory problems.  It's critical that you know and report available free memory before launching an application.  In all likelihood, the reproducibility of a crashing iPhone app bug is related to low memory conditions.  Consequently, a crashing defect may disappear when there's plenty of free memory.  In a previous article (Using Memory Sweep for iPhone App Testing) we described a tool that can be used to determine free memory.

2. Provide 'crash reporter' logs with your defect reports.  Each time an iPhone application crashes, a .crash file is created on your iPhone.  You can retrieve this file when you synch your iPhone with iTunes.  Here's a link that describes where those files are stored: iPhone Crash Logs

3. Spy on the app from the console.  iPhone apps will report application and system level warnings to the console.  You can view these warnings in real-time using Apple's iPhone Configuration Tool.  By knowing what's being reported when interacting with an app can help you refine the steps you need to reproduce tricky (and memory related) problems.

Mobile Device Application Testing Coverage Expands

The mobile device app space is expanding quickly. Testing for compatibility across different devices within a platform family and across platforms is becoming increasingly complex. Where will development and testing organizations draw the line?

Microsoft luring iPhone developers for Zune apps Developers being offered money to port their apps? If they are successful in luring iPhone developers, you can bet that these apps will need to be tested on both platforms. Are we entering another phase of supporting apps on multiple platforms?

Second webOS-Based Smartphone May Be Delayed Probably news to most that Palm's second webOS device is called the Palm Eos. Was to be available this fall and supposedly smaller than the Palm Pre. Will there be a significant update to webOS when the new devices arrive?

Microsoft Renaming Windows Mobile as Windows Phone Windows Mobile, or WinMo, is expected to change names (WinPho?), probably to reflect Microsoft's new emphasis on its Zune and the mobile device market, particularly the iPhone market.

Dell Confirms Mobile Device Development Based on the rumors, this will expand the Android app market. Looks a lot like the iPhone in this picture.

Samsung's TouchWiz App Development Kit Hobbles Into the Light Samsung's TouchWiz app dev kit is out and it looks like it immediately expands compatibility testing across 3 different smartphone operating systems, Windows Mobile, Symbian and Samsung's own.

iPhone App Testing With iSimulate

The iSimulate iPhone app is interesting in so many ways. If you've read the articles posted about it (here's one from my favorite Apple blog, TUAW: iSimulate brings iPhone apps to the big screen), you probably were impressed by the idea of using your iPhone to control an iPhone app while watching it on a separate display. Playing a game on a large display while controlling it through an iPhone, as if the iPhone is some sort of game controller accessory, is intriguing.

And for the many iPhone app developers that are looking for a way to create marketing material of their app in action, iSimulate is probably just what they've been looking for. Recording your iPhone app in real time on a higher resolution display with actual motion inputs and then being able to later edit the recording has got to be better than the various video cam recordings one finds on YouTube.

Using Memory Sweep for iPhone App Testing

As mentioned in a previous article here (Testing For iPhone App Memory Problems), chasing down memory problems when doing iPhone app testing can be difficult. In particular, providing developers with concrete evidence that a memory issue is or is not their problem is a big challenge. Recently, while testing an iPhone app that was consistently presenting low memory alerts, we needed an independent memory assessment and discovered Memory Sweep, written by Gary Fung. It's a free app and works great for our testing purposes. Here's the information it provides you:


The numbers you see for Memory Status will help you diagnose memory problems when testing iPhone app. We used it for memory stress testing. We needed to force a low memory condition to see how an iPhone app behaved. Memory Sweep validated this process. We have a way of setting up iPhones/iPod touches so that memory is forced to be about 3-4MB. Once we do this, we launch Memory Sweep to make sure free memory is as low as we expected (you can see that free memory in the image above is 4.21MB), then quit and launch the iPhone app under test. In this way we can characterize our defects in a precise way so that the developer has confidence in the results.

When you read the reviews on Memory Sweep, you'll find that are very critical about the app. The main complaint is that the app is named Memory Sweep on the iTunes App Store and ends up being called "Scan" on your iPhone. From what I can tell, the critical reviews have been mostly about cosmetics and not functionality. There are plenty of positive reviews, particularly from those that appreciate the capability to force more memory on their iPhones. My main complaint is that this app only runs under iPhone OS 3.0. This will force us to find a similar app to run on older iPhone OS versions for our testing.