Testing For iPhone App Memory Problems

Problems with memory management has got to be the #1 issue for iPhone app development, and presents one of the biggest challenges to testers. Our experience with testing over a half dozen iPhone apps in the past few months has reinforced the need for developers to nail down their memory management techniques early and for testers to create test cases that stress the way an iPhone app handles memory. So far the process has been bumpy but we've arrived at some basic memory management testing concepts along with recommendations for how developers can improve their memory management techniques.

Basically there are 2 types of memory problems to watch out for:

1. Primary Memory Leaks (in the "sandbox")
2. Secondary Memory Build-Up (out of the "sandbox")

Primary memory leaks are the most common and most troublesome to developers. These leaks create the mysterious crashes for iPhone apps in early development and are the hardest to document in a bug report, mainly because primary memory can change out from under the developer depending on other apps such as Safari and Mail. Safari can use up a lot of memory especially if multiple pages are open. Primary memory leak bugs seem to go away by simply restarting the application.

Secondary memory build-up creates situations that, over time, seem to create a permanently smaller amount of memory available for the app to run in, i.e. smaller primary memory. This does not come about because of memory leaks per se, and hence is harder to find with memory checking tools. As opposed to primary memory leaks, crashes or memory error alerts will seem reproducible for this type of memory problem. Consequently, the tester will write up repeatable steps leading up to the crash or memory alert, only to find out that, after restarting their iPhone, these steps will no longer be reproducible.

Our approach to rooting out these types of bugs is to first be able to identify them as categorized above, and then employ exploratory techniques to attempt to find reproducible steps. However, these memory issues can be elusive, and, as stated above, often lead to misleading bug reports. The main thing is to be able to communicate to development what type of memory issue you're seeing so they can use debugging tools to find the problem.

When we find these types of memory problems, our bug reports provide a summary, steps to reproduce, other apps that are running (i.e. Mail or Safari) and a result, e.g. app crashes or memory alert is presented. If the bug is not 100% reproducible, then we comment on where it mostly occurs. Our main effort is to write our bugs up so that a developer is convinced that it's a memory problem of some sort. Once convinced, then they can instrument their code to find it. But if you don't provide a bug report that describes the defect in the context of a memory problem, then the bug will most likely end up as "cannot reproduce" and be ignored.

We then recommend that the developers instrument their code to find the memory issues. But since we are testers, we can only go so far in what we recommend. It's been my experience that iPhone developers, when faced with memory issues, appreciate all the help they can get. Here are the links we send them to:

• Apple iPhone Developer Forum
  
• Finding Memory Leaks With The LLVM/Clang Static Analyzer
• CLANG Static Analyzer
• Finding iPhone Memory Leaks: A “Leaks” Tool Tutorial
• NSZombie and XCode Oh My!

Most developers visit the Apple iPhone Developers Forum and it's a great place for testers too. By reading the posts there, you get a feel for just how tough memory management is. But more importantly, you gain confidence that bugs you are seeing really can be fixed and should not be attributed to peculiarities of the iPhone. Apple developers write a lot of responses in those forums to help counter the claims that the OS is to blame for all these memory problems.

By understanding memory problems, testers can be authoritative in their defect reports. By knowing where to point developers for help, the test group becomes a resource for helpful investigation and research for tough memory problems.