Basically there are 2 types of memory problems to watch out for:
- Primary Memory Leaks (in the "sandbox")
- Secondary Memory Build-Up (out of the "sandbox")
Secondary memory build-up creates situations that, over time, seem to create a permanently smaller amount of memory available for the app to run in, i.e. smaller primary memory. This does not come about because of memory leaks per se, and hence is harder to find with memory checking tools. As opposed to primary memory leaks, crashes or memory error alerts will seem reproducible for this type of memory problem. Consequently, the tester will write up repeatable steps leading up to the crash or memory alert, only to find out that, after restarting their iPhone, these steps will no longer be reproducible.
Our approach to rooting out these types of bugs is to first be able to identify them as categorized above, and then employ exploratory techniques to attempt to find reproducible steps. However, these memory issues can be elusive, and, as stated above, often lead to misleading bug reports. The main thing is to be able to communicate to development what type of memory issue you're seeing so they can use debugging tools to find the problem.
When we find these types of memory problems, our bug reports provide a summary, steps to reproduce, other apps that are running (i.e. Mail or Safari) and a result, e.g. app crashes or memory alert is presented. If the bug is not 100% reproducible, then we comment on where it mostly occurs. Our main effort is to write our bugs up so that a developer is convinced that it's a memory problem of some sort. Once convinced, then they can instrument their code to find it. But if you don't provide a bug report that describes the defect in the context of a memory problem, then the bug will most likely end up as "cannot reproduce" and be ignored.
We then recommend that the developers instrument their code to find the memory issues. But since we are testers, we can only go so far in what we recommend. It's been my experience that iPhone developers, when faced with memory issues, appreciate all the help they can get. Here are the links we send them to:
- Apple iPhone Developer Forum
- Finding Memory Leaks With The LLVM/Clang Static Analyzer
- CLANG Static Analyzer
- Finding iPhone Memory Leaks: A “Leaks” Tool Tutorial
- NSZombie and XCode Oh My!
By understanding memory problems, testers can be authoritative in their defect reports. By knowing where to point developers for help, the test group becomes a resource for helpful investigation and research for tough memory problems.