iPhone Development Basics - Memory Management Part 5

This week is the final video from Mark Johnson on iPhone memory management. In case you missed any of the previous videos, here's a list of the articles we've posted with Mark's first 4 memory management videos:

iPhone Development Basics - Memory Management Part 1
iPhone Development Basics - Memory Management Part 2
iPhone Development Basics - Memory Management Part 3
iPhone Development Basics - Memory Management Part 4

Learning How To Test Mobile Device Applications (Part 3) - BlackBerry

In this HOWTO series, we've covered two popular mobile device platforms, webOS and Android, in these articles:

Learning How To Test Mobile Device Applications (Part 1) - Palm webOS
Learning How To Test Mobile Device Applications (Part 2) - Android

Today we'll look at BlackBerry. As we've stated in previous articles, our goal is learn enough from developing simple apps on each platform so that we have a better understanding of the platform as testers. And we want to accomplish this without having to buy each mobile device or any development software. Fortunately, developer offerings from RIM meet these requirements, albeit with less hipness than webOS and Android.

iPhone App Project Managers - How Did You Test Your App?

iPhone application development is exploding (How Hot Is The iPhone Application Development Market?). And users are demanding better quality - no longer is it acceptable for an iPhone app to crash unexpectedly with the explanation that apps crash every so often because of the iPhone OS. Moreover, with more-and-more iPhone and iPod touch models to choose from, developers face a compatibility testing challenge. So how are project managers testing their iPhone apps?

We'd like to hear from iPhone application developers and managers who have navigated their way onto the App Store and can share with us the problems they faced in testing their apps, as well as tips and recommendations.

If you are interested in helping out, please leave a comment or email us. We will send you questions by email and publish your interview here.

Learning How To Test Mobile Device Applications (Part 2) - Android

Last week we introduced this series on learning how to test mobile device applications based on the idea that developing a simple mobile app on each platform will help you understand how to test them (Learning How To Test Mobile Device Applications (Part 1) - Palm webOS). And we want to do this for free and without having to buy an actual device. That's worked for webOS from Palm and will work with this week's platform - Android.

And as we recommended in Part 1, testers should start by downloading and installing the SDK and building one of the included apps. For the Android platform, the steps for building your first app is explicitly laid out at Android Developer's SDK page. The instructions provided take you through the following steps:

iPhone Development Basics - Memory Management Part 4

So far we've seen 3 videos by Mark Johnson on iPhone memory management:

iPhone Development Basics - Memory Management Part 1
iPhone Development Basics - Memory Management Part 2
iPhone Development Basics - Memory Management Part 3

And we've been using these videos to understand iPhone application memory management and gain some insight has to why iPhone apps crash.  In today's video, Mark describes a memory management programming convention used by iPhone developers for managing objects - either they're in the Autorelease pool or they are not. It's a very technical video but worth listening to because Mark warns of things not to do otherwise an app will crash - something worth being curious about when talking to your dev team.

Suggested questions for Part 4:

(4:02min) Why does returning a reference to a deleted object cause a crash? Why would that happen?
(5:17min) What does it mean to "own" an object versus having it be part of an Autorelease pool?

Learning How To Test Mobile Device Applications (Part 1) - Palm webOS

I recently visited Palm and our friends there gave us a Palm webOS book - signed by the author Mitch Allen (VP, Software CTO at Palm)!  Very cool.  While reading this book I thought I would use this opportunity to start a HOWTO series on learning to test mobile device apps, and model it after the process we used in our labs - develop apps to learn how to test them. Basically, we took testers through the process of setting up a development environment, compiling a simple app and then installing and testing that app.

We'll start with Palm webOS and then cover the iPhone and Android platforms.  The place to start for any of these platforms is the software development toolkit, or SDK.  And in the case of webOS, as the book points out, the SDK is available for free.  It's called the Mojo SDK and it's available at http://developer.palm.com or can be directly downloaded here.

iPhone Development Basics - Memory Management Part 3

In our previous articles on iPhone memory management:

iPhone Development Basics - Memory Management Part 1
iPhone Development Basics - Memory Management Part 2

we've seen how developers allocate and de-allocate memory. And we've created some questions, for developers, on how crashes may occur when memory is not handled correctly. In today's video, the author, Mark Johnson introduces a variation on these things called "Autorelease pools". In the video, he shows how an object that is part of an autorelease pool can be retained despite releasing the entire pool. Seems to me that this could cause problems if not handled correctly.

Suggested questions for Part 3:

I've heard about a way to retain individual UI objects when releasing memory using autorelease pools. Do we do that and can you explain why you would retain something when it's part of an autorelease pool?

Top 10 iPhone Groups On LinkedIn

Last month we posted an article on the Top 10 Software Testing and QA Groups On LinkedIn. Since I also belong to an iPhone group (iPhone Developers - www.iPhoneintouch.com) on LinkedIn, I thought I'd look to see if there were enough iPhone groups for a similar Top 10 list. As it turns out, there are plenty of iPhone and mobile device groups on LinkedIn and definitely enough to round out a Top 10 list.

Based on membership, the Top 10 11 iPhone groups on LinkedIn are:

11. iPhone Developers Connection (424 members): This group intends to connect iPhone developers, designers, and entrepreneurs with each other. It intends to serve as platform for members to be able to share exclusive info, get peer reviews, get lead user feedback on apps, discuss projects etc.
10. iphone developers worldwide (463 members): Worldwide group for professionals in iphone application development. Contact others, share ideas, business and job opportunities.
9. Cocoa Touch - iPhone Technology Users Group (568 members): Cocoa Touch Developers Network for software developers, managers and marketing professionals working with the Cocoa Touch platform for iPhone and iPod touch.
8. Mad4iPhone - The official iPhone users group (614 members): Connect with the Mad4iPhone community today! Discover 3rd party solutions. This is a constantly growing, online, independent iPhone user group...a network of users and professionals. Solve iPhone problems. Get solutions. Give advice. Connect with other users and experts.
7. iPhone Dev Team (619 members): For all iPhone developers.

iPhone Development Basics - Memory Management Part 2

We introduced this series, created by Mark Johnson, last Monday (iPhone Development Basics - Memory Management Part 1) to help testers understand memory management from the developers point-of-view in order to develop some insight as to why applications crash because of memory problems. As mentioned last week, these videos should be used to develop questions for the developer team that may lead to more effective and reproducible testing.

Suggested questions for Part 2:

(0:52min)Does our iPhone application use something called "Autorelease pools"? I've heard that this method can help avoid bugs because you don't have to keep track of so many memory management calls.

Can autorelease pools cause crashes by releasing UI objects too early?

How Hot Is The iPhone Application Development Market?

We've just returned from the STARWEST 2009 software testing conference. While there we talked to hundreds of testing and QA people. And as we shared with them our outsource testing services and our new product, myQMT, we asked them what their iPhone development plans were. Since we both develop and test iPhone applications, we were interested in how many were developing for the iPhone market. What we found out was that most companies were planning an iPhone app. And this information was coming from their QA folks.

Today I have some links to articles that provide some insight on how hot this market is becoming:

39% average growth was experienced by most developers from adding mobile application development to their business models. NATIONAL SURVEY RESULTS: Developers Shift Business Model To Ride iPhone App Wave Developers and entrepreneurs change course and thrive as iPhone app development increases

University of Utah researchers created new iPhone programs. iPhone the body electric

The exploding iPhone Application development community. iPhoneAppQuotes Forges Strategic Partnership with Appular

American heavy metal band Metallica has launched a new iPhone application to allow fans to listen to its live show recordings on the move. Metallica launches iPhone application for fans

The App Store taps the creative energy of entrepreneurial developers. From cars to TVs, apps are spreading to the real world

Mercedes-Benz joins other consumer-oriented companies, including Pizza Hut, Kraft, and Whole Foods, in developing apps for customers. Mercedes-Benz Launches iPhone App

Technology Advisory: Apple Releases iPhone OS 3.1.2

Apple released iPhone OS 3.1.2 today. According to Apple, this update contains bug fixes and improvements, including the following:

- Resolves sporadic issue that may cause iPhone to not wake from sleep
- Resolves intermittent issue that may interrupt cellular network services until restart
- Fixes bug that could cause occasional crash during video streaming

iPhone users can download the update by connecting to iTunes and pressing Check for Update.

According to www.macrumors.com, a corresponding update is also available for the iPod touch, and the iPhone OS 3.1.2 update for iPhone in U.S. is accompanied by an update to AT&T's carrier settings file, which brings the settings to version 5.6.

iPhone Development Basics - Memory Management Part 1

For those of us involved in iPhone applications testing, it's useful to understand how memory management works on the iPhone OS. Most of the problems we see that result in some sort of application crash or failure are related to memory problems. Understanding how these memory problems can occur can help with trapping memory related defects so you can report reproducible defects.

We've discussed memory management issues in previous articles. In this series of articles, which will be posted each Monday, we have video presentations from Mark Johnson that introduce the basics of iPhone memory management in Objective-C for programmers starting out with the iPhone. These videos are very instructive for both developers and testers.

Tips for Testers: if you've never done any programming, the terms discussed by Mark may be new for you. I suggest that you use this information to develop a curiosity about iPhone app memory management so that when you talk with the developers, you can ask probing questions. And, as developers answer your questions, it often helps them sort through the complexity they deal with, as well as giving you more insight as to what may be cause the bugs you're seeing.

Suggested developer question for Part 1:

(2:50min) What is a NSMutableString object and why would the app ever release this object after it has been deleted? Would this cause a crash?

Hello MonoTouch - .NET and C# Developement For iPhone In Action

Last week we wrote about MonoTouch for iPhone (Five Million .NET Developers Can Now Cross Over To The iPhone). This week we found a video of a quick screencast showing how to make a simple iPhone Hello World type app in MonoDevelop with MonoTouch and C#. This was done using the MonoTouch preview.



There's also a 3 part series that goes more in-depth on installing and using MonoTouch. Here are the links for that series:

Getting Started with MonoTouch - Part 1
Getting Started with MonoTouch - Part 2
Getting Started with MonoTouch - Part 3

Five Million .NET Developers Can Now Cross Over To The iPhone

Novell released MonoTouch on September 14 2009, that allows developers to create C# and .NET based applications that run on the iPhone while taking full  advantage of the iPhone APIs as well as reusing both code and libraries that have been built for .NET as well as existing skills.  That means the door is opened to a lot more developers. "...the big win with it is that it opens the door for some 5 million .Net developers to begin to do iPhone applications" according to analyst Al Hilwa, program director for application software at IDC, as reported in the ComputerWorld article Novell offers iPhone .Net app development kit.

Mobile Device/Smartphone Testing News: Apple, Palm, Android, and BlackBerry

This week has been eventful for all the mobile device platforms with news from each of the major players.

Apple is now at over 75,000 apps in the App Store and shows no sign of slowing down. This week brings plenty of mobile device compatibility testing news with the announcement and shipping of iPhone OS 3.1 and iTunes 9. Apple announces iPhone 3.1



Palm's second webOS-based smartphone was announced this week. A new form factor, both with the keyboard and screen, should present some interesting testing challenges.
Palm debuts Pixi smartphone, its second using webOS





New and very popular apps appear on Android. Is this an indication of app development and testing growth for this platform? Facebook, Pandora Apps Hit Android (Just added: iPhone users aren't the only ones to get cool apps)


RIM's Blackberry is still in the fight for app development market share with news this week. Yahoo announced Finance for Mobile for the BlackBerry Bold, Tour and 8900 series, as well as Yahoo Fantasy Football for Mobile. It's interesting to see specific models listed instead of just "BlackBerry". That means the development and testing compatibility issues are only compounded when you offer diverse form factors. Yahoo intros new iPhone and BlackBerry apps

Free iPhone Simulator - No Assembly Required

The image on the left is a view of yesterday's article on USB 3.0 and was produced by TestiPhone.com, a clever web-based tool for testing content as it would appear on the iPhone. It's intended use was to test iPhone web apps prior to the advent of native apps.  Now that the SDK is out, which provides a full blown simulator (with some limitation), this free online tool would seem to be obsolete.  But I think it and others like it (iPhoney: iPhone web simulator) can still be of service for browser compatibility testing. With iPhones accounting for most of the US mobile browser web traffic, website developers need to pay attention to how their pages look on mobile Safari. Hence testers need to add mobile Safari to the browser compatibility test matrix. And by having a tool that simulates the iPhone running on the same system with desktop browsers, you can view web page rendering side-by-side. And best of all it doesn't require installation of the iPhone SDK.

Certainly you'd want to do at least one pass on an actual iPhone, but during most of the development cycle, having a quick and easy way to check how your website looks on mobile Safari, makes these simulators worth adding to your tester toolbox.

Why Are iPhone Application Developers Doing This?

iPhone application developers have been adding functionality that allow users to "free memory" on demand.  Unfortunately, Apple has begun telling these developers to remove this functionality from their iPhone applications (Apple Forces Removal of 'Free Memory' Functionality From iPhone Applications).  This has forced companies like Recession Apps, to remove their app from sale.

I talked to an iPhone app developer about this and asked him why this sort of functionality is needed.  He told me that memory management for the iPhone, or lack thereof, is woefully inadequate in some situations.  He did say that Apple provides plenty of information as to what can happen under low memory conditions and what to do about it.  And there are numerous examples provided, as well as advice (from real Apple engineers) given in the developer forums.  The problem is that the guidance given does not work in all situations, especially in very critical memory situations. 

Top 10 Tips For Testing iPhone Applications

This article started out as a Top 5 list but quickly grew as we researched all the tips and techniques we use at RTL for iPhone application testing. Most of these tips came from tracking down memory-related problems, which often resulted in defects that were very difficult to capture.

1. Accurately report available memory.  Many of the non-reproducible bugs you run into when testing iPhone apps are related to memory problems.  It's critical that you know and report available free memory before launching an application.  In all likelihood, the reproducibility of a crashing iPhone app bug is related to low memory conditions.  Consequently, a crashing defect may disappear when there's plenty of free memory.  In a previous article (Using Memory Sweep for iPhone App Testing) we described a tool that can be used to determine free memory.

2. Provide 'crash reporter' logs with your defect reports.  Each time an iPhone application crashes, a .crash file is created on your iPhone.  You can retrieve this file when you synch your iPhone with iTunes.  Here's a link that describes where those files are stored: iPhone Crash Logs

3. Spy on the app from the console.  iPhone apps will report application and system level warnings to the console.  You can view these warnings in real-time using Apple's iPhone Configuration Tool.  By knowing what's being reported when interacting with an app can help you refine the steps you need to reproduce tricky (and memory related) problems.

Mobile Device Application Testing Coverage Expands

The mobile device app space is expanding quickly. Testing for compatibility across different devices within a platform family and across platforms is becoming increasingly complex. Where will development and testing organizations draw the line?

Microsoft luring iPhone developers for Zune apps Developers being offered money to port their apps? If they are successful in luring iPhone developers, you can bet that these apps will need to be tested on both platforms. Are we entering another phase of supporting apps on multiple platforms?

Second webOS-Based Smartphone May Be Delayed Probably news to most that Palm's second webOS device is called the Palm Eos. Was to be available this fall and supposedly smaller than the Palm Pre. Will there be a significant update to webOS when the new devices arrive?

Microsoft Renaming Windows Mobile as Windows Phone Windows Mobile, or WinMo, is expected to change names (WinPho?), probably to reflect Microsoft's new emphasis on its Zune and the mobile device market, particularly the iPhone market.

Dell Confirms Mobile Device Development Based on the rumors, this will expand the Android app market. Looks a lot like the iPhone in this picture.

Samsung's TouchWiz App Development Kit Hobbles Into the Light Samsung's TouchWiz app dev kit is out and it looks like it immediately expands compatibility testing across 3 different smartphone operating systems, Windows Mobile, Symbian and Samsung's own.

iPhone App Testing With iSimulate

The iSimulate iPhone app is interesting in so many ways. If you've read the articles posted about it (here's one from my favorite Apple blog, TUAW: iSimulate brings iPhone apps to the big screen), you probably were impressed by the idea of using your iPhone to control an iPhone app while watching it on a separate display. Playing a game on a large display while controlling it through an iPhone, as if the iPhone is some sort of game controller accessory, is intriguing.

And for the many iPhone app developers that are looking for a way to create marketing material of their app in action, iSimulate is probably just what they've been looking for. Recording your iPhone app in real time on a higher resolution display with actual motion inputs and then being able to later edit the recording has got to be better than the various video cam recordings one finds on YouTube.

Using Memory Sweep for iPhone App Testing

As mentioned in a previous article here (Testing For iPhone App Memory Problems), chasing down memory problems when doing iPhone app testing can be difficult. In particular, providing developers with concrete evidence that a memory issue is or is not their problem is a big challenge. Recently, while testing an iPhone app that was consistently presenting low memory alerts, we needed an independent memory assessment and discovered Memory Sweep, written by Gary Fung. It's a free app and works great for our testing purposes. Here's the information it provides you:


The numbers you see for Memory Status will help you diagnose memory problems when testing iPhone app. We used it for memory stress testing. We needed to force a low memory condition to see how an iPhone app behaved. Memory Sweep validated this process. We have a way of setting up iPhones/iPod touches so that memory is forced to be about 3-4MB. Once we do this, we launch Memory Sweep to make sure free memory is as low as we expected (you can see that free memory in the image above is 4.21MB), then quit and launch the iPhone app under test. In this way we can characterize our defects in a precise way so that the developer has confidence in the results.

When you read the reviews on Memory Sweep, you'll find that are very critical about the app. The main complaint is that the app is named Memory Sweep on the iTunes App Store and ends up being called "Scan" on your iPhone. From what I can tell, the critical reviews have been mostly about cosmetics and not functionality. There are plenty of positive reviews, particularly from those that appreciate the capability to force more memory on their iPhones. My main complaint is that this app only runs under iPhone OS 3.0. This will force us to find a similar app to run on older iPhone OS versions for our testing.

A Lesson In Defect Characterization - How A Twitter Security Enhancement Created a Twitgoo Problem

Sometimes defects aren't what they appear to be. I'm sure developers have countless stories of how defects they set out to fix were not at all like the defect report described. We ran into just such a situation recently while testing a twitter-enabled iPhone App - Photo Tweet. Photo Tweet would mysteriously, and seemingly arbitrarily, fail to upload photos to Twitgoo. It was one of those situations where one day a feature works and the next day it fails. And, of course, the developer had not made any changes.

Most of the time when you find a bug and the developer claims not to have made any changes, it's always best to quiz the developer with "what-if" questions. That allows you, as a tester, to avoid calling the developer's integrity into question. I usually try questions like this:

• What if during last night's build, an older version of a component was used?
• Even though you didn't make a change for that feature, what if some other change caused this bug?

Usually this sort of brainstorming session can tease out needed clues that lead up to finding the root cause of a mysterious bug. If that doesn't work, then you're left trying to find out what else could have contributed to the defect. Such was the case with Photo Tweet and it's inability to upload photos after a period of time. And so, a bug was written up that said Photo Tweet would stop uploading photos arbitrarily.

Subsequent conversations with the developer, including some what-if questions, lead us to believe that no changes were made for this feature and the bug resided outside the developers code. Well since Photo Tweet uploads its photos to Photo Bucket's Twitgoo service, we suspected they had instituted some sort of throttling for uploads. This is reasonable to expect given the popularity of Twitgoo. Maybe they just didn't want rogue apps flooding their service, and apps like Photo Tweet were paying a penalty for the nefarious activities of others. We could not have been more wrong about the root cause of our defect.

We reported our defect to Twitgoo and asked if they had instituted an upload limitation. I was amazed at how fast these folks responded and how helpful they were. Initially they were stumped since this was the first they heard of this problem and HAD NOT put a limit on uploads. At that point, I expected that they could not help any further and we would be left with having to do more debugging. But instead, Michael P. Clark, their SVP of Technology and Partnerships rallied his technical folks to find out what was at the heart of this problem. The next day I got an email from Justin Hart, a Sr. Engineer for Photobucket, informing me that they discovered an unannounced change to Twitter's service. Twitgoo relies on Twitter for those users wanting to "tweet" the photos they upload to Twitgoo. It turns out that Twitter had made a security enhancement. Michael and Justin found this issue very quickly and pointed me to a threaded discussion concerning what Twitter did. Much to Twitter's credit, they quickly made this issue public and extended an apology and an offer for input from developers. Here's a snippet of their statement:

A change shipped last week that limited the number of times a user could access the account/verify_credentials method [1] in a given hour. This change proved hasty and short-sighted as pointed out by the subsequent discussion [2]. We apologize to any developer that was adversely affected. Given the problems, we want to fix this in a public and transparent manner.

We learned a lot from this experience. First, the folks at Twitgoo are fantastic and serious about providing a great customer and technology partner experience. Second, some defects are certainly not what they appear to be. Third, when testing products that use APIs that connect to services that connect to other services, it's hard to determine root causes no matter how well intentioned your defect characterization process is. Finally, as you investigate why a defect occurs, don't assume you know the root cause until you have all the facts.

Agile Testing Framework For iPhone App Development

As we continue developing and adding to our iPhone App Testing Lab, we are on the lookout for helpful testing tools. While doing some research on iPhone App testing tools, I ran across a post by a developer asking for help in this area:

My iPhone app has grown to the point where I need some sort of automated testing to keep me from breaking things whenever I make a change. I thought Instruments would do the trick, but I can't seem to make it work. What do you use for automated regression testing?

This is an appropriate question to ask, but one I would suspect results in an answer with a limited set of choices. In fact, the only tool we've come across, and posted an article about, is yet to be delivered (Squish Support For Automated GUI Testing on Apple's iPhone and iPod Touch).

Fortunately, as we all share information in blogs and forums, others points out additional solutions. In particular, one of our readers pointed me to uispec, Behavior Driven Development for the iPhone. I especially liked his comment "It's open source too.". That's always an attention getter for those of us looking for useful tools.

The author of this tool describes it as a "Behavior Driven Development framework for the iPhone that provides a full automated testing solution that drives the actual iPhone UI. It is modeled after the very popular RSpec for Ruby." For those of you interested in learning more this type of Agile development framework, take a look at Behavior Driven Development on Wikipedia.

When you visit the uispec link, you'll find documentation, installation instructions and examples. It looks like a great way to build in BDD testing for those of you that do continuous iPhone app development.

Testing For iPhone App Memory Problems

Problems with memory management has got to be the #1 issue for iPhone app development, and presents one of the biggest challenges to testers. Our experience with testing over a half dozen iPhone apps in the past few months has reinforced the need for developers to nail down their memory management techniques early and for testers to create test cases that stress the way an iPhone app handles memory. So far the process has been bumpy but we've arrived at some basic memory management testing concepts along with recommendations for how developers can improve their memory management techniques.

Basically there are 2 types of memory problems to watch out for:

1. Primary Memory Leaks (in the "sandbox")
2. Secondary Memory Build-Up (out of the "sandbox")

Primary memory leaks are the most common and most troublesome to developers. These leaks create the mysterious crashes for iPhone apps in early development and are the hardest to document in a bug report, mainly because primary memory can change out from under the developer depending on other apps such as Safari and Mail. Safari can use up a lot of memory especially if multiple pages are open. Primary memory leak bugs seem to go away by simply restarting the application.

Secondary memory build-up creates situations that, over time, seem to create a permanently smaller amount of memory available for the app to run in, i.e. smaller primary memory. This does not come about because of memory leaks per se, and hence is harder to find with memory checking tools. As opposed to primary memory leaks, crashes or memory error alerts will seem reproducible for this type of memory problem. Consequently, the tester will write up repeatable steps leading up to the crash or memory alert, only to find out that, after restarting their iPhone, these steps will no longer be reproducible.

Our approach to rooting out these types of bugs is to first be able to identify them as categorized above, and then employ exploratory techniques to attempt to find reproducible steps. However, these memory issues can be elusive, and, as stated above, often lead to misleading bug reports. The main thing is to be able to communicate to development what type of memory issue you're seeing so they can use debugging tools to find the problem.

When we find these types of memory problems, our bug reports provide a summary, steps to reproduce, other apps that are running (i.e. Mail or Safari) and a result, e.g. app crashes or memory alert is presented. If the bug is not 100% reproducible, then we comment on where it mostly occurs. Our main effort is to write our bugs up so that a developer is convinced that it's a memory problem of some sort. Once convinced, then they can instrument their code to find it. But if you don't provide a bug report that describes the defect in the context of a memory problem, then the bug will most likely end up as "cannot reproduce" and be ignored.

We then recommend that the developers instrument their code to find the memory issues. But since we are testers, we can only go so far in what we recommend. It's been my experience that iPhone developers, when faced with memory issues, appreciate all the help they can get. Here are the links we send them to:

• Apple iPhone Developer Forum
  
• Finding Memory Leaks With The LLVM/Clang Static Analyzer
• CLANG Static Analyzer
• Finding iPhone Memory Leaks: A “Leaks” Tool Tutorial
• NSZombie and XCode Oh My!

Most developers visit the Apple iPhone Developers Forum and it's a great place for testers too. By reading the posts there, you get a feel for just how tough memory management is. But more importantly, you gain confidence that bugs you are seeing really can be fixed and should not be attributed to peculiarities of the iPhone. Apple developers write a lot of responses in those forums to help counter the claims that the OS is to blame for all these memory problems.

By understanding memory problems, testers can be authoritative in their defect reports. By knowing where to point developers for help, the test group becomes a resource for helpful investigation and research for tough memory problems.

Gearing Up To Develop and Test Palm webOS Apps

Palm and O'Reilly are offering developers and testers a look at the Palm webOS and what it takes to develop for the platform. You can get a sneak peek overview of webOS here, where you can learn how the OS works and the terminology for the UI. Basically, webOS is an embedded Linux operation system with a custom UI built on a browser. I think the biggest deal about this device and its OS is the underlying development environment embodied in their Mojo SDK. Mojo offers a JavaScript framework that provides standardized UI widgets, and access to selected device hardware and services. I think this will attract a lot more developers to the platform than other devices because of JavaScript, versus Objective-C or Java. However, it's not the language choice that attracts developers - it's the richness of the device and its APIs. From the looks of things, Palm's webOS brings a lot to the party. Moreover, developers are free to develop their apps from scratch if they don't want to use Palm's SDK.

If you are looking to catch up on webOS and the Mojo SDK, then I recommend watching Mitch Allen's (Palm's Software CTO) presentation on webOS development. In the video below, he gives a preview into application development with the Mojo SDK, the development environment and toolset for this new mobile web platform.

DeployStudio HOWTO - Installing on Mac OS 10.5

Our first article on DeployStudio came about because we needed a replacement for NetRestore at RTL (Using DeployStudio For Imaging and Restoring OS X Setups for Testing). Subsequently, we wrote a second article with a very useful tutorial video (DeployStudio 101 Tutorial). Now having used this tool at RTL for awhile now, we've honed our setup process and can share that with you in this article.

The steps below will guide you through the setup process for using DeployStudio with Mac OS 10.5. What you will end up with is a DeployStudio drive that can boot on both PPC and Intel Macs.

1. Partition an External Firewire Drive to Apple Partition Map.
2. Install Mac OS 10.5 (does not have to be Server, though it could be) on the external drive from a PPC machine*.
3. Boot into the OS X installed.
4. Download the latest build of Deploy Studio.

iPhone App Testing With The App Store Approval Process In Mind

Apple's App Store approval process can be a frustrating experience if you do not approach your iPhone/iPod touch app testing with the right end-goal in mind. There are countless stories of developers submitting their apps several times before Apple approves them for the App Store. Most of the publicized stories about failed App Store submissions usually are about content or illegal functionality. The stories you don't hear about are those that have to do with apps not meeting Apple's basic human interface guidelines. In talking with iPhone developers, we've found that Apple provides detailed feedback on why an app is not approved along with references to their guidelines. This is very helpful, but problems can be avoided and apps can be approved faster if developers utilize a suite of test cases and environments to ensure their app not only functions as intended but also meets applicable Human Interface guidelines. And now with iPhone OS 3.0, those guidelines have increased with the new functionality provided by 3.0.

What we've done to help our clients avoid known pitfalls in the App Store approval process, is develop a Core Tasks test suite that is used along with the functionality and compatibility tests we perform. This Core Tasks test suite has been developed using Apple's Human Interface guidelines. Here are a few examples from the test suite that we use to help developers determine if their app functions correctly and appropriately when:

1. Using Cut, Copy & Paste
2. Receiving a Push Notification (unrelated to their app)
3. Unexpectedly terminating when receiving a call
4. Internet connections are lost
5. Launching while a user is listening to music from the iPod app
6. Requiring location services

There's a lot to focus on when developing an iPhone app and the main concentration is usually on the unique functionality of the application. But leaving out some of these core task tests can lead to delays in the App Store submission process.

Warning: Testing on iPhone OS 3.0 May Not Be Enough

Developers who update their iPhone OS 2.x apps have been warned that they need to make sure their apps work on iPhone OS 3.0. This has not been an unreasonable requirement and has seemed pretty straightforward to most developers: test your app on a device with the latest 3.0 beta, fix any problems and upload your app. Well, as we found out today, this process can be problematic.

We got caught in the middle of this policy change when we uploaded an update last month and it ran into a problem with 3.0. We immediately corrected that problem and tested our app on 3.0 on our own devices and at the Apple lab in Cupertino. We then awaited Apple's final review and approval.

Much to our surprise we received an email from Apple claiming our app had another 3.0 problem. Upon reading the report, we discovered that the Apple tester had not reported a crash or other showstopper, but instead had made a determination that a specific feature did not work the way he/she had thought it should work. And since they attributed this to iPhone OS 3.0, they rejected our update once again. The feature that was misinterpreted had not changed since the last time Apple approved our app. In fact, we regressed their report under 2.x and 3.0 and the feature worked identically in both configurations.

So beware, if your app update ends up in the hands of an over zealous Apple tester who, in my opinion, oversteps his/her responsibilities, you may find yourself with more delays in getting your app approved.

Planning to Develop/Test Mobile Device Applications?

If your company is currently providing development and/or testing services for mobile device applications, as we are at RTL, then you will face some business decisions on how best to configure your test lab(s). We began the process to procure and configure our mobile device test lab over a year ago based on current market trends. This was mostly influenced by the success Apple was having with their iPhone and iPod touch App Store.

As it turned out, the establishment of an iPhone/iPod touch test lab at RTL paid off, as we've tested for several iPhone OS developers and expect this to continue as this market matures and developers expand their operations. And, given the success many have had in the iPhone app market, I would expect these established developers, as well as those now entering this market, to look for opportunities on other mobile device platforms.


Naturally, those of use that provide testing services, need to grow our mobile device test labs to best serve those developers that take advantage of the development opportunities offered by each mobile device vendor. One way to analyze the potential opportunity is to "follow the money." And by "money", I mean application distribution. Of the 4 dominant mobile device vendors (Apple, RIM/BlackBerry, Google/Android and Microsoft), there exists 3 application distributions channels, with one (Microsoft's) on it's way:

• Apple's App Store
• Google's Android Market
• BlackBerry App World
• Microsoft's Windows Marketplace for Mobile

But what's most interesting and instructive about these distribution channels is how successful (or not) they've been to attract applications and hence, developer mind share. Based on a article from the mocoNews.net titled Apple Vs. BlackBerry Vs. Google: Testing The App Stores, the number of apps sold through these channels are telling:

• Apple's App Store: 48,000+
• Google's Android Market: Low thousands
• BlackBerry App World: About 700

And the stores vary considerably when it comes to app quality, pricing, and billing, according to Tricia Duryee at mocoNews.net. She says there's "... no consensus on price. For example, Glu (NSDQ: GLUU) Mobile's Build-A-Lot game costs $9.99 on BlackBerry, but $4.99 on Android and $1.99 on Apple (NSDQ: AAPL)". Apple uses their convenient iTunes store for app billing, while BlackBerry requires a PayPal account.

Based on this assessment from a application distribution perspective, I would expect additions to our mobile device test lab to include, at least, a Google Android device in the near future, and probably a Windows for Mobile device just because of Microsoft's ability to take market share away from the current leaders. But what about BlackBerry, and, for that matter, mobile device players like Nokia (Ovi Store) and Palm Pre (App Catalog)? Nokia is well established, but Palm Pre may pull off an upset given their support for WebKit.

It will be both exciting and difficult to make the best business decisions about how to expand your mobile device testing labs. We expect this market to grow and along with it the demand for collaborative testing services on a broad selection of the most popular mobile device platforms.

iPhone 3.0 OS Testing Tips

All App Store testing and review will occur on iPhone OS 3.0 to prepare for a smooth transition this summer, and incompatible applications won't be approved. Here's an excerpt of what Apple's has emailed to developers:

Beginning today, all submissions to the App Store will be reviewed on the latest beta of iPhone OS 3.0. If your app submission is not compatible with iPhone OS 3.0, it will not be approved.

Existing apps in the App Store should already run on iPhone OS 3.0 without modification, but you should test your existing apps with iPhone OS 3.0 to ensure there are no compatibility issues. After iPhone OS 3.0 becomes available to customers, any app that is incompatible with iPhone OS 3.0 may be removed from the App Store.

For many developers this presents a dilemma has to how to test existing iPhone OS 2.x apps to make sure their updates will continue to be accepted before iPhone OS 3.0 is generally available. Most small developers have limited devices to develop and test on, and once you install iPhone OS 3.0, you can't go back. The folks over at The Apple Core, quote this from Apple: "Devices updated to iPhone 3.0 beta can not be restored to earlier versions of iPhone OS . Devices will be able to upgrade to future beta releases and the final iPhone OS 3.0 software."

Here's what we recommend for testing purposes:

PhoneGap: Cross-Platform Mobile Device Development Solution

Nitobi's PhoneGap is an open source solution designed to give web developers JavaScript access to popular mobile device features, like the camera, GPS, the accelerometer, local SQLite databases and more, without having to write full applications.

Here's what the Nitobi developers have to say about PhoneGap at their website:

PhoneGap is an development environment for mobile application developers who want to write their code once and deploy it to multiple smartphones with as few headaches as possible. PhoneGaps supports or will support iPhone, Android, Blackberry, Palm's Pre, and Windows Mobile.

I found this informative video on PhoneGap that explains the advantages of using it and even talks about how some current iPhone apps have been build using PhoneGap:

iPhone Test Tools - iPhone Forensics

Featured in the video below is Jonathan Zdziarski, author of iPhone Open Application Development: Write Native Objective-C Applications for the iPhone, iPhone SDK Application Development: Building Applications for the AppStore and iPhone Open Application Development: Write Native Applications Using the Open Source Tool Chain. He helped lead the effort to port the first open source applications, and taught developers how to write applications for the iPhone before Apple introduced its own SDK. His prior work has been, a forensics manual, has been distributed exclusively to law enforcement. Jonathan frequently consults with law enforcement agencies and assists forensic examiners in their investigations.


In the video below, Jonathan demonstrates how to install his custom forensics toolkit on any existing model iPhone and send a raw disk image to a desktop machine. He also shows you how to recover files specific to the iPhone including deleted keyboard caches, photos, web objects, and much more. His other iPhone development videos on YouTube are Getting the iPhone Open Source Tool Chain, and iPhone Forensic 101: Bypassing the iPhone Passcode.

Squish Support For Automated GUI Testing on Apple's iPhone and iPod Touch

A company called froglogic has announced that it's automated GUI testing tool, Squish, will support the testing of Cocoa Touch based applications on iPhone and iPod Touch devices and simulators. Here's an excerpt from their press release:

Cocoa Touch provides an abstraction layer over iPhone OS, the operating system used by the iPhone and iPod Touch. Cocoa Touch is based on the Cocoa API and toolset used for building software for Mac OS X computers.

Squish's distributed network architecture enables tests to be controlled from desktop PCs, or from a server, while the application under test is executed and tested remotely, for example, on other PCs, or on embedded devices, such as the iPhone or iPod Touch.

A prototype of Squish for Cocoa Touch has been completed. A final version of Squish for Cocoa Touch will be included in the upcoming Squish 4.0 release.

Based on this press release, it looks like this tool will test apps directly on the an iPhone or iPod touch, although the accompanying picture appears to be running a test on the iPhone simulator. It will be interesting to see how they enable their GUI testing tool on actual iPhone and iPod touch devices.

Book Review: iPhone in Action

iPhone in Action, Introduction to Web and SDK Development, is 50% off the regular price until April 30, 2009. To get the discount, just go to the publisher's site and when you order use this code: tuaw50. This is only for the current edition of iPhone in Action. If you miss the sale, Amazon has it for sale at 34% off.

This book is a great starter book and we used it to help us with the development of our first iPhone application. The best part of buying this book was getting the downloadable PDF version for free. That helped a lot, especially when we needed to copy examples from the book.

The book does provide ample coverage iPhone development. This includes web and native apps. The tutorials are very good and there are code examples for most major iPhone areas. Although our app was not a web app, which a full third of the book covers, we found it interesting to read this part of the book. Later on it helped us with adding web access to our app. The native app part of the book introduces the SDK, starting with an overview of Objective-C and XCode, and then follows up with excellent step by step tutorials. These tutorial teach you how to use Interface Builder and the different kinds of view controllers to create your GUI. The part of the book covers graphics, web interaction, SQLite databases, using the address book, etc. iPhone in Action is great for experienced programmers and iPhone development beginners, mainly because it covers both web and then SDK.

If you want to sample the book before you buy, the publisher has 3 chapters you can download, as well as all the source code from the book. Just visit the Manning Publications website to check it out. The free ebook that accompanies the book is provided by the publisher at their website as well. You have to buy the book to get the code they require when you click on the ebook link.

As always, we will provide you with some comments from other reviewers that do not share our opinion on the book:

A very basic introductory book to iPhone programming, says "IT Developer," my major complain is that it either left out the important stuff or only roughly addressed them. For example, NSException handling is not even mentioned in the book.

Barry A. Starrfield writes: All about HTML Programming. "Be aware that this book is strongly focused towards iPhone Web applications development. Sadly, your Web based iPhone app is not what consumers want - they want an SDK application, and those are the applications that you'll get paid for."

Book Review: The iPhone Developer's Cookbook

Overwhelming, reviewers have both praised Erica Sadun's book The iPhone Developer's Cookbook: Building Applications with the iPhone SDK and warned that it's not for beginners. We found this book very helpful during the development of our first iPhone app (Phone Alarm) and were pleased that it came in both paperback and PDF. I particularly like books that come with a companion PDF version.

Erica Sadun first explores the iPhone delivery platform and SDK, helping you set up your development environment, and then shows you how iPhone applications are constructed. Next, she offers single-task recipes for the full spectrum of iPhone/iPod touch programming jobs:

• Utilize views and tables
• Organize interface elements
• Alert and respond to users
• Access the Address Book (people), Core Location (places), and Sensors (things)
• Connect to the Internet and Web services
• Display media content
• Create secure Keychain entries

I thought it was a great book whether you had experience or not, but I found several other reviews that disagree:

Jason R. Weiss (Katy, TX USA) writes: New to Apple's Developer Tools? This book is not for you!, The book states it is aimed "squarely at anyone just getting started with iPhone programming." It is not...

Perfect for seasoned programmers, writes J. Badger. If you're an accomplished Objective C programmer then this book really helps you understand the fine points of iPhone programming...

Bernhard Müller cleary warns that This should be your SECOND iPhone programming book!She shows a lot of tricks you won't see in the standard books. But to really enjoy this book, you should know iPhone programming at least basically...

Push Services Too Costly For Some iPhone Developers?

The long awaited for push notification service for the iPhone is available in iPhone OS 3.0 and developers are being encouraged to start testing push with Apple's server. Push services have the potential of bringing a whole new level of innovation to iPhone app development. However, the potential cost involved in developing and maintaining a push server for your app may be prohibitive. Yes, Apples does maintain and provide access to the Push Notification Service, but you, the developer, must provide the "3rd Party Server" as shown in the picture above. Apple only provides the piece that allows a well formed message to be sent to a designated (and registered) device. You have to form that message and designate the receiver. And for that you'll need your own server. Erica Sadun put it this way in a recent article at Ars Technica:

Consider an application with just 10,000 users. It might service a million uses per day, assuming update checks every 15 minutes. More time-critical uses might demand checks every few minutes or even several times a minute. As the computational burden builds, so do the hosting costs. While cloud computing provides an excellent match to these kinds of needs, that kind of solution comes with a real price in development, maintenance, and day-to-day operations.

She makes it sound fairly daunting and this may scare off some developers. But I think developers will find a way to make this work either by using cloud computing or sharing resources. There's one company, macminicolo.net, that's already offered to host push services for developers. They advertise their service starting at $35 per month. And if you are staying current with the iPhone Developer Forums, you'll see that code for these back end push servers is already being shared and is running on Mac and Linux servers. I think it would be worthwhile to take some of this work and put it on the cloud using Amazon's EC2. The initial costs of running a low-end Amazon Machine Instance is around $72 per month (doesn't include data storage or network usage). Between that and the results of the experiment macminicolo.net is conducting, we may find that the costs can be contained.

iPhone Application Programming Course

Stanford University has begun publishing video podcasts and slides from its popular "iPhone Application Programming" course on iTunes U for free to the general public, beginning this week.

Course description:
Tools and APIs required to build applications for the iPhone platform using the iPhone SDK. User interface designs for mobile devices and unique user interactions using multitouch technologies. Object-oriented design using model-view-controller pattern, memory management, Objective-c programming language. iPhone APIs and tools including Xcode, Interface Builder and Instruments on Mac OS X. Other topics include: core animation, bonjour networking, mobile device power management and performance considerations.

Offered by Stanford's School of Engineering, the course will last ten weeks and include both the lecture videos and PDF documents. A new lecture will be posted each Wednesday and Friday.

I checked iTunes last night and so far 2 videos have been posted, accompanied by lecture slides (PDF):

1. Introduction to Mac OS X and Cocoa Touch (April 1, 2009) 1:04:45
   
2. Using Objective-C, Foundation Framework (April 6, 2009) 1:09:00
   

Here's a direct link to Stanford on iTunes U's iPhone Application Programming.

Unit Testing Tips and Techniques for iPhone

"...Unit tests help ensure low-level code correctness, reduce software development cycle time, improve developer productivity, and produce more robust software."

-Unit Test Frameworks

As important as unit tests are to modern development efforts, the iPhone SDK makes it difficult, if not impossible, to introduce unit tests. However, there are some useful resources online that can help developers shoe-horn a unit test framework into the SDK. One of these is based on the "Google Toolbox for Mac" project. They've extended their toolbox to provide iPhone Unit testing. You can find their well-written HOWTO at the iPhoneUnitTesting page.

The other resource I suggest you look at is from the folks that brought the OCUnit framework to Apple's Xcode. At the iPhone Unit Testing page of their website, you'll find the history of OCUnit and precise instruction on how to get it to work with the iPhone despite Apple having put OCUnit into iPhone SDK 2.2.!

iPhone Application Backward Compatibility With iPhone OS 3.0

According to Ralf Rottmann at 24100.net, you can keep developing your iPhone OS 2.x apps using the new SDK and run them either on an iPhone with 2.x firmware or one with 3.0 beta. Here's what Ralf did:

"I’ve upgraded one of our 2G models from 2.0 to 3.0 beta. Rebuilding our projects with the updated SDK and Xcode with a “Device 2.2 - Release” configuration worked like a charm, including the debugger functionality."

Other tips and comments from developers posting to 24100.net:

• Many apps downloaded from the App Store crash on 3.0 beta; probably need to be re-build
• Apple has specifically requested that developers not use the Xcode that gets installed with 3.0 for compiling apps to go on the App Store.
• If needed, You can install multiple versions of XCode by installing each SDK with a new location for the Developer Essentials. For more details, see the comments posted here.

DeployStudio 101 Tutorial

DeployStudio is very popular in our various test labs at RTL, especially since NetRestore is no longer being developed. In this introductory video, Andrew, from ADaMac.org, takes you through the first steps in setting up DeployStudio. The basics are covered and by the end, you should be netbooting, creating and restoring images.

Testing iPhone Applications - The Matrix

In our previous post, Testing iPhone Applications, we described our approach to testing currently shipping iPhone applications. That testing required that we provide compatibility coverage across all shipping devices. This included testing on each variation of iPod touch and iPhone.

If you are planning on testing an iPhone application, here's the matrix you will need to address during your testing:



This matrix does not address the OS version on each device. You will need to add that depending on how backward compatible you want your app to be, e.g. runs on iPhone OS 2.0 and above.

If you don't have all the equipment on this list or just want to have an independent party test your iPhone/iPod touch app, contact Recommended Test Labs. We can test your device in our compatibility lab.

Testing iPhone Applications

Developing applications for Apple's iPhone and iPod touch is an exciting and interesting experience. Last year RTL established an iPhone Apps Test Lab and set about to develop its own app (Phone Alarm - available at the iTunes App Store) to understand what developers go through in order to better serve their testing needs. In the middle of creating our iPhone Apps Test Lab, we were fortunate enough to work with the developers at NewToy, the creators of Chess With Friends*. These guys were great to work with and their app was fun to test.

Here's how we planned and executed the testing.

The first thing we did was outline a compatibility test matrix. You would think that the compatibility list would simply be an iPhone and iPod, but it ended up growing much larger - not unlike most compatibility matrices. We had to take into account both the first generation iPhone and the iPhone 3G, as well as the iPod touch (and now the iPod touch 2G). On top of that, we needed to apply the connectivity variables: Edge, 3G, 802.11, as well as no connection. This made for a rather large coverage matrix.

Once we had the compatibility matrix laid out, we composed a Basic Acceptance Test procedure (BAT) that covered all the main features as well as some corner cases. Then it was just a matter of running a BAT for each compatibility configuration in the test matrix.

Along the way, we did run into challenges and issues unique to testing on the iPhone platform. The following 3 areas are worth sharing:

- Installation Logistics
- Defect Characterization
- Debugging Information

Installation Logistics
Because of the way Apple controls the installation of applications on the iPhone, you can't simply receive an app and install it locally to your iPhone. You have to provide the developer with the unique IDs for each device. This is described on the developer portal and requires that you connect your device to a computer and use iTunes to find the ID. Here are the steps for doing this:

1. Launch iTunes.
2. Connect your device to your computer.
3. Select the device in the Devices list.
4. In the Summary pane, click the Serial Number label. It changes to Identifier.
5.Choose Edit > Copy.
6. Submit your device identifier to the developer.

Once we collected all the device IDs, we sent them to the developers and they in turn had to create a certificate that was installed on each test device - and that required connecting the device to a computer and using iTunes for the installation.

Defect Characterization
For the most part, the defect characterization process for testing an iPhone App is no different than what you would use for any other testing. The challenge with iPhone Apps is in the number of environmental combinations you'll encounter. Not only are you testing across multi-generational devices, but each device can have multiple connectivity options. And when you're testing an app like Chess With Friends, which requires internet connectivity, verifying defects on each option adds to the overall defect characterization process and tester time.

Debugging Information
Providing debugger info when testing iPhone Apps can be tricky. In fact, unless you use the SDK, there are no simple options for providing standard debug logs. However, depending on the nature of the app, there are a number of things you can do. First and foremost, learning how to take screenshots is imperative. It's easy enough to do on the iPhone platform (hold down the Home button while momentarily pressing the power button). Secondly, knowing if an app is storing data elsewhere, e.g. a back-end server, allows you to use that data as debugger info so long as you can isolate it. If the app you're testing uses online account info, then you can help the developers isolate needed debug info by reporting user account info and time stamp data with each defect.

Overall, the process for testing an iPhone application is no different than for any other product. However, learning how iPhone apps are developed and reading the posts at the developers forum will better inform you on what testing issues may arise.

*Chess with Friends was the winner of the 2008 Best Free or Ad Supported Game