Warning: Testing on iPhone OS 3.0 May Not Be Enough

Developers who update their iPhone OS 2.x apps have been warned that they need to make sure their apps work on iPhone OS 3.0. This has not been an unreasonable requirement and has seemed pretty straightforward to most developers: test your app on a device with the latest 3.0 beta, fix any problems and upload your app. Well, as we found out today, this process can be problematic.

We got caught in the middle of this policy change when we uploaded an update last month and it ran into a problem with 3.0. We immediately corrected that problem and tested our app on 3.0 on our own devices and at the Apple lab in Cupertino. We then awaited Apple's final review and approval.

Much to our surprise we received an email from Apple claiming our app had another 3.0 problem. Upon reading the report, we discovered that the Apple tester had not reported a crash or other showstopper, but instead had made a determination that a specific feature did not work the way he/she had thought it should work. And since they attributed this to iPhone OS 3.0, they rejected our update once again. The feature that was misinterpreted had not changed since the last time Apple approved our app. In fact, we regressed their report under 2.x and 3.0 and the feature worked identically in both configurations.

So beware, if your app update ends up in the hands of an over zealous Apple tester who, in my opinion, oversteps his/her responsibilities, you may find yourself with more delays in getting your app approved.

Planning to Develop/Test Mobile Device Applications?

If your company is currently providing development and/or testing services for mobile device applications, as we are at RTL, then you will face some business decisions on how best to configure your test lab(s). We began the process to procure and configure our mobile device test lab over a year ago based on current market trends. This was mostly influenced by the success Apple was having with their iPhone and iPod touch App Store.

As it turned out, the establishment of an iPhone/iPod touch test lab at RTL paid off, as we've tested for several iPhone OS developers and expect this to continue as this market matures and developers expand their operations. And, given the success many have had in the iPhone app market, I would expect these established developers, as well as those now entering this market, to look for opportunities on other mobile device platforms.


Naturally, those of use that provide testing services, need to grow our mobile device test labs to best serve those developers that take advantage of the development opportunities offered by each mobile device vendor. One way to analyze the potential opportunity is to "follow the money." And by "money", I mean application distribution. Of the 4 dominant mobile device vendors (Apple, RIM/BlackBerry, Google/Android and Microsoft), there exists 3 application distributions channels, with one (Microsoft's) on it's way:

• Apple's App Store
• Google's Android Market
• BlackBerry App World
• Microsoft's Windows Marketplace for Mobile

But what's most interesting and instructive about these distribution channels is how successful (or not) they've been to attract applications and hence, developer mind share. Based on a article from the mocoNews.net titled Apple Vs. BlackBerry Vs. Google: Testing The App Stores, the number of apps sold through these channels are telling:

• Apple's App Store: 48,000+
• Google's Android Market: Low thousands
• BlackBerry App World: About 700

And the stores vary considerably when it comes to app quality, pricing, and billing, according to Tricia Duryee at mocoNews.net. She says there's "... no consensus on price. For example, Glu (NSDQ: GLUU) Mobile's Build-A-Lot game costs $9.99 on BlackBerry, but $4.99 on Android and $1.99 on Apple (NSDQ: AAPL)". Apple uses their convenient iTunes store for app billing, while BlackBerry requires a PayPal account.

Based on this assessment from a application distribution perspective, I would expect additions to our mobile device test lab to include, at least, a Google Android device in the near future, and probably a Windows for Mobile device just because of Microsoft's ability to take market share away from the current leaders. But what about BlackBerry, and, for that matter, mobile device players like Nokia (Ovi Store) and Palm Pre (App Catalog)? Nokia is well established, but Palm Pre may pull off an upset given their support for WebKit.

It will be both exciting and difficult to make the best business decisions about how to expand your mobile device testing labs. We expect this market to grow and along with it the demand for collaborative testing services on a broad selection of the most popular mobile device platforms.