Testing Android Apps - The Compatibility Testing Matrix Continues To Grow

Android, an open source operating system, must avoid the fate of J2ME, an open source mobile applications platform. Open source is great, until everyone splinters off into their own world. That’s what happened to J2ME, and a number of frustrated Android developers are now saying that there is a risk Android will follow the same path.

-Michael Arrington, A Chink In Android’s Armor, 10/11/2009

Android developers continue to voice their concerns, and in a more recent article, "Android's rapid growth has some developers worried", there's even more evidence that development and testing compatibility issues have a potential to grow out of control. Developers identify a slew of compatibility concerns:

• 3 versions of the OS (Android 1.5, 1.6 and 2.0)
• Differences across handsets
• Custom user interfaces, e.g. HTC Sense, Motorola Blur, etc.
• Phones with and without camera flash
• Keyboard differences (physical vs. GUI)
• Different carrier ROMs
• Non-standard commands for triggering standard functions, e.g. GPS on Hero vs. other Android devices
• Uncooperative graphics drivers

These differences have created significant compatibility issues, so much so that developers are creating separate builds and maintaining separate code for different Android devices. The best analogy for the problems faced by Android developers can be found in Michael Arrington's article:

Imagine if Windows developers had to build different versions of their applications for different PC manufacturers. Or even different versions for various models by a single manufacturer. That’s what some Android developers are saying they are facing now.

I'm not so sure Android development and testing is any different than with most software development - witness the issues developer face with supporting their web apps across different browsers. But I do think that their expectations are reasonable: an OS developer (in this case Google) needs to do all they can to standardize how the application layer will behave. Certainly Apple's iPhone has set the bar high in this case and ultimately developers will decide how much they are willing to put up with.