Sometimes defects aren't what they appear to be. I'm sure developers have countless stories of how defects they set out to fix were not at all like the defect report described. We ran into just such a situation recently while testing a twitter-enabled iPhone App - Photo Tweet. Photo Tweet would mysteriously, and seemingly arbitrarily, fail to upload photos to
Twitgoo. It was one of those situations where one day a feature works and the next day it fails. And, of course, the developer had not made any changes.
Most of the time when you find a bug and the developer claims not to have made any changes, it's always best to quiz the developer with "what-if" questions. That allows you, as a tester, to avoid calling the developer's integrity into question. I usually try questions like this:
- What if during last night's build, an older version of a component was used?
- Even though you didn't make a change for that feature, what if some other change caused this bug?
Usually this sort of brainstorming session can tease out needed clues that lead up to finding the root cause of a mysterious bug. If that doesn't work, then you're left trying to find out what else could have contributed to the defect. Such was the case with Photo Tweet and it's inability to upload photos after a period of time. And so, a bug was written up that said Photo Tweet would stop uploading photos arbitrarily.
Subsequent conversations with the developer, including some what-if questions, lead us to believe that no changes were made for this feature and the bug resided outside the developers code. Well since Photo Tweet uploads its photos to Photo Bucket's Twitgoo service, we suspected they had instituted some sort of throttling for uploads. This is reasonable to expect given the popularity of Twitgoo. Maybe they just didn't want rogue apps flooding their service, and apps like Photo Tweet were paying a penalty for the nefarious activities of others. We could not have been more wrong about the root cause of our defect.
We reported our defect to Twitgoo and asked if they had instituted an upload limitation. I was amazed at how fast these folks responded and how helpful they were. Initially they were stumped since this was the first they heard of this problem and HAD NOT put a limit on uploads. At that point, I expected that they could not help any further and we would be left with having to do more debugging. But instead, Michael P. Clark, their SVP of Technology and Partnerships rallied his technical folks to find out what was at the heart of this problem. The next day I got an email from Justin Hart, a Sr. Engineer for Photobucket, informing me that they discovered an unannounced change to Twitter's service. Twitgoo relies on Twitter for those users wanting to "tweet" the photos they upload to Twitgoo. It turns out that Twitter had made a security enhancement. Michael and Justin found this issue very quickly and pointed me to a
threaded discussion concerning what Twitter did. Much to Twitter's credit, they quickly made this issue public and extended an apology and an offer for input from developers. Here's a snippet of their statement:
A change shipped last week that limited the number of times a user could access the account/verify_credentials method [1] in a given hour. This change proved hasty and short-sighted as pointed out by the subsequent discussion [2]. We apologize to any developer that was adversely affected. Given the problems, we want to fix this in a public and transparent manner.
We learned a lot from this experience. First, the folks at Twitgoo are fantastic and serious about providing a great customer and technology partner experience. Second, some defects are certainly not what they appear to be. Third, when testing products that use APIs that connect to services that connect to other services, it's hard to determine root causes no matter how well intentioned your defect characterization process is. Finally, as you investigate why a defect occurs, don't assume you know the root cause until you have all the facts.
