iPhone App Beta Testing

Last year I was working on a Nine Men's Morris mobile game called Doublemill. My job was to implement a REST-based server back-end on the Google App Engine. You might remember a couple of posts on that. My colleagues were responsible for coding the Android client. Since I'm an iPhone user and I'm not going to switch to Android anytime soon, I decided to port the game to the iPhone/iPod touch platform.

I'm closing in on the first release of the paid version of Doublemill. The only thing missing is a beta test. I'm past the technical problems, which I encountered when I released a beta version of Doublemill Lite a couple of weeks ago. This is more about how to find people, willing to thoroughly and extensively test your App.

Unfortunately, I can't use Amazon's Mechanical Turk - although I'm willing to pay a small amount for people to find bugs in my App. Amazon doesn't let me use it without a US credit card, so I had to look for an alternative. I came accross iBetaTest.com. I used it for Doublemill Lite and I'm also using it for the paid version. Although the site has little flaws (it's still in beta itself), the idea is great and it works pretty well. However, there is still a problem: the people who are using the site and who are supposed to test your App.

After I added the beta for Doublemill, I received a lot of requests, mostly from testers which are not testing any other Apps or look like they have just signed up. I gave a couple of them the benefit of the doubt and let them in, only to experience, that I wasted my valuable device IDs.

At the time this blog was published, I have only received feedback from 2 of my 7 testers. The other just haven't replied. I know, that at least one has downloaded and signed up for the multiplayer feature, but hasn't provided feedback on any aspect of the App. That's just plain disappointing. After Doublemill Lite, this is the second time I wasted device IDs, it won't happen to me anymore.

After all, there's always something to learn from your mistakes: be picky on beta testers and choose people you know they'll give you valuable, honest and thorough feedback! It takes quite some time to setup a beta test and organize all device IDs, so don't neglect Q&A in your project planning.

I'm fortunate enough to have a couple of people around me, which help out with testing and give me exactly the feedback I need - although it's not always what I want to hear.