I have been extremely frustrated by the App Store review system. At first anyone could post anything they wanted about your app, whether they bought it or not. In September 2008, Apple stopped accepting reviews for apps that the reviewer hadn't bought. Good right, but then Apple didn't actually go back and remove old reviews recorded without owning the app until late February 2009! Great, another baby step toward a more fair system to app producers and consumers. But it still wasn't fair to app producers for a number of reasons:
- Apps were getting rated on deletes from devices. No way to log a rating without deleting.
- App rating distribution, meaning how many times your app had been rated a particular star level, weren't visible
- Reviews and ratings had no visible date associated with them
- Reviews and ratings had no visible app version associated with them
- The relationship between reviews and ratings was unclear
iTimeZone All Ratings
What Changed March 10, 2009 - March 13, 2009?
Apple slowly trickled out an overhaul of the review system for the App Store. It has been very strange how this has been done. Bits of functionality here, more there. It seems by March 13th, 2009 (Friday the 13th, must be an omen) Apple started breaking out the distribution of rating scores by stars. In the screenshot on the left, current version means iTimeZone 1.2.1
Apple slowly trickled out an overhaul of the review system for the App Store. It has been very strange how this has been done. Bits of functionality here, more there. It seems by March 13th, 2009 (Friday the 13th, must be an omen) Apple started breaking out the distribution of rating scores by stars. In the screenshot on the left, current version means iTimeZone 1.2.1
These changes made great strides forward, answering at first glance all of my bullet points above. Sadly, I have to conclude Apple still has a slight miss here.
What's Still Wrong?
Here's why based on my analysis of the data for iTimeZone:
But you don't take my word for it, download the spreadsheet in your format of choice and look at or mess with the data:Here's why based on my analysis of the data for iTimeZone:
- Average Customer Ratings do not include Customer Reviews. The only way the math works, that Average rating for all versions is 3 stars, is if you exclude customer reviews. A 3 star rating has to at least fall between 3.0 and 3.25. Since we don't know Apple's rounding policy, and we know Apple does half-stars, it is entirely possible anything between 3.0 and 3.49 is a 3 star rating. Luckily for this exercise, the average of iTimeZone's customer reviews is 4.26. That happens to be just enough to make the combing ratings and review score 3.5, yet iTunes only shows a 3 start average rating for all versions. Thus, the aggregate rating can't include customer reviews, only ratings. So Customer Ratings are averaged for the current version, and across all versions, but customer reviews are not, they are only listed. If a customer took the time to write a review, surely that star score should be included in the ratings? That is just not intuitive at all. I would have to recommend at this point that if you write a review, you also leave a rating.
- Tracking Reviews/Ratings by Version is too literal. The accepted convention for version numbers, and the one I am using follows Apple's generic versioning scheme. Major.Minor.Revision, X.Y.Z, 10.5.6, 1.2.1. Major means a platform release, minor means a feature release, revision means changes/fixes to existing features. Mac and iPhone apps are thought of feature stable when X.Y remains the same. This is not controversial, except the App Store is now boxing reviews by X.Y.Z. So reviews for version 1.2 of iTimeZone mean the current version doesn't have any reviews because the current version is 1.2.1, and there are no written reviews yet for that version. Current version literally means the one on sale, there is no intelligence behind it. This is going to result in developers being reluctant to release fixes quickly, and instead only take the version review reset hit when they are going to release features. The other possibility is gaming the version system so that the app gets updated, but the version number doesn't change so reviews don't get reset. I haven't investigated this thoroughly yet, but I am sure it has already crossed more than my mind. Sure, developers could try and game the review by version system the way I propose, but if Apple had clear rules posted on versioning and then bounced apps during review for adding features without upping the X.Y version, developers would get the hint.
One More Thing..
I have real trouble drawing conclusions from data like this. Either people really like/love the app, or they just hate it. There is no middle ground. Even if you look at the ratings for all versions, the app is top heavy with 4 and 5 star ratings, but 1 stars are just as frequent as 2 and 3 star ratings. If you rate an app 1 star, you are saying it just flat out sucks, no mincing words. I would only give an app 1 star if it were a total failure at what it was doing. iTimeZone is not that. You may not like how some things are implemented, think things could be done better, but its not an out and out failure by any stretch. I haven't received any feedback from anyone that harsh, so if you think iTimeZone sucks, tell me why at tangerineelement.support at gmail.com. That said, 1 star ratings seem frequent enough I wonder if there isn't a systemic problem where users feel they have to go nuclear on an app.
Conclusion
I can only hope Apple further tweaks the system and fixes the problems I have highlighted. Given that it took 8 months from App Store launch to get the current changes pushed through and the amount of work they surely have ahead of them with iPhone OS 3.0, I am not very hopeful.
Update
Since I wrote the bulk of this, iTimeZone has received 2 more customer ratings both 5 stars. So the current version has a 3.5, yeah for me, but it doesn't change the unintuitive disconnection between Ratings and Reviews.