You can't teach computers to think like your customers, or, apparently the American Public

With some fanfare last week, my competitor, Biz 360 predicted that Adam Lambert would win the American Idol competition using their automated sentiment analysis system. As always, I was  highly skeptical about any claims of accuracy and predictability from automated sentiment analysis. My premise is that you might teach a computer to think like a human, but you can't teach it to think like your customers. Apparently I was right.
Biz360 refutes my argument but I still question the statistical validity.

Comments

[Kathleen]

Thanks to Jeffrey Catlin for articulating that point so well, couldn't agree more. Also, for Susan, check the following link - Biz 360 did in fact predict correctly for the 3 weeks leading up to the finale.

http://marketiq.biz360.com/2009/05/our-american-idol-finale-prediction/

[Jeffrey Catlin]

As a vendor of automated sentiment software, I can say with 100% certainty that machines don't think like people (and probably never will). But jumping on Biz360 for getting it wrong in this case seems a thinly veiled poke at a competitor, they did the analysis and made a single prediction which turned out to be wrong. You could be just as wrong having done the whole thing by hand. There are places where automated sentiment is great, and places where it's not, but making sweeping statements based on a single piece of data is at best careless.

[Don Bartholomew]

While I agree with Katie on the issue of accuracy of automated sentiment analysis, in this case the question is not so much the accuracy of their sentiment grading but to what degree is the population of online commenters representative of the population of AI voters. There are most likely some differences that make it impossible to be predictive.

[steve dodd]

The problem with the entire comment is that the sentiment was not determined at the point the voting took place. All any sentiment process can detect (manual or automated) is what is being "felt" at a specific point in time (or historically if data is available). When there is still something to happen that will determine outcome I'm not sure how future predictions are possible unless you can control the outcome of the future activity.

[Susan Getgood]

I don't watch American Idol but it's my impression that when it gets down to the final 2 it is a horserace, with a very small margin between #1 and #2. No matter how you predict you have a 50% chance of getting it right. If automated tools really wanted to prove accuracy, they should try and predict when there are still 3 - 6 contestants left.

Not gonna happen is it?