Can I say I told you so?

According to Fresh Networks, Automated Sentiment isn't accurate -- surprise surprise! . Although I wish they'd tested SAS's Social Media Analytics, since our experience is that it is the ONLY solution that approaches human accuracy across the board. analysis-gives-poor-showing-in-accuracy-test/4002844.article

Comments

[electronic cigarette usa]

Well very interesting and informative discussion going on.. Keep up the good work.. Cheers

[kdpaine@kdpaine.com]

David, my minimum level of reliability for our staff is 90% because that's where SAS SMA is and my theory is, that if you're not more accurate than a machine, you can be replaced :)

[David Geddes]

Katie -- In terms of "95% right," what are the inter-observer (or inter-rater) reliability scores you usually find? 95% seems high based on my experience. When feasible, humans are usually best. Technology, used properly, can be a big help.

[kdpaine@kdpaine.com]

Great questions Ulf an David. Ulf,I think that many of the automated sentiment machines already handle foreign language and I'm pretty sure that Swedish is one of them. However, as far as I know, there are no accuracy studies done in Swedish as yet.
David, I don't think that firms like mine are just defending their turf. For big pictures stuff like elections, auto sentiment can work because the terms are relatively easy to define and the volume is such that 75% accuracy is close enough. But when you're trying to judge a brand like SAS vs Oracle it's hard enough to get a computer to collect the right data, never mind determine the difference between sarcasm and irony.
Finally, from my perspective, it has nothing to do with being "overly invested" in manually scoring -- it has to do with what my clients demand/expect and what I deliver. I find that when clients aren't comfortable with the data you give them, they tend to lose trust and ultimately fire you. That's WHY I'm so invested in human coding. A report is much easier to defend when you know that the data behind it is 95% right

[Ulf Lindholm]

Just a brief comment from the outskirts of the world - Sweden. I have looked at several automated sentiment solutions and none sofar has delivered good enough results. Still it would be extremely foolish not to follow the development, since these products seem to get better and better. I'm just wondering - when there is a working solution for the English language, how long will it take before it's actually "translated" into working for e.g. Swedish?

[David Geddes]

Katie -- I have a few comments and questions. First, I agree that auto sentiment scoring alone is never adequate. However, auto scoring can be a very valuable tool to enhance the efficiency of staff. Second,you have mentioned the accuracy of SAS in several posts. Has SAS published any independent studies of the accuracy of their toning? Third, when I look at the literature in computer science -- where they logically are heavily invested in automated scoring -- I continue to be impressed at the accuracy of their models. This implies that, for certain objectives, auto scoring may be good enough. For example, if auto scoring can be used in models to predict stock market movements or election results, this may be sufficient. Could it be that traditional media measurement firms are overly invested in the business model of manual scoring? This is a new media world, after all, when old models are not necessarily the best. Finally, the Freshminds release and white paper are not fully transparent about their statistical methods.