« Does Sentiment Matter? | Main | What We Learned by Trying to Automate Relationship Measurement »

May 04, 2011


Feed You can follow this conversation by subscribing to the comment feed for this post.

David Geddes

Katie -- While you raise useful questions for thought, I am troubled by two issues. First, "accuracy" is not an appropriate measure for sentiment analysis. The right statistical test is an inter-observer reliability statistic, of which Krippendorff's alpha is one specifically adapted to content analysis. Second, you and many others throw around statistics such as "most automated sentiment analysis tools get sentiment right about 40% to 60% of the time." This begs the question of the level of agreement among two humans analyzing the same data. Are we sure that they agree more than 60% of the time?

Katie Delahaye Paine

I can't speak for anyone else's systems, David. But Yes, we conduct intercoder reliability testing using Scott's Pi, for our human readers to ensure that they agree at least 88% of the time. If any individual reader's scores fall below 80% they are given one month to improve. If during a second month they don't improve, they are dismissed.


As a network monitoring system few things beat SysOrb from Evalesco if any. This network monitoring software is highly scalable, has incredible features and is at the same time one of the easiest to use and therefore fastest to learn. Free trial.

David Phillips

There are a number of issues to be addressed.

In the first instance any sentiment analysis needs inter-coder reliability with a perspective bias. Factual content variance does need to be considered.

Interactions are not limited to customers. The financial and political spheres, not to mention the influence of competitive politics are important too.

This is possible using semanics and harder with human coders.

The problem with using Grunig's six components of relationships is that it is not really very good in content analysis compared to much more granular applications of values analysis (bayesian derived mutuality of semantic concepts being one approach).

I guess, the automated systems that you are using are based on POS tagging and word counts which will always be limited in their applications.

Google,Bing and Amazon abandoned such approaches a number of years ago.

The comments to this entry are closed.

Get us every month as a free email newsletter:

  • Just type your address here and click GO
    For Email Marketing you can trust

New Every Morning at 7:30 am...

  • The Measurement Standard Daily Digest is an automatically compiled collection of public relations measurement-related posts from around the Web.

Great Minds on Measurement

  • Lawrence M. Krauss“I cannot stress often enough that what science is all about is not proving things to be true but proving them to be false.” --Lawrence M. Krauss

Become a Fan

Recent Blog Posts on Measurement