« Does Sentiment Matter? | Main | What We Learned by Trying to Automate Relationship Measurement »

May 04, 2011

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d83451658a69e2015432146360970c

Listed below are links to weblogs that reference Should You Analyze Sentiment? With Computers or Humans?: 7 questions to help you decide.:

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

David Geddes

Katie -- While you raise useful questions for thought, I am troubled by two issues. First, "accuracy" is not an appropriate measure for sentiment analysis. The right statistical test is an inter-observer reliability statistic, of which Krippendorff's alpha is one specifically adapted to content analysis. Second, you and many others throw around statistics such as "most automated sentiment analysis tools get sentiment right about 40% to 60% of the time." This begs the question of the level of agreement among two humans analyzing the same data. Are we sure that they agree more than 60% of the time?

Katie Delahaye Paine

I can't speak for anyone else's systems, David. But Yes, we conduct intercoder reliability testing using Scott's Pi, for our human readers to ensure that they agree at least 88% of the time. If any individual reader's scores fall below 80% they are given one month to improve. If during a second month they don't improve, they are dismissed.

evalesco

As a network monitoring system few things beat SysOrb from Evalesco if any. This network monitoring software is highly scalable, has incredible features and is at the same time one of the easiest to use and therefore fastest to learn. Free trial.

David Phillips

There are a number of issues to be addressed.

In the first instance any sentiment analysis needs inter-coder reliability with a perspective bias. Factual content variance does need to be considered.

Interactions are not limited to customers. The financial and political spheres, not to mention the influence of competitive politics are important too.

This is possible using semanics and harder with human coders.

The problem with using Grunig's six components of relationships is that it is not really very good in content analysis compared to much more granular applications of values analysis (bayesian derived mutuality of semantic concepts being one approach).

I guess, the automated systems that you are using are based on POS tagging and word counts which will always be limited in their applications.

Google,Bing and Amazon abandoned such approaches a number of years ago.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Get us every month as a free email newsletter:

  • Just type your address here and click GO
    For Email Marketing you can trust

New Every Morning at 7:30 am...

  • The Measurement Standard Daily Digest is an automatically compiled collection of public relations measurement-related posts from around the Web.

Great Minds on Measurement

  • Niels Bohr“When you are done changing, you're done.”
    -- Benjamin Franklin

Become a Fan

Recent Blog Posts on Measurement