Five Twitter Sentiment Analyzers Reviewed

*

Twitter Search, Social Mention, Twendz, Twitter Sentiment, and Twitrratr

by Chris Near, Director of Research, KDPaine & Partners

Don't miss Chris Near's other article on measuring Twitter, also in this issue of The Measurement Standard: "Which Twitter Profile Analysis Tool Rules the Nest?" The sentiment analysis tools in the article on this page look specifically at the tone of individual tweets, while the Twitter analysis tools of the other article rank a person's entire Twitter profile in terms of popularity and engagement.

There is big, big, big demand for the ability to measure sentiment in social media. Sentiment -- also called "tone," and typically rated as positive, negative or neutral -- is shorthand for what the world thinks about something. And everyone, from junior high kids ("Do they like me or not?") to billion-dollar corporations ("Do they like our latest product?"), wants to know what the world thinks of them.

So, now that Twitter rules the social media roost, it would be very, very handy to have an accurate sentiment analysis tool that could quickly and easily gauge the tone of Twitter conversations.

When my company, KDPaine & Partners, wants to do very accurate measurement of Twitter, we have to use human readers. It's the only way to really understand the language. But, to have human eyes read and rate every tweet often takes a lot of time and money. For fast and free measurement, there are automated online sentiment analysis tools, five of which I review below. They, however, have their own set of problems.

Yeah, right. I got your total accuracy right here!

For a computer to determine sentiment for traditional print media is difficult enough. But when you try to get a computer to understand and interpret social media, it gets infinitely harder.

Social media, Twitter especially, is a very casual format, one that lends itself to inside jokes, symbols, emoticons, abbreviations, and jargon. Oh, and don't forget about sarcasm; what might appear positive might really be negative, and vice versa. Add to that the fact that Twitter only allows 140 characters (fewer characters and words makes it more difficult to assess tone) and errors in automated analysis go way up. So we may never attain total accuracy in measuring sentiment for Twitter. (Even with human readers you often don't get 100% accuracy.)

Most automated Twitter sentiment analyzers are very up front about the difficulty of what they are trying to do, and that they will be changing how they do it in the future. Before we get to the reviews, let's think about...

The Ideal Twitter Sentiment Tool

To my way of thinking, and for the type of work we typically do at KDPaine & Partners, the perfect Twitter sentiment analysis tool would include the following features:

• The ability to search by any word, abbreviation, symbol, or emoticon that would ever occur in a tweet.
• Advanced search features, including by date, by user, by hashtag, and by tone.
• The ability to distinguish between posts from a person/company, posts to a person/company, and posts referencing a person/company.
• Results charts with colors that distinguish between different sentiments.
• A dashboard with quick totals for each tone type, as well as overall numbers/percentages.
• The ability to search over long periods of time.
• Instant charts to show trends and relevant occurrences.
• At least 80% accuracy. 90% - 95% would be better.
• Real time updates.
• It should be free. (That's not too much to ask, is it?)

None of the tools reviewed below come even close to meeting this wish list. But some are on the right track. Here they are:

Twitter Search

Twitter's own Twitter Search sentiment option does not use words to gauge message tone, rather, it is limited to only those tweets that include certain characters used to symbolize mood, e.g., :) or :( or :D or :P. For this reason, when you use Twitter Search to search any given topic, you may find only one or two results showing positive or negative tone, because they are the only Tweets that used the above symbols.

On the day I used Twitter Search to run a sentiment search on #unfollowdiddy, it was the most popular topic of the day, with 2629 tweets (according to #hashtags.org.) But my search brought up only a single tweet, which Twitter Search mistakenly rated as positive:

Knot2serious: @TheTZA Glad to see you doing the #unfollowdiddy thing too! :D He really has a crappy ego & no talent. He's all hype. So, how are you? :) :P

The eye of the beholder: The above tweet exemplifies a major problem with sentiment search: Sentiment is many times a matter of perspective. Most sentiment analyses grade an item's sentiment based on the overall tone of the message and not necessarily the sentiment towards the subject of the message. For example, the above tweet has three positive sentiment symbols, yet the words make it clear that the tweet is actually extremely negative towards P. Diddy. To be fair to Twitter Search, all the tools reviewed here suffer from the same difficulty.

(#unfollowdiddy was a controversial effort to encourage people to stop following P. Diddy on Twitter. Each post on the subject usually included a reason why people should unfollow him. Taken from P. Diddy's perspective, most of those posts would be considered negative. Taken from the perspective of the person who started #unfollowdiddy, most of the posts attacking P. Diddy would be considered positive.)

Conclusion: Pretty much useless. If you are trying to gauge the overall tone of a topic or Twitter user, then Twitter Search is not the tool to use. Symbols should be part of a sentiment search, but they shouldn't be the only part. Twitter Search is sentiment search in its infancy, at best: It's a fun Twitter toy people can use if they want to see how many happy or unhappy faces were used in a tweet.

Social Mention

Social Mention, a social media search platform, assigns a tone/sentiment to every tweet: Positive, neutral, or negative, distinguished on their charts by color (green, gray, red). They also provide a sentiment ratio designed to give you an overall feel for the tone of a given topic. Social Mention offers an exhaustive breakdown of information by top keywords, top users, top hashtags, and top sources beyond Twitter. Their breakdown column lets you sort by sentiment as well, allowing you to see all the positive, neutral, or negative tweets grouped together.

According to Social Mention's creator, Jon Cianciullo, they, "use textual analysis, symbols, emoticons, and a few other things," to measure tweet sentiment. "We developed a method to achieve the highest level of accuracy that was reasonable to implement. We based it on some great open source projects and designed it specifically for the type of media we process. We leverage word, symbol, and phrase analysis to yield a ratio which is then used to grade the overall sentiment."

I tweeted back and forth a little bit with him, and he told me candidly that sentiment analysis is extremely difficult. Their research has found Social Mention's sentiment analysis to be roughly 60% - 80% accurate. He also said that they are developing a new Social Mention product which will provide more analysis, saved searches, and reporting.

Based on a #unfollowdiddy analysis using Social Mention, those numbers seem to be accurate. Social Mention's Sentiment Ratio was listed as 2:1, twice as much positive as negative. Of course it's all a matter of perspective as I noted above, (for Twitter Search), and as demonstrated by the following examples:

• Mistakenly listed as having positive sentiment:
  • 1. @jamalahmad lol yeah the #unfollowdiddy is getting pretty old and quite mean, why cant they just not follow him like us?
  • 2. #unfollowdiddy cuz it's his fault that the Palm Pre ain't out in stores yet, they had to put a Let's Go button on it, BOOO, lol...
• Mistakenly listed as having negative sentiment:
  • 1. got lots done...time to relax but I won't #UnFollowDiddy!! Go diddy go!
  • 2. #unfollowdiddy or #followdiddy.. oh shit damn he so gooood lol

Conclusion: Moderately useful. If 60-80% accuracy is close enough, then Social Mention is a tool you can use. It's the best overall among the five reviewed here.

Twendz (beta)

The Twendz site, a project of Waggener Edstrom Worldwide, is an in-the-works-project entirely devoted to measuring sentiment in Twitter conversations. It uses a combination of keywords and symbols to compare and cross reference against a dictionary to make an educated guess on the sentiment of the posts. It measures tweets as positive, neutral or negative. Beyond that it breaks down sentiment by topic and then by reoccurring words found in the tweets. Twendz lets you highlight all tweets by individual tone type as well.

Twendz is really geared toward current topics. When a search term is entered, it pulls the 70 most recent tweets with that term, and then updates the results as newer tweets on that subject come in. There is currently no way to look at tweets prior to the first 70 they pull up. My impression is that Twendz is somewhat less accurate than Social Mention.

Conclusion: May be useful, depending on your needs. If you are only interested in the most recent 70 tweets, great. If you want to do any history at all, this one can't do it.

Twitter Sentiment

Twitter Sentiment, also entirely devoted to measuring sentiment in Twitter conversations, is, as it says, "strictly a school project." Like many of the other sites it ranks sentiment as positive, neutral, or negative (distinguished by color – green, white, red). It's probably the simplest looking of the five sites reviewed here.

It offers a feature that allows you to change the sentiment results if you think they aren't correct and submit your feedback. That doesn't mean that your edit becomes permanent, but I like the idea.

The bad news -- really bad news -- is that no matter what topic you search, Twitter Sentiment only pulls up one page of results: If you click the "next page" button, it starts your search over. So your results are limited to about 17 or so tweets. With such a small sample, it's hard to determine how accurate Twitter Sentiment is. The good news is that the site's "submit feedback" button works. So you can tell them the "next page" button doesn't work. So maybe they'll fix it.

Conclusion: Very limited usefulness. For a school project, it probably gets a A. But in the real world, it's only useful if you are only interested in the most recent 15-20 tweets. I'd use Twendz before I'd use this one.

Twitrratr

Twitrratr is another site devoted to tracking sentiment on Twitter. It cross-references your search term against a dictionary of positive and negative keywords. It offers a clean, polished dashboard that shows the total number of tweets containing your search term and how many were positive, neutral, or negative. The individual tweets are then shown in corresponding columns, with the category-triggering words highlighted.

At the bottom of the page are links showing exactly which words and symbols count as positive and which words and symbols count as negative. Twitrratr shows 174 positive words, abbreviations, and symbols, and 185 negative words, abbreviations, and symbols. Certainly not an exhaustive search and a lot of room for contextual error, but at least you know what you're dealing with. My impression is that there were not enough tweets categorized negative, and that Twitrratr is less accurate than Twendz.

At the top of the page they show the number of all the posts, but I couldn't find any way to actually see any of the other posts beyond the results that appear on the first page. Under their About link they make it quite clear that this site is a Startup Weekend project, and a work in progress. At least they are aware of their limitations and freely admit it.

Conclusion: Only moderately useful. Smooth layout, and the total numbers are nice, but still limited by lack of history. I'd use Social Mention rather than Twitrratr. A bit limited in concept right now, but future potential to watch for.

And finally...

The current efforts to track sentiment through a machine are a great beginning and, even for the tools I ranked poorly, what they've accomplished thus far is quite impressive. The best of the group is Social Mention, but it's certainly not accurate enough for many purposes.

The big picture here is that, although there is demand for free automated sentiment search, the technology to pull it off isn't quite there, at least not yet. What it comes down to is: What percentage of accuracy are we willing to live with?

Don't miss Chris Near's other article on measuring Twitter, also in this issue of The Measurement Standard: "Which Twitter Profile Analysis Tool Rules the Nest?"

Chris Near is Director of Research for KDPaine & Partners. Chris recently graduated with his master's in communications and currently devotes most of his time to measuring PR and developing social media methodologies. That is, of course, when he's not at home tending to his lovely wife, Valerie, or chasing around his tireless two year-old son, Brendan.

* Thanks for the icon, SmashingMagazine