Automation may have potential but for now it may be doing more harm than good

Someone on Twitter actually thought I was being ironic when I said that I was looking forward to spending the day at Seth Grimes' Sentiment Analysis Symposium (#SAS11) last week. I guess I'm better known for my skepticism about automated sentiment analysis than I thought. In truth it was a great day, full of fascinating conversations and no shortage of spirited debate. Perhaps most surprising was that the overriding takeaway for me was that while sentiment analysis is making great progress, you still needed humans to clean it up and interpret the data. (The other big point of agreement was that all the sentiment in the world really didn't matter unless it was tied to actual business impact. ) Others have done a better job summarizing the day than I ever could so I strongly urge you to read their write ups. My take on it is: automation is producing interesting results, that might be useful if you're marketing movies or want to get a general sense of the conversation, but I woudn't bet my life or my brand on it.

So it was particularly amusing to come back to a friendly notice from my health insurance company Anthem, clearly automatically generated from data gathered from my pharmacy and equally clearly not vetted by any human (although there is a link to a Harvard Medical School website for more information. What Harvard gets out of this association I'm afraid to find out.) The pamphlet was titled: "My Health Note: A confidential health care summary for Katharine Paine, February 2011."
It then proceeded to offer 4 suggestions, color coded for urgency. The first in red said "Call your doctor" regarding one of my blood pressure medications. The second in yellow referred to the exact same medication and suggested I keep taking it. The third told me to get a flu shot and the fourth said "Ask your doctor about a check up" - On the inside of this pamphlet was a list of all my recent medical activity and a few more details including the point that "you prescription claims show you may not be refilling it often enough."

Never mind the creepiness factor, the strategy behind this completely befuddles me. First of all, it indicates to me that Anthem's data is hopelessly out of date, since it is giving me advice based on data from last summer. Since then, Mr. Anthem, I have in fact had a check up and a flu shot, but your systems are so antiquated that you don't know that? Secondly, it is clearly pushing medication not results. The reason I don't take MORE medicine is that the stuff I do take works, and because I take my own blood pressure at least once a week I know that. But you don't, so stop giving me advice. Finally, in a time when health insurance companies are more hated than lawyers, politicians and banks, it seems extraordinary that you would deploy a system like this without considering the implications on something like this to your overall relationships with your members. Do you understand how much more this makes me mistrust and loathe you? Do you know that this makes me want to advocate for a single-payer system?

Podiums and panels these days are filled with talking heads explaining how they've used social media monitoring and sentiment analysis to learn all kinds of interesting things about their brands. But some of the dirty little secret behind those PowerPoints came out last week when one speaker after another acknowledged the difficulties they'd encountered in getting it right. My personal favorite was the difficulty one vendor experienced in measuring sentiment about the city of Milan. Seems that Milano Cookies generate more positive buzz. Now in the interests of transparency, KDPaine & Partners has a staff of dedicated wonderful people who spend most of their days sorting out differences like "The Cleveland Fed said today…" and "Cleveland is FED up with Lebron James, " to ensure that the our clients don't jump to any erroneous conclusions. The good news is that I get to work with the really smart companies who undertand the need for human intervention before you can draw any false conclusions. The bad news is that the vast majority of companies are like Anthem – far too willing to trust an automated system because it appears to make their job easier. They just don't realize that ultimately the bad data it produces may ultimately yield bad decisions that will inevitably make their job a whole lot harder. To paraphrase Ben Franklin -- an ounce of human intervention is worth a pound of cure.

Comments

[TomHCAnderson]

Thanks for the link back Katie, I very much enjoyed your talk.

While I too have been frustrated with earlier efforts in this field, I really do believe more of us are starting to answer that ever important 'So What?' question.

Blogged about it again yesterday: http://www.tomhcanderson.com/2011/04/29/text-analytics-interview-and-webinar/

Tom

[Vickie Bates]

Hi Katie,

Great discussion on an interesting topic. As you indicate, it's amazing, with all of the tools out there, that companies and brands aren't doing smarter audience engagement.

Recently, I couldn't find a favorite brand at my regular supermarket nor after making trips to several other groceries and left a query on the company's website. Weeks later, I received a form letter with $5 in coupons. The brand has since returned to the shelves, but I'm left puzzled. If the brand can take the time, and spare the cost of the stamp, to ship me a few coupons, why not engage me by answering my question directly?

From a PR perspective, I'm now wondering whether there was a recall they're not telling me about. From a measurement standpoint, I have the suspicion the brand believes this is a positive engagement with a now-satisfied consumer and customer service gets a gold-star ranking for closing out the ticket.

[John Cass]

Hi Katie, I enjoyed your post, great examples of the dangers of relying too much on data that has not been considered.

You talked about your biggest takeaways from the conference. I wondered if you might comment on how the industry is evolving in terms of integration with CRM systems?

[Richard Bagnall]

Great post Katie, wholeheartedly agree.
Best,
Richard

[MiChmski]

Always a pleasure reading your thoughts on this, Katie. Seth also convinced me that although technology has greatly advanced and there are certain tasks that can save time and money and others that (like you) we at Synthesio also have teams working on for accuracy and detail. It is curious that Anthem doesn't seem to have tested their pamphlets on anyone outside the organization...

[Annie Pettit]

I've always been completely open about the downfalls of sentiment, for that matter ANY research method. But, saying that sentiment is faulty because someone failed to exclude Milan Cookies from a Milan City dataset is just an atrocious data quality error, not a sentiment error. Have a look at the link I've shared. Social media research can be great when data quality is your number 1 priority.

http://www.conversition.com/social-media-monitor-analysis-newmr-listening/