The Measurement Standard: Blog Edition

Yet More Data to Show that Marketers Are Clueless About Data

Just a couple weeks ago we posted about research implying that marketers are out of touch with what their audiences want. Well, thanks to Rob Peterson at Biznology, here is a lot more evidence that marketers are behind the eight ball. His article, "21 Stats Show CMOs Leap Before They Look at ROI," lists, yes, 21 studies that imply that CMOs are having a tough time with, not just ROI, but with making sense of the data. If they even try. For instance, #16: 28% of CMO’s say their decisions are based on gut instinct. Go read the whole list right now. --WTP

How to Spot Bad Research: Tips from the IPR Research Fellows

Here's a great little article by Frank Oviatt at the Institute for Public Relations that gathers tips on How to Identify Bad Research from PR measurement heavy hitters Don Stacks, David Michaelson, Don Wright, and David Dozier.

My favorite: "...assess if the questions are self-serving and biased. This starts with the basic principal of ‘garbage in/garbage out.’ If the questions are not valid or reliable and are designed to bias results, the research is unreliable from the start." Read the article at the IPR website. --WTP

Barcelona Principles Report Card: How Do 12 PR Awards Programs Measure Up?

by Katie Delahaye Paine

Back in 1999, the IPR Measurement Commission began an initiative to improve the measurement criteria for PR industry awards.

Wallop! OnDemand Agency Calculator for PR Measurement: Gimmick or Stunt?

At first glance I loved the Wallop! OnDemand Agency Calculator that purports to calculate The Cost of No PR Measurement.

Seems to be an effective way to encourage thinking about doing public relations measurement.

But, wait a minute. When you actually do the calculator, you'll discover that you just enter one dollar value, after which it allegedly calculates several costs for not measuring.

Isn't that a bit over-simplified?

Well, of course it is. Because the Wallop! “calculator” is much more of an ad and a come-on than a provider of real actionable data. And that’s a mistake, because what PR measurement types really want is decent data and insight to help make decisions. And they know that it takes a bit of work. Not just a bogus calculator gimmick.

So I hope Wallop! encourages more people to think seriously about doing PR measurement. But I wonder: Who is going to trust a company to do serious measurement when it has such an obviously bogus "calculator" as a pitch?

--Bill Paarlberg, Editor, The Measurement Standard
The Measurement Standard is a publication of KDPaine & Partners, a company that delivers custom research to measure brand image, public relationships, and engagement.

5 More PR-Related Things We Need to Get Rid Of

by Katie Delahaye Paine

Last month's article “Five PR-related Things We Need to Get Rid Of” sparked many excellent comments, both here and when it was reprinted in ragan.com. So we thought we’d follow up on the feedback. Plus I’ve got five more things to get rid of.

First, it was amazing to learn how many people are still convinced of the efficacy of some old fashioned techniques...

Still In Love with the Press Release?

I was stunned to find how many people are still in love with the press release...

Bad Measurement: This Is The Kind of Chart that Really Pisses Me Off

Don't you just hate it when somebody crows, “Our numbers have doubled in the last four years!” but provides no comparison with any other numbers so you don't know if doubling is good or bad or whatever?

Well, feast your data-starved eyes on this beauty that arrived yesterday thanks to Silicon Ally Insider's Chart of the Day:

Yeah, so Mac's installed base has about doubled in the last four years. So what? The chart and accompanying text include no information provided about changes in the installed base of any other platforms. Or even about the population of computer users: How has the potential market increased in the last four years? Has, for instance, Apple moved strongly into any new markets during this period?

The text with the chart says:

“At today's big Apple event COO Tim Cook presented the chart below which shows the installed base of the Mac. It's just shy of 50 million right now. He added that the Mac business has been growing faster than the overall PC market for 18 quarters running.”

Which of course does not help much. We are given no idea how much faster the Mac is growing than the PC. Wouldn't have made sense to include the PC market on the chart? In fact, the absence of the PC data makes me suspicious. I’m close to assuming that the Mac grew only .05% faster than the PC, and that feeble difference is exactly why the Mac people don't want to show the numbers side by side.

And, speaking of side by side, is that quote above even comparing like measures? Is the “installed base” for the Mac a comparable measure to “the overall PC market?”

Geeze, charts like this really piss me off. Am I being too picky here?

--Bill Paarlberg, Editor, The Measurement Standard
By the way, Bill Paarlberg has never owned any computer but a Mac. The Measurement Standard is a publication of KDPaine & Partners, a company that delivers custom research to measure brand image, public relationships, and engagement.

 

 

 

Forrester Misinterprets Creator Data: It's percentages, not absolute value.

Several blogs have made a big but mistaken deal out of recently released Forrester social media research showing that the percentage of Creators dropped last year. They are confusing a change in the percentage with a change in the absolute numbers.

This confusion began at Forrester itself: Both Jackie Rousseau-Anderson and Augie Ray make vague statements about how Creator growth has "plateaued," implying that a decrease in the percentage of Creators means a decrease in absolute value.

Other blogs have passed on and amplified this confusion:

-- ReadWriteWeb titled their post: "Social Media Users are Creating Less Content"

-- Mashable said: "New research from Forrester suggests that while participation is on the rise, actual content creation may not be."

Two points on this:

First off, none of these posts use data from the actual Forrester report (which you can order here) to address this confusion. What people refer to is a summary chart of the percentages. Like myself, nobody wants to shell out $499 to read it. So none of us know what the real data says.

Secondly, there are a few level-headed readers who have pointed out this confusion. See the comments to the ReadWriteWeb post.

My own guess is the data shows that—as we might expect—early adopters tend to be Creators, and later adopters tend to be lurkers. As time goes on the percentage of lurkers rises, and the percentage of Creators falls. The really interesting part of this story might actually be about the late adopting lurkers.

--Bill Paarlberg, Editor, The Measurement Standard
The Measurement Standard is a publication of KDPaine & Partners, a company that delivers custom research to measure brand image, public relationships, and engagement.

 

 

Puzzling comScore Video Data: Bad Numbers, P0rn, or What?

Recent data from comScore shows that 178 million U.S. Internet users watched an average of almost 15 hours of online video last month.

Something seems very strange with these numbers, though. Take a look in comScore's chart below at the Minutes per Viewer column, and notice that the total audience average is 882 minutes per viewer. That's 14.7 hours. But now notice that Google Sites is the property with the most unique viewers, and the most minutes per viewer, at 283 minutes. That's only 4.7 hours. And minutes per viewer for all the other properties listed are much lower than that.

So where are all the other viewers that account for so many hours of viewing that the overall average is boosted to almost 15 hours per viewer? There must be a huge number of people not accounted for in this data watching a huge amount of video on sites that are not listed.

Maybe there are an awful lot of people out there watching an awful lot of Internet p0rn. Wait a minute, of course there are. But enough to alter the averages so dramatically?

Or maybe the numbers are off somehow. If I had to guess, I'd say that someone added the average minutes per viewer instead of averaging them.

-- Bill Paarlberg, Editor, The Measurement Standard
The Measurement Standard is a publication of KDPaine & Partners, a company that delivers custom research to measure brand image, public relationships, and engagement.

Top U.S. Online Video Properties by Video Content Views Ranked by Unique Video Viewers July 2010 Total U.S. – Home/Work/University Locations Source: comScore Video Metrix
Property	Total Unique Viewers (000)	Viewing Sessions (000)	Minutes per Viewer
Total Internet : Total Audience	178,148	5,234,655	882.0
Google Sites	143,226	1,884,498	282.7
Yahoo! Sites	55,107	238,322	28.6
Facebook.com	46,571	166,186	18.3
Microsoft Sites	45,558	219,149	40.2
VEVO	43,911	202,091	68.5
Fox Interactive Media	38,136	164,760	27.2
Turner Network	33,442	107,793	25.3
Viacom Digital	30,715	70,617	44.8
Disney Online	28,475	64,104	6.0
Hulu	28,455	153,845	158.0

 

Daily Kos: Stat Detectives Prove R2K Poll Results Bogus

It's a detective story for statistics geeks: A group of math brainiacs dissect a major polling organization's suspicious results to reveal statistical anomalies that could not have happened by chance. Read the article "Research 2000, Problems in Plain Sight" at the Daily Kos. 

Here is part of the introduction:

A bit over two weeks ago, a group of statistic wizards... approached me with a disturbing premise -- they had been poring over the crosstabs of the weekly Research 2000 polling we had been running, and were concerned that the numbers weren't legit... The investigators' report is below, but its conclusion speaks volumes: "We do not know exactly how the weekly R2K results were created, but we are confident they could not accurately describe random polls."

Even if you don't have much math or statistical background, you will find the process these statistical detectives used fascinating. A mystery with statistical clues! I love this stuff. (Thanks to Katie Paine for the hot tip. She loves this stuff too.)  -- Bill Paarlberg, Editor

See more coverage of this situation (Hint: Some lawyers are going to be very busy, and a pollster needs a new job.) in TPM Muckraker:

• Kos Alleges Top Pollster Provided Bogus Data, Will Sue
• Kos Lawyer On R2K: 'He Handed Daily Kos Fiction And Claimed It Was Fact'

I'm 100% Convinced that 50% of PR and Social Media Metrics Are Wrong

...and if this was accounting, you'd be in jail.

by Katie Delahaye Paine

Having recently attended a number of measurement presentations and a variety of conferences, I’m now convinced that most marketers and communications professionals are cheerfully going through life with blinders on. Those blinders are made out of a flimsy gauze of questionable accuracy, incomplete variables, and general apathy. Today’s marketers have taken “fuzzy math” to an all new level.

The most egregious example of today's inaccurate public relations and social media measurement is the use of free automated sentiment analysis. The vast majority of sentiment analysis tools get it right about 45% of the time. Which means that if you use those ”measurement“ tools, then your results are at least half wrong. And if this were accounting, you'd be in jail. (In the interest of transparency and full disclosure, I work with SAS which has a sentiment analysis tool that is 90+% accurate, and is tested against human coders.)

No one seems to mind about this sloppy work because it’s “just PR” or “just marketing.” Well I'm here to tell you it's your job and our industry, and our credibility is on the line. The only way we in PR and communications can be credible is to at least attempt to base our decisions on reliable, complete, and accurate data. Which is why I created Katie Delahaye Paine's Accuracy Checklist for Public Relations Measurement and Social Media Measurement. Go get your free copy right now.

There are four areas where I think most of the industry gets it wrong:

1. Relevancy of content
2. Commercial services omit results
3. Accuracy of content/sentiment analysis
4. Incomplete assessment of variables

Here we go:

1. Spiders Aren't Smart Enough to Pick Your Content

Back in the old days, I’d have a team of people physically looking at publications and selecting only those articles that matched the client's criteria. In other words, the content was actually about the company and/or the product and had some bearing on a customer’s purchasing decisions. Today's electronic searches are a big help, but we still need human reviewers to check up on things.

Unfortunately, most spiders today just aren’t very smart. They aren't smart enough, for instance, to determine that an article that talks about a tax bill to which “small business objects” has nothing to do with the database company Business Objects. And they can’t tell the difference between a spike in coverage because of good PR for “Visa, a sponsor of the Olympics” and “I need a visa to go to the Olympics.”

In some cases up to 90% of what we collect with an electronic search can be irrelevant. You need a very sophisticated Boolean search string to even get close to accurate results, and those still need to be checked by humans. Or else you end up with “I met a really sassy intelligent chick in the Business School,” when you search for ”SAS business intelligence.”

2. Commercial Services Omit Results

Then there’s the issue of omission. The average content provider picks up just a fraction of actual Tweets and an even smaller selection of Facebook threads. If they say they can do better, do your own search on search.twitter.com or just compare with your average Google search. In about 5 out of 6 systems we tested, Google and Twitter outperformed the commercial services.

3. Accuracy of Content Analysis

After you’ve screened out all the crap and have a solid database of mentions, you then need a way to accurately analyze that content. As I said above, the solution for everyone today seems to be automated sentiment analysis. There’s a good reason it is so popular: Wouldn’t it be wonderful to simply hit a few buttons to determine what customers actually thought about your brand? Well, dream on. Most sentiment analysis doesn’t even come close.

First of all, most sentiment analysis systems get it right about 50% of the time, and you get what you pay for. A cheap system will get it wrong even more often. You need a sophisticated system supplemented with human coders to get anywhere close to accurate results.

Secondly, no amount of automated sentiment analysis can tell you what people think. You either need to ask them their thoughts, or hook them up to a sophisticated brain scanner that will ferret out the information. What sentiment analysis does is report back to you the words associated with your brand, and how people are discussing your product or services.

Lots of times computers can misinterpret those words. So if I say I found a wicked cool restaurant, the computer has no way of knowing that I’m from New England and that’s a compliment. Worse still if I mentioned that I saw the play Wicked after eating at that wicked cool dining spot, it would perhaps suggest burning the restaurant and all its occupants at the stake. Most computers don’t understand the irony and sarcasm of today’s conversations.

So what’s an acceptable level of accuracy? If you can get can get computers to agree with human coders 80% of the time you’re doing really well.

4. Incomplete Assessment of Variables

The biggest blinders of all are the assumptions we all make of what “causes” something to happen. So you put a whole lot of effort and energy into a program and you expect web traffic, or registrations, or whatever to increase. And many times it does. But not always. And most of the time you don’t know why because you’ve left out some key variable in your analysis.

Take for instance some work my company, KDPaine & Partners, did for a major national charity. After they did a fabulous PR job and saw overall exposure triple, we surveyed the national audience and found zero increase in awareness. Some would conclude that the entire PR program was a colossal failure. Except that the target audience wasn’t “everyone in America,” it was people with a connection to the military. And when we narrowed our analysis to that target audience, awareness and relationship scores went up, as did likelihood to contribute and volunteer.

We had enough foresight to include a question about military affiliation in the national survey. But if we hadn’t, we’d never have known that the program was successful only among those groups who were actively being targeted.

ATT and Bruce Jeffries-Fox have done a great study on the importance of the interaction of variables, finding that PR and certain key messages actually impact sales and loyalty far more than they thought.

Frequently it’s the presence (or absence) of a key message that has the greatest impact on consumer behavior. But if you’re not tracking your key messages, you have no way of knowing which message is driving behavior.

And just as frequently, it is the presence of conversations about the competition that drives behavior concerning the organization you are interested in. Again, if you’re not tracking the competition, you’ve left out a key variable that you will need if you want your research to be accurate.

(Thanks for the image.)

Research Bias: Another example of why you must consider the source of research.

Biased research results can creep into even the most carefully designed and executed studies. That's why medical research is often double-blind, so, hopefully, the experimenter's bias can't affect the results. In public relations research, bias is endemic because most every measurement project may affect someone's job or budget, and every measurement vendor wants their results to justify their fees.

So here's a review of cell-phone health research to remind all of us just how bias can affect results ("Source of Funding and Results of Studies of Health Effects of Mobile Phone Use: Systematic Review of Experimental Studies" by Anke Huss, Matthias Egger, Kerstin Hug, Karin Huwiler-Müntener, and Martin Röösli.) (Or see a summary here.) It analyzed research on the biologic effects of cell phone use, and found that industry-funded studies were far less likely to identify negative consequences than studies funded by governments and non-profits. Researchers analyzed 57 studies that appeared in the academic literature between 1995 and 2005. Only a third of the industry-funded studies identified a biologic effect with possible health consequences from exposure to cell phone radio waves, while 82% of the government- or non-profits-funded studies found such effects, as did 77% of the studies whose funding source was not identified.

How can public relations measurement identify and reduce research bias? --WTP

Why Public Relations Measurement Is Like Phrenology, or... Just because it is easy to measure, doesn't mean it's the best thing to measure.

PR measurement and marketing measurement often find themselves in a situation analogous to the 19th-century practice of phrenology. Way back then, some people used to think that the shape of your skull indicated your personality or intelligence. It was a very easy thing to measure, so they did. And they hung on to it as a serious area of study for much longer than they should have, because it was far more difficult to measure personality or intelligence by other, more effective, means.

Click-throughs aren't quite as useless as bumps on the head, but still... In the news today is a study by Starcom USA, behavioral targeting firm Tacoda, and comScore that suggests that clickthroughs might not be a very effective measure of the usefulness of online ads. See here for the news and here for Sam Harrelson's ReveNews discussion.

The obvious point here is that clickthroughs, while easy to measure, probably are not very effective at measuring what we would like to measure. The more interesting point is: Why aren't we measuring what we really want to know?

I have noticed that PR and marketing metrics are often determined by what technology makes easily possible, rather than by what we would really like to know. Thus people tend to measure what technology drops in their laps (the easy things like click-throughs or AVEs) rather than something more relevant but difficult to measure (like, "Did the ad with the red type result in more sales among wealthy women?"). Then at some point they realize that their easy-but-less-relevant metric is not very effective, and then they spend a lot of time asking why the metric doesn't work.

What they should be asking themselves is: "What made us think that that metric would work in the first place?"

Instead of focussing on what technology is available (in this case click-throughs), wouldn't it be more productive to think about human thought and behavior first? (I've written about this before: "Measurement's Empty Head: Measurement ignores the most complex part of PR.") First think about what it is that might make PR work -- why it affects people's thoughts and behaviors -- and then decide what might make a good metric. First determine what we really want to measure (regardless of the technology or data available), then figure out a way to measure it. -- Bill Paarlberg, editor, The Measurement Standard

Sex, Surveys and Faulty PR Measurement!

Sex, surveys and faulty PR measurement: Can an article about public relations research get any more exciting than that? PR measurement depends an awful lot on surveys for its raw data, and "The Myth, the Math, the Sex" (in yesterday's New York Times) shows we can't trust that data. I'm as wicked keen about about this article as I was about Freakonomics. And it's about sex, too, did I mention that? --WTP