The Measurement Standard: Blog Edition

The Measurement Menace of the Month: Anyone Claiming to Have a Universal Measure of Social Media

Definition of a Menace:

Any vendor or agency claiming to have a “universal measure” of anything having to do with social media.

The latest examples of this type of villain are Starcom MediaVest Group and ShareThis, who earlier this week announced that they've used the magic of algorithms to blend two proprietary indices into a “universal measure of social activity on the web.”

The Measurement Mavens and Menaces of the Month, as Revealed by Ragan's Survey

by Katie Delahaye Paine

Ragan and NASDAQ OMX just released the results of what could be the most comprehensive study of attitudes about measurement in a decade. Click here to download the Ragan_PR Measurement_study. They surveyed 1467 public relations professionals, asking them everything from how satisfied they were with measurement, to “Do we need standards?” The results could be incredibly depressing if there weren’t a few signs of progress. For example, most respondents said that measuring outcomes was more important than measuring outputs. 

Measurement Menaces of the Month: Anyone Using Klout as Their Primary Measure of Performance

Our Measurement Menace of the Month Award goes out this time to anyone that uses a Klout score as a primary measure of performance. Here are a couple of examples.

Klout as an MBO? Get a Klue.

Klout, a controversial free tool that measures online social media activity, caused major agita recently when it changed its algorithm to include more social media channels and to “make it more accurate.” This was to the bubble that is social media marketing what October 23, 1929 was to Wall Street: People were (virtually) jumping out of windows. 

Bob LeBrew captured a number of the comments in his post on Danny Brown’s blog.  My favorite was:

I have been feverishly working at increasing the Klout score for my company, as it is part of my MBOs. Now, with a sudden 12 point drop, it will reflect poorly (and inaccurately) upon my efforts.

[Stunned look of total disbelief] Let’s hear that one again?  

1. Your MBOs are based on Klout???
   What kind of an organization do you work for that relies on a free tool that, until recently, based its scores primarily on Twitter? Sure, go ahead and use Klout as one metric in your mix, but understand it for what it is. As Brad Fay said at the recent Social Media Measurement Standards Conclave, “90% of word-of-mouth is offline.” And, by the way, never put all your eggs in one basket.
2. The 12 point drop will reflect poorly???
   No. What reflects poorly is the fact that you are relying on Klout as a primary measure of success. Klout is just one tool, and it has both strengths and limitations. Remember, Klout recently improved its algorithm. Which means it wasn’t very reliable before. (As anyone who has ever looked carefully at their Klout results will tell you. Mine say that I’m influential on the topic of Syria. Is it just because I am a  follower of @Andy Carvin and am fascinated with the Middle East? I certainly don’t influence anyone’s opinions on the topic.)

Don’t get me wrong, I’m not blaming Klout. (Well, maybe just a little.) They provide a fun free service that lets people rank themselves against other people. It’s essentially Hot or Not for social media types. And just as I wouldn’t evaluate a potential husband on Hot or Not, I certainly wouldn’t consider Klout a valid metric for someone’s hire or raise. But lots of companies do.

A Klout Score: The Latest Fashion Accessory?

Bal Harbour Shops is promoting Kout as a must-have fashion accessory. They required attendees to have a Klout Score over 40 to get into their “oh so exclusive” Fashion Week party.  

Now, I grew up literally surrounded by fashion; my mother was Editor-in-Chief of Harper’s Bazaar magazine. Which would probably qualify me as being at least somewhat influential in fashion. (And, anyway, my Klout score of 67 would have admitted me to the party. So there.)

But here's the wrinkle: I find spending a lot of money on clothes to be morally repugnant. In fact, I buy all my business and dress clothes at Goodwill. You'll never see me speak in anything but a Goodwill outfit. It's my way of reducing my carbon footprint, keeping stuff out of  landfills, and creating jobs in America. And, oh yeah, getting well-crafted classic fashions at low prices. So I would far rather spend a month in a tent at Occupy Wall Street than be a customer of Bal Harbour Shops.

Now, if I blog about that often enough will I be even more influential on fashion? And would that influence ever lead to a purchase at Bal Harbour Shops? Probably not.

So to these examples and to all organizations out there that are measuring success via Klout scores, you are our Menaces of the Month. Congrats. --KDP

###

Katie Delahaye Paine is CEO of KDPaine & Partners, a company that delivers custom research to measure brand image, public relationships, and engagement. Katie Paine is a dynamic and experienced speaker on public relations and social media measurement. Click here for the schedule of Katie’s upcoming speaking engagements. 

Silos are The Measurement Menace of the Month

I was recently in a meeting of five people who worked for five different parts of the same nonprofit. Guess what happened when we started talking about content collection and analysis?

• One person said, “We don’t have anything in place.”
• One said, “We can’t do anything like that.”
• Another one said, “We use Cision.”
• And another one said, “No, we just signed a contract with a new vendor.”

To make this even more frustrating, I knew that this organization was a client of a major data analytics company and was actually already capturing much of the data we were talking about. But, somewhere along the line, someone erected a virtual moat around public affairs to keep it isolated from marketing and development. Which of course was separate from public policy and donor relations. So these five people all thought they had to go out and buy new systems.

Silos Don't Share

Silos are dysfunctional, compartmentalized departments that resist the sharing of resources, research, and data. They are the result of all those invisible but very real walls that organizations build up over time. Beth Kanter calls organizations with lots of silos “fortresses,” because they have trouble talking or reaching out or networking with anyone.

We’re talking here about something that is anti-democratic, limits growth, limits informed decision making, perpetuates myths and fallacies, and inhibits change. If I were in charge of the Department of Defense, I’d redirect all my drone strikes to every silo in both corporate and non-profit America. As one of the major impediments to measurement, silos are a very deserving recipient of our Measurement Menace of the Month Award.

Silos Make Everything More Expensive

As the story of my meeting above demonstrates, silos result in a very expensive problem: Organizations spend a fortune on duplicated data because no one wants to share the information that they have. 

Silos take money away from the mission. Silos keep data away from the people who need it to do their jobs, make better decisions, and be more efficient. Silos make people prisoners of the past, forcing them to rely on what they already know. Silos prevent people from understanding how what they do affects their outcomes.

As a big supporter of non-profits who is very concerned about where my money goes, this sort of thing makes me crazy!  And it should make everyone crazy. To paraphrase President Reagan, the time has come to “Tear down those silos!” --KDP

###

Katie Delahaye Paine is CEO of KDPaine & Partners, a company that delivers custom research to measure brand image, public relationships, and engagement. Katie Paine is a dynamic and experienced speaker on public relations and social media measurement. Click here for the schedule of Katie’s upcoming speaking engagements.

Why Are Sex in the City and Ragan’s PR Daily Alike?

Sex in the City and Ragan’s PR Daily receive our Measurement Menace of the Month Award for promoting make-believe public relations.

I read a disturbing post the other day on the Samantha Effect. It suggests that a surge in applications for PR programs in higher education is driven by interest in the glamorous life of Samantha Jones, a character on Sex in the City.

Thousands of the show’s young fans have been Carried away by the alluring yet utterly fictitious career of Samantha, who owns her own PR agency. To them, the PR profession appears to be a series of fun parties and events, requiring only a pair of Jimmy Choo heels, a little black dress, and a “people personality” for success. 

How do you measure that? AVEs, according to Ragan’s PR Daily.

I’ve never met Jessica Epperly, but judging from her recent piece in Ragan’s PR Daily, her role model is Samantha Jones. “Advertising vs. PR: How to measure the value of editorial coverage” has no more credibility than Samantha’s career. 

Need more proof? The very first comment on Jessica’s piece was from Lauren, a journalism student at the University of Missouri, who says, “This article has taught me more than most of my professors.”  Shame on those professors if a publicist’s puff piece full of inaccurate information can replace classes at a leading educational institution.

What is up with Ragan running such a piece? You know and I know that AVEs are bad, and that every reputable organization, PR firm, and measurement provider has signed onto the Barcelona Principles (which discredit the use and practice of comparing PR to advertising). So why did Ragan run it? Don’t they vet anything?

In fairness to the folks at Ragan, the brouhaha that erupted in the comment section did prompt Ragan to solicit an oped from my colleague Shonali Burke who urged us to consign AVEs to the graveyard. (See also Chuck Hemann's response to that article, with many excellent comments. And if you’d like to learn more about AVEs, we have plenty of coverage here at The Measurement Standard. Start with “AVEs Are Porn for PR,” then see the list here.)

Finally, this issue raises an interesting question about the media that cover our industry: What is—and should be—the role of the media in the coverage of PR? Is it to follow the guidance of the leaders of the industry and help lead the profession to a higher plane? (As PR Week did when they banned the use of AVEs as a metric in their annual award submissions.) Or is it to serve the lowest common denominator by recommending a fictional measurement tool?

Our purpose here at The Measurement Standard is to encourage people to base decisions on reality. And so we award both Sex in the City and Ragan’s PR Daily our Menace of the Month Award, for promoting make-believe public relations.

###

Katie Delahaye Paine is CEO of KDPaine & Partners, a company that delivers custom research to measure brand image, public relationships, and engagement. Katie Paine is a dynamic and experienced speaker on public relations and social media measurement. Click here for the schedule of Katie’s upcoming speaking engagements.

The Measurement Menaces of the Month: People who Perpetuate the Myth that There Is One Simple Answer to Social Media ROI.

As I’ve said elsewhere recently, there is no single easy answer to social media ROI measurement. Right now there are at least half a dozen august bodies trying to define a standard. Which, so far, has meant at least six different standards. (See The Paine of Measurement in this month’s issue.)

ROI Does Not Mean “Results”

To make matters much worse, the term ROI is very commonly misused to mean “whatever you get for your efforts.” One more time for the record: ROI is a standard accounting calculation meaning “percent return on a financial investment.”

The disheartening tendency to misdefine the term ROI causes confusion and miscommunication. Especially between the people who use it incorrectly and the board room. Who definitely do not use it incorrectly. (See, for instance, “The Dumbing-Down of "ROI:" Shel Holtz Makes Sense of It”.)

A recent study by Tom Watson, Professor of PR at The Media School at Bournemouth University in the UK, found that 66% of PR people he surveyed used the term regularly but only 14% used it in the context of financial objectives. (Download WatsonROIpaperIPRRC2011) In fact, among the PR professionals he surveyed, 64% didn't think there should be a standard definition of ROI! Even worse, about 10% said they used an AVE-based formula. Talk about menaces!

Those who try to dumb down social media ROI do themselves and the industry no favors. For a particularly egregious example, take Mia Iverson’s attempt to tackle social media ROI.  Her “formula” for ROI calculation is not even remotely related to real ROI. And to put initials around it doesn’t change that fact.  (Honestly, Mia, “SMROI?” It sounds like something you would eat around a campfire. I’d love to see your COO or CFO’s reaction.) 

Mia uses tweets and likes for the “R” in ROI. She suggests that you take the “desired action” (the likes or click throughs) and divide by the total impressions. To quote: “...you can take your number of followers [impressions] and the number of click throughs [desired actions] and calculate your ROI accordingly. Problem solved.” 

WTF?

ROI is Not That Simple

My point here is not to dump on Mia or blame her or anyone else struggling to simplify metrics. My point is that it’s just not that easy. The elusive ROI Magic Formula is not going to be discovered by just picking up a couple of numbers that are lying around the Internet. As with anything else, it’s Garbage In, Garbage Out. If you aren’t willing to do some hard work up front, you will get garbage for metrics. I can’t say it better than Amber Naslund already has. 

So, folks, calculations like Mia’s are like trying to get to the moon on a tricycle. Yes, it’s a lot easier to ride a tricycle, but it really doesn’t get you anywhere close. Next time someone tries to feed you a simple ROI calculation for social media, please congratulate them on being a Measurement Menace of the Month. —KDP

A really big thanks to Tom Humbarger for the Oreo cookie illustration.

Katie Delahaye Paine is CEO of KDPaine & Partners, a company that delivers custom research to measure brand image, public relationships, and engagement. Katie Paine is a dynamic and experienced speaker on public relations and social media measurement. Click here for the schedule of Katie’s upcoming speaking engagements.

ReelForge is The Measurement Menace of the Month

It astounds me how many people still believe that public relations can be evaluated based on AVEs. After the Barcelona Principles, after years of research and discussion that show that AVEs do not reflect the real value of PR, and after the advent of so many more meaningful metrics, people are still determined to use AVEs.

Some have questioned why I continue to focus on AVEs now that both the Barcelona Principles and the IPR have come out against them. This is the reason. As long as professional PR people and their vendors are using AVEs, I will continue to shine a spotlight on these reprehensible practices. I hope to be able to turn that spotlight off sometime soon, but for now it is obviously still needed.

Take ReelForge, the East African media monitoring company that reported that: “Unprecedented media coverage of the Kenyan Premier League last season puts its value at a fat Sh686 million...”

There is so much wrong with this report I’m not sure where to begin. First off, to dignify the study that ReelForge conducted as a “survey” is a complete misnomer. ReelForge is a media monitoring company. They monitor media, and I seriously doubt they conducted a statistically valid survey of the Kenyan population to determine whether readers of articles about the Premier Leagues were likely to actually spend more on sponsor’s products.

Secondly, without any details behind the monitoring methodology, one has to wonder if they included all the negative articles.

According to Richard Manduku, ReelForge’s operations director, the valuation of the coverage was based on the current advertising rate cards of newspapers, TV, and radio firms. How many companies in Kenya pay for ads according to official rate cards? If it’s anything like the U.S., it’s probably about 5%. Manduku told The Daily Nation that: “The coverage of football on newspapers, TV, and radio is done for free. If one had to pay for equivalent space in the press and time on TV and radio, these are the figures that would be on his bill.” 

But would the Kenyan Premier League have actually been willing to pay that bill?

Katie Delahaye Paine is CEO of KDPaine & Partners, a company that delivers custom research to measure brand image, public relationships, and engagement. Katie Paine is a dynamic and experienced speaker on public relations and social media measurement. Click here for the schedule of Katie’s upcoming speaking engagements.

Martin Ferguson, Australia’s Minister for Tourism, is Measurement Menace of the Month

The victim here isn’t good measurement, but the taxpayers of Australia. Martin Ferguson, Australia’s Minister for Tourism, has been spending his time and lots of dollars to get Oprah Winfrey and 500 of her fans to visit Australia. And now that she’s there, Australia is spending even more dollars to entertain her. Mr. Ferguson’s website includes at least four articles on how much fun Oprah and her audience is having while touring Kangaroo Island and other sights in the land Down Under.

Now there may be a very good economic outcome of all this. But so far Ferguson is justifying the expense with AVEs, telling FT.com that Australia “...had already reaped a benefit of A$85m in equivalent advertising dollars in Australia and the US as a result of the publicity the visit had so far generated,” 

Measurement Menaces of the Month: Companies Who Sell the Dream of Cheap Measurement

 Thanks to whirled musings for the illustration.

Proper measurement requires time and expertise, not just cheap technology.

This month’s Measurement Menace Award goes to all those companies who are selling the dream of cheap measurement. By implying that measurement requires neither know-how nor effort, these companies do our industry a great disservice.

The reason that auditors and specialists are hired to measure financial performance is that it requires expertise and industry knowledge to glean insight from the data. I don’t know a lot of CEOs who would trust a $9-a-month accounting package to run their companies. So why do PR people jump at new tools like Swix, Social Mention, Reputation Defender, and others that promise measurement for less than the price of lunch at Mcdonald’s?

The reality is that good measurement requires brains and expertise as well as data..

Measurement Menaces of the Month: All of Australian PR

(...except Jim Macnamara and Rochelle Burbury)

In contrast to the IPR (who are our Measurement Mavens of the Month), it seems that the PR industry in Australia is heading in the opposite direction, essentially throwing up its hands and caving to the “I just need a number” crowd. Given that Australia has so many great educational institutions turning out more and more PR professionals every year, this is kind of scary if all of them believe that AVE is the only answer.  --KDP

--Katie Delahaye Paine is CEO of KDPaine & Partners, a company that delivers custom research to measure brand image, public relationships, and engagement.  

(Eds. note: That article linked to above does say that the Public Relations Institute of Australia criticizes the use of AVEs. So maybe there are some folks down under who see the light.)

 

Jack O'Dwyer: The Anti-Measurer

Jack O'Dwyer knows PR. Recently he has been writing a lot about public relations measurement, and how useless it is. There has been a lot of fur flying in the rather large space between his blog and that of Katie Delahaye Paine, the publisher of The Measurement Standard.

His recent post "Measurers Want PR Results? Try D.C." says measurement people... 

 ...want proof that PR pros are bringing home any “bacon.” They have all sorts of scales, rulers, calipers, adding machines, gauges and meters—just about everything but an electron microscope—to track what PR people are up to.

We wonder if any of these measurers have a creative bone in their bodies.

Their favorite words are “clear goals” and “outcomes.”

They don’t care if a PR campaign got zillions of pickups. They especially don’t want it measured in terms of what the ad space would have cost.

Jack makes some points that will resonate with a lot of PR people. And I expect he speaks for a sizable population that we measurement types don't often hear from. There are possibly many working PR people for whom the kind of measurement that we promote here is not necessary or important. (And we'd love to hear from you.)

But those who are familiar with measurement tools and techniques, and who want an accurate way to focus and tune their programs, might find his views a little old fashioned, and perhaps not relevant. Just as he finds most measurement not relevant.

For his contrarian views, we just named Jack O'Dwyer our Measurement Menace of the Month. But we do enjoy the debate. It's healthy. 

--Bill Paarlberg, Editor, The Measurement Standard

Jack O’Dwyer is Measurement Menace of the Month

One could argue that Jack O’Dwyer is the Helen Thomas of PR journalism. He’s probably covered every organization, every launch and every trend in PR since Edward Bernais started writing press releases.

And so most of us in the industry sit up and take notice when he has something to say.

Unfortunately, a recent series of blog posts has demonstrated his woeful ignorance of the basic tenets of PR measurement and indeed the recent standard-setting progress of the entire industry.

He began last week by calling me out on my stand against Advertising Value Equivalency (“Paine Calls PR Ad Values 'Cancer' - Wrong!.”) And then dug himself in deeper by calling the measurement industry, “...an uncreative barnacle on the hull of PR,” adding that, “It needs to be put in its place.”

He also asserted that “The job of PR and advertising is to bring the attention of target audiences to the product or service. Print and broadcast clips should be virtually the sole measurement.” At which point about a dozen people on Twitter wondered what century he was living in.

You can read my response here, “The Blogging Barnacle Responds.” I'm sure there will be lots more back and forth on this. (Update: See our post "Jack O'Dwyer, the Anti-Measurer.")

Whatever the dueling blog posts to come, the problem here is that Jack O'Dwyer has missed the entire point of the Barcelona Principles. Not to mention the point of most of my writing and, in fact, the entire careers of almost everyone doing serious, state-of-the-art public relations measurement.

One more time: Measuring impressions is very different from measuring outcomes. The value of public relations is not about the clips or the impressions — it's about how you are impacting the business.

Maybe that impact is on reputation, maybe it’s on relationships and maybe it’s persuading more people to buy the product. The point is that the business impacts of PR go far, far beyond how many times someone's name appears in the paper. And with appropriate metrics you can understand those impacts and then make better business decisions.

Which is why, for his advocacy to reduce the accountability of PR people, we’ve named him our Measurement Menace of the Month. Congratulations, Jack. -- KDP

The Measurement Menaces of the Month: Barcelona Delegates Who Voted "Nay"

The attendees who voted against the principles refused to take a stand: They should all do us a favor and find other lines of work.

The Measurement Menace of the Month Award this time goes to the delegates in Barcelona (some 3-30%) who voted against the Barcelona Principles of Measurement. (You can read the final tallies of the votes at Katie Delahaye Paine's Measurement Blog. And you can read the Principles in this article in The Measurement Standard.)

I know that at least few delegates (including myself) voted against principle number 3 (that rejects AVEs) because we felt that the actual wording didn’t go far enough. But for the others, I just don't understand it: How can you be interested enough in measurement to attend an international summit to set standards -- and then reject the standards?  

This was an historic opportunity to take a stand against stupid, inaccurate and antiquated beliefs that have haunted our profession for years. Those attendees who voted against the principles refused to take that stand. They should all do us a favor and find other lines of work. --KDP

Measurement Menace of the Month: CAANZ

Communication Agencies Association New Zealand

First let me say that there are lots of really good measurement folks in New Zealand, my former partner Mary McNamara being one of them. But this travesty makes you wonder about the overall state of marketing and PR in that country.

Essentially, the CAANZ Marketing Group took a survey and decided that because “everyone does it” that AVE’s should be the standard metric for PR success.

I'm horrified. And, if you read the comments on the blog post in the link above, you'll see that I’m not the only one. Their position seems to be: Never mind if it isn’t accurate, it’s “something,” and therefore should be adopted.

This is like say saying that because everyone has some rice in their cupboard, all people need to eat is rice. Even if the rice is rotten, and regardless of the nutritional benefits or long term consequences of that declaration.

Let's hope that smarter voices prevail in this battle. --KDP

Measurement Menace of the Month is a Presenter Who Recommended AVEs

The Measurement Menace of the Month Award

Runners Up:
You Who Are Not Measuring

First we need to point to the 84% of people not measuring the ROI of their social media activities. This number is appalling. People who aren’t measuring their social media activities have no idea what is working or not working, so they have no clue as to where to invest more money, or where to stop wasting money. In this economy, this is incomprehensible. So all of you out there—you know who you are—read Jim Sterne's excellent book Social Media Metrics (read my review of Jim Sterne's book Social Media Metrics it here) and get with the program.

And the Winner Is:
You Who Are Still Using AVEs!

The Measurement Menace that sent me over the top this month is a particular presenter who made me feel like all my efforts to educate people on the best, most accurate ways of measuring have been in vain.

Picture this, if you will: 250 people are gathered for an all-day measurement conference. Speaker after speaker takes the stage and provides truly excellent advice about how to accurately and meaningfully measure their programs. Then comes this particular presenter and her talk on how to measure branding. The audience is treated to an in-depth and very insightful presentation on how communications and branding efforts resulted in significant improvement in both understanding of the category as well as awareness of her brand.

Then came the question: "How did you show ROI to your CEO?" And the answer: “Well, we counted up the impressions and calculated the Ad Value Equivalency of the publicity.”

Just shoot me!

But seriously folks, are we so focused on counting column inches that we can’t be bothered to calculate the value of increased awareness or understanding of the brand?

It’s really not that hard to calculate the ROI of a program:

1. First, find out how many leads it takes to make a sale and what the dollar value of the average sale is.
2. Then compare revenue, new business, and leads before and after the program.
3. Now use the increase in new leads after the program to calculate the potential revenue it brought in. (Ideally you would do an analysis of variance to determine correlation.)
4. Finally, take the cost of the campaign, deduct it from the potential revenue, and you have ROI.

Please do your own version of this calculation instead of using AVEs. Why compare yourself to the cost of advertising when you can estimate the actual value in sales? --Katie Delahaye Paine

The Measurement Menaces of the Month: Web Analytics Guys Who Fail to Account for Social or “Earned” Media

I was at the OMMA Measurement and Metrics conference this week and heard a number of riveting presentations about the metrics being used by major corporations and research firms to measure the success of online marketing. And, let me say up front, there’s an impressive amount of measurement and analysis going on there. But invariably when I asked these people if any of their metrics take into account the conversations taking place in social media or the earned media coverage their companies get in traditional media, the answer was "No."

Which makes me wonder HOW DO THEY GET AWAY WITH THAT! If someone in PR ever presented data without accounting for the impact of direct mail, or display ads, or SEO, we’d be laughed out of the board room faster than you can say "regression analysis."  So what’s up with their research departments?

Do the folks at SeaWorld or Domino's think that online chatter doesn’t matter? Not likely. I can name a number of companies who know for a fact that it matters, because they measure it: Dell, Southwest, Jet Blue, SAP, Wells Fargo, and Sodexho, and that's just a few. Note what they all have in common: They’re all doing well in a down economy. 

So enough is enough, guys. Anyone within earshot of this post? Next time some web analytics wonk tries to attribute all the marketing or communications success to that banner ad, challenge him or her. Call them on their numbers and point out the inherent flaws in their research. Make them truly measure social media’s impact. And remind them that they have been named The Measurement Menace of the Month. --KDP

 

The Measurement Menace of the Month: AVEs, Yet Again

Oh the silliness of counting ad value when you can count real value instead.

I thought we'd put the AVE debate to rest. A unanimous vote of the IPR has rejected its use, practice and concept. Yet the pestilence raises its head again.

First, this blog post, which suggests using AVEs to assess a very expensive publicity stunt. The silliness of this exercise is that for non-profits, the really useful measures are so easy: Count the donations. Why bother with calculating ad values when you can count up what you really want to know?

And then there's this post by Jason Falls. It makes me crazy to see the words "ad" and "equivalency" in the same sentence. I will always prefer measuring real value – i.e. the true benefit you are bringing to the bottom line of your organization. But, I suppose one could argue that search ranking is "real value," so his ideas are worth considering. I think he may be on to something.

Don't get me wrong. My good friend Jason Falls has done so much great good for the cause of measurement over the years that I could never award him Menace of the Month. It's AVEs that are the Menace. --KDP 

The Measurement Maven and Menace of the Month Awards

The Measurement Maven of the Month:

Philip Sheldrake
Influence Scorecard

Phil pulled together the first Influence Scorecard meeting a couple weeks ago in New York's Buckingham Hotel. And while it didn't have the glamour and glitz and guest list of the Copenhagen Summit, in its own way it may have as much impact on the conversation about social media measurement as Copenhagen will have on the conversation surrounding climate change. (Read details about the Influence Scorecard meeting here.)

What Phil did was put together a nicely diverse set of perspectives, from traditional ad agencies (Diane Meier), to bleeding edge firms like The Conversation Group (Ted Shelton), from integrated communications measurement firms like my own (KDPaine & Partners), to social media tracking firms like Sysomos. And thanks to a collective commitment to stick to our objective, we actually got a remarkable amount accomplished (perhaps the Copenhagen folks could learn something).

And if it hadn't been for the vision that Philip articulated at the beginning of the year, it wouldn't have happened. So, thank you Philip! -- KDP

The Measurement Menaces of the Month:

Everyone Who Confuses ROI with Results, and Measurement with Counting

First, let's get this straight: ROI is not synonymous with "whatever results."

ROI (Return on Investment) refers to a financial return on a financial investment (in dollars, or pounds, or rands, or whatever your currency is) and it's expressed as a percentage -- the percent you made on your money. That's what ROI is. And if someone tells you differently, they're either lying or intentionally obfuscating. Or, to be generous, they're not really sure what they're talking about.

Yes, yes, yes: Nowadays PR and social media people are always "ROI for this" and "ROI for that." And they are almost always in error. In an effort to sound business-like, they've borrowed a term from the hard-nosed guys in accounting. But what gets lost in translation is their credibility. (See Shel Holtz' blog for an excellent discussion of this problem.)

Secondly: Counting is not measurement.

When you google "social media measurement" you'll find some 3 million items. Nary a day goes by that I don't get yet another announcement of a PR firm or a new technology or a new startup that is going to "measure" social media reputation, results, or success with a new "tool."

The reality is that most of this "measurement" and most of these "tools" are nothing of the sort. They not only do not measure reputation (or results or success), but in reality they don't measure anything at all!

Measurement is not counting. Or monitoring. It is not the number of followers, friends, rankings, or scores.

Measurement is a process that requires you to compare results to something -- either to your competition or to your own results over time. You note the change, analyze the reasons why, and improve your program accordingly.

Yes, of course, followers and friends are important. And it is important to monitor and count them as part of the measurement process. But if your stated goal is to increase your presence in social media, then the measure you want to work with is the percent increase in that presence over last month, or last week, or last quarter, or least year. Then you analyze why that change occurred, make an adjustment to your program, and thereby improve your results.

And there's another thing, too: It devalues our profession to suggest, as Robert Wynne did in this article in Forbes.com, that counting something actually demonstrates the value of PR.

And even worse are what I call the "parlor games" of social media. Like the online TweetPsych that claims to psychologically profile a person through deep linguistic analysis of their tweets. Tools like this are just frivolous fun until someone actually proves they have legitimate value. An algorithm is just someone's opinion expressed in an equation, until and unless someone actually tests it. So, for instance, does your mind really dwell on the past? Or do you just happen to use the past tense a lot? Are you really an agreeable person? Or are your tweets just lacking disagreeable words? Not to be lacking in agreeable words myself, I just have to wonder on what basis anyone find these tools accurate or useful? And I would love to hear from TweetPysch or anyone else on this.

So let's all make a New Year's resolution: No more fuzzy semantics. ROI is a financial calculation. Monitoring is not measurement, Counting is not evaluating. --KDP

The Measurement Maven and Menace of the Month Awards

The Measurement Maven of the Month:

Nora Barnes

For her amazing longitudinal study on the use of social media in educational, non-profit, Inc. 500, and Fortune 500 organizations.

When Nora Barnes, (Director, Center for Marketing Research, UMass Dartmouth) did the first study in 2007 it was interesting. When I saw the results in 2008 it confirmed some hunches. But the 2009 data is really fascinating. The full studies will be available on www.sncr.org in a couple of weeks. But until then you can read my summary here.

The Measurement Menaces of the Month:

Organizations that Use Social Media
But Aren’t Measuring

In this year's UMass Dartmouth study, Nora Barnes found that the vast majority of organizations and companies aren't measuring or don't know how. Amazingly, 40% of higher education organizations, Inc. 500, and Fortune 500 companies say that they are not monitoring their brand, products, or company name in social media. In addition, the Tribalization of Business Study of organization communities found that lurkers aren't being measured. For their lack of measurement effort, these organizations receive our Measurement Menace of the Month Award. --KDP

The Measurement Menace of the Month Award Goes To: Advertising Age, TechCrunch...

... and everyone else that ran

"Twitter Generates $48 Million of Media Coverage in a Month."

Recently, Ad Age ran a story that caused quite a stir in the blogosphere. Well-known and highly respected blogger Kami Huyse called the numbers cited in the story "rubbish" and explains why in this post.

VMS VP Angie Jeffrey defended her company's numbers on Twitter this way:

@kamichat @billsledzik @kdpaine @brandbuilder @commAMMO Sry for confusion. The $48MM is media cost of traditional media mentioning Twitter

@kamichat @billsledzik @kdpaine @brandbuilder @commAMMO Media costs correlate btr to outcomes than imps or # of clips; twitter has no costs

Here's what I know:

• VMS uses the cost of buying ad space in print publications as the basis of their "Media Value" -- call it what you like, it's still Ad Value Equivalency.
• There is no such thing as "equivalency" between PR and Advertising. There is no evidence that reading an ad about Twitter (which is impossible, because Twitter doesn't advertise) and reading a story about Twitter have the same impact on a reader. Much of Twitter's recent publicity has been about documents stolen from its offices and have been a source of embarrassment. We have no way of knowing whether those articles were included in the number, but I would question whether those would "correlate" to any outcomes at all. And I question whether anyone, other than perhaps Twitter's competitors, would be willing to actually "pay" anything for those stories.
• VMS says that "media value" correlates to outcomes. But, as far as I know, Twitter didn't inform VMS of its desired outcomes, so how would they know what to correlate to? According to the leaked documents and common wisdom, Twitter wants to generate profit. If VMS had correlated the $48 million to increased profits, there wouldn't be a controversy. As it is, they issued a press release with a big number in it, designed to generate a lot of press for VMS. I've issued similar releases myself. But when I do, I expect the editors at TechCrunch and Ad Age to do their jobs and fact check my release and to reveal the methodology behind my research.

Which is why I'm naming the editors and not VMS, the Menaces of the Month. --KDP