The Measurement Standard: Blog Edition

The Measurement Menace of the Month Award Goes To: Advertising Age, TechCrunch...

... and everyone else that ran

"Twitter Generates $48 Million of Media Coverage in a Month."

Recently, Ad Age ran a story that caused quite a stir in the blogosphere. Well-known and highly respected blogger Kami Huyse called the numbers cited in the story "rubbish" and explains why in this post.

VMS VP Angie Jeffrey defended her company's numbers on Twitter this way:

@kamichat @billsledzik @kdpaine @brandbuilder @commAMMO Sry for confusion. The $48MM is media cost of traditional media mentioning Twitter

@kamichat @billsledzik @kdpaine @brandbuilder @commAMMO Media costs correlate btr to outcomes than imps or # of clips; twitter has no costs

Here's what I know:

• VMS uses the cost of buying ad space in print publications as the basis of their "Media Value" -- call it what you like, it's still Ad Value Equivalency.
• There is no such thing as "equivalency" between PR and Advertising. There is no evidence that reading an ad about Twitter (which is impossible, because Twitter doesn't advertise) and reading a story about Twitter have the same impact on a reader. Much of Twitter's recent publicity has been about documents stolen from its offices and have been a source of embarrassment. We have no way of knowing whether those articles were included in the number, but I would question whether those would "correlate" to any outcomes at all. And I question whether anyone, other than perhaps Twitter's competitors, would be willing to actually "pay" anything for those stories.
• VMS says that "media value" correlates to outcomes. But, as far as I know, Twitter didn't inform VMS of its desired outcomes, so how would they know what to correlate to? According to the leaked documents and common wisdom, Twitter wants to generate profit. If VMS had correlated the $48 million to increased profits, there wouldn't be a controversy. As it is, they issued a press release with a big number in it, designed to generate a lot of press for VMS. I've issued similar releases myself. But when I do, I expect the editors at TechCrunch and Ad Age to do their jobs and fact check my release and to reveal the methodology behind my research.

Which is why I'm naming the editors and not VMS, the Menaces of the Month. --KDP

The Measurement Menace of the Month Award Goes To...

The Organizers of SXSW...
...for ignoring measurement.

SXSW Interactive showed a lot of pure, unadulterated enthusiasm. But the sad thing was that there was no discussion whatsoever about the return on investment in social media (other than one panel on ROI of non-profits, see our Measurement Mavens, above). No discussion of value, no conversations about what all these "shiny new objects" were worth. For the most part, the SXSW crew really wanted the whole issue of accountability to go away.

• Where were the stories of the value that all these social media programs were delivering?
• Where was any realistic conversation about how, in tight economic times, we were going to justify all this effort and energy?

To be fair, I had a number of great conversations about the importance of measurement with the folks at Dell, with my non-profit panelists Holly Ross and Geoff Livingston, and with Jason Falls. And SXSW was nice enough to designate me, Katie Delahaye Paine, as "The Queen of Social Media Measurement." Not that I didn't enjoy the honor, but it was just a token. Where was Avinash Kaushik? --KDP

The Measurement Menaces of the Month: People Who Rely on One Number to Judge Success

Delivering one number will not help you keep your job. Delivering the wrong number, however, may help you lose it.

I was at a workshop yesterday at which one of my fellow panelists (who shall remain nameless) relayed that, despite the increasing diffusion of media and complexity of communicating in these times, this person's clients still needed "one number" (a.k.a. Ad Value Equivalency, or AVE) to give to their bosses to demonstrate success. This person even suggested that by not using AVEs, peoples' jobs were in jeopardy.

My initial thought was to name this person the Measurement Menace of the Month. But then I realized that he/she was just doing his/her job, promoting a number and a score that his/her firm sells. (And, believe it or not, I do understand and sympathize when a client demands one number. It puts you in a tough spot.)

In fact, the real Menaces are those corporate marketing and finance people who think that there is value in that one number and would fire someone for failing to deliver it.

Measuring the wrong thing is the fastest way to failure. Perhaps one reason that so many corporations are in trouble these days is that they have considered a PR program successful when it simply delivered a high AVE number. What about communicating messages? Consider the insurance company we worked with that consistently demonstrated ever higher AVE numbers, while their coverage turned more negative every month. Eventually, they laid off half their PR department!

Delivering one number will not help you keep your job. Delivering the wrong number, however, may help you lose it. --KDP

The Measurement Menace of the Month: The U.S. Department of the Treasury

"We just spent $700 billion on a program, now how can we measure it?"

It must seriously frost most PR and marketing departments -- who are constantly hounded to measure the ROI of everything they do -- to watch while our U.S. government gives away $700 billion in Troubled Asset Relief Fund bailout money without a clue as to how to measure what they get in return.

It reminds me of the early days of measurement when clients would come to me and say, "We just spend $100K on this program, now how can we measure it?"

The problem facing Uncle Sam (as it was for my old clients) is, if you don't have a benchmark -- a measure of the level of awareness, preference or whatever it is you are trying to achieve -- to start with, before the dollars are out of the treasury, you really can't measure whether or not the needle has moved. You have to design in the metrics ahead of time, to ensure that the data is accurate, relevant and available after the program has begun.

Because neither the Treasury nor anyone else in the Bush administration put metrics in place ahead of time, we have no idea whether the bailout is working. Worse still, the banks and other financial institutions to whom we loaned the money don't either. They are, apparently, completely incapable of telling anyone what they've done with it or where its gone or what its effectiveness has been.

For that we award the U.S. Treasury our Measurement Menace of the Month Award, which, in this case -- since there is the unfathomably large sum of $700 billion in the balance -- should be the "Most Irresponsible Measurement in the World Award." --KDP

The Measurement Menaces of the Month: The Pollsters and Media Outlets that Knowingly Produce Faulty Polls

Relying on flawed polls conducted by biased media is not the way to run a democracy.

If one accepts the premise that polling is a necessary requirement of democracy, then it is imperative that polls accurately reflect the opinion of the people. In England this is such a fundamental foundation of democracy that all polling is done door-to-door. Sadly, in America we have sacrificed accuracy for the speed of phone and Internet polls. We increasingly find the results seriously flawed, as David Moore has shown so well in The Opinion Makers. (Read our review of this excellent and recommended book here.)

But it's not just the pollsters that are to blame, it's also the media who pay for the polls. The media's job is to sell space, airtime and newspapers. And admitting that polls include the 80% of Americans who haven't yet made up their minds doesn't sell more of anything. So polls persist in asking the wrong questions, biasing results, and misleading both the public and politicians. What good is a vote if it's based on flawed data?

If ever there was the epitome of a Measurement Menace, this is it. It's time that Americans either fix the system or come up with a new one. Clearly, relying on flawed polls conducted by biased media is not the way to run a democracy. --KDP

The Measurement Menace of the Month: David Carriere

David Carriere, author of 7 Steps to Publicize Just About Anything, says that nothing in PR is measureable: "Neither the immediate impact of a publicity campaign or the long-term consequences of its success are actually quantifiable." He suggests that the way to do research is to gather people around the kitchen table and generate discussions. And worse is his recommendation for evaluation: put together a clip book. Talk about a dinosaur! Read more in this review of the book. --KDP

Measurement Menace of the Month

All those CMOs and VPs of Advertising...
...who demand "guaranteed eyeball reach" from social media.

What part of "it's a conversation" don't they understand? Do they count the success of cocktail parties and sponsorship hospitality opportunities by how many people see the invitation? I'm going to go out on a limb here and guess that the success of those sorts of conversations are probably measured in sales, and/or relationship building. So why should online conversations be measured differently?

Social media is not about how many eyeballs you reach, but about how many hearts you engage and the loyalty you build for your brand. It's about the degree to which you are reaching audiences you've never had a clue how to reach before. Audiences that are honestly interested in what you have to say or sell.

I just organized a fundraising concert that taught me this lesson yet again. We had ClearChannel as our media sponsor, and were initially thrilled; with multiple stations here on the seacoast of New Hampshire, they reached a huge number of eyeballs (or, in this case, I guess "ears" is a more accurate statement). But, come the day of the event, attendance was the lowest it's been in 4 years, and significantly lower than when we had a smaller independent radio station as our sponsor.

What went wrong? Turns out that the bands that performed at our concert were played more frequently on the smaller independent station. ClearChannel reached a ton of ears, but they didn't reach the fan base that connected with our entertainment. Lesson learned.

I wonder how long before this month's menaces reach the same conclusion. --KDP

Global Industry Analysts, Inc. Is Measurement Menace of the Month

I made the enormous mistake of spending $75 for one of Global Industry Analysts' "reports" that sounded interesting. It was allegedly going to tell me about the size and growth prospects for the PR and Advertising research markets.

First they charged my credit card and then didn't deliver the report. After numerous requests, they finally sent me the report I ordered. It was all of one page long and told me absolutely nothing: No useful data, no description of methodology, nothing useful whatsoever.

Now, at KDPaine & Partners we occasionally disappoint someone with one of our reports. And if someone complains, we're happy to refund the money. In response to my complaint about their useless report, Global Industry Analysts said this: "Once purchased, the publications may not be returned or exchanged for any reason. All sales are final."

Consider yourself warned. --KDP

The Measurement Menace of the Month Award: Microsoft

Microsoft's
"Engagement Mapping"

(Okay, so I'm hopelessly prejudiced. I just got my new beautiful HP laptop pre-loaded with Vista and I hate it. It crashes about five times an hour, and my experience so far with Vista guarantees that my next laptop will be a Mac.)

That having been said, we are naming Microsoft our Measurement Menace of the Month for their "Engagement Mapping" product. Not that we don't believe in measuring engagement, (we really like that whole concept, see this recent article), but for throwing its weight into the whole discussion without providing any transparency or information as to how they "map engagement," or any details at all as to what is behind this black box. --KDP

Measurement Menace of the Month

Measurement Menace of the Month Award

Don't get me wrong, comScore does a lot of things right, and in the past we've even nominated them for our Maven of the Month. But their latest black box introduction of the "comScore Conversational Media report" does far more harm than good. It purports to measure audience behavior for conversational media sites, but it is proprietary; they provide no details about how the measures are developed. So it simply confuses the marketplace. Once again, old-fashioned thinking and techniques are being applied to the measurement of a new way of communicating and sharing ideas. If you want to look at real measures in the blogosphere, check out BuzzLogic. -KDP