The Measurement Standard: Blog Edition

Truth, Deception and Lies: Read Errol Morris in The NY Times

One of our themes at The Measurement Standard newsletter is "How to Tell the Truth with Statistics." It is often not easy to do. The problem is that statistics, and often public relations measurement in general, are so commonly used to make a point (or grind an axe) that the temptation is huge and almost irresistible to use them to bend the truth. 

Errol Morris said recently in "Seven Lies About Lying" in the NY Times that:

 "...lying is often identified with the avoidance of the truth or the acceptance of a falsehood." 

You could take that statement and substitute "statistics" or "public relations measurement" for "lying."  Don't get me wrong here, I am not trying to run down our profession. For me the interesting point is that our best efforts at measurement or benchmarking or efficiency improvement are all subject to manipulation by human nature. --Bill Paarlberg

Benchpoint Survey Says People Like Measurement, Are Doing It More

That's the good news.
The bad news: They don't know what measurement is.

by Katie Delahaye Paine

Five years ago, at the first ever Measurement Summit, Don Wright of the Institute for Public Relations and Richard Gaunt of Benchpoint teamed up to assess the state of public relations measurement around the globe. The survey results were not entirely surprising: Many people thought PR measurement was a good idea but not very many people were doing it in any substantive way.

So time passes, new metrics get invented, new methodologies get tried out. And we're all feeling pretty good about the progress of measurement.

Until the results of this year's Benchpoint survey, (sponsored by the International Association for Measurement and Evaluation of Communication (AMEC) and the Institute for Public Relations) the first since the one five years ago, were revealed. My initial response was summed up in my blog post: I'm amazed how many people think clips and AVEs are real measurement. But I have a lot more to say about the results and the conclusions that can be drawn from them.

Read the authors' interim report here (it's a 1.5 MB pdf file). They are still working on a final report (due at the end of July 2009), so don't be shy about telling them what you think.

You can view the charts and read what the authors have to say in the report. There is a lot of hard work in it, and I'm glad they did it. The authors and I interpret the data in much the same way, but my view is somewhat more acidic than theirs...

First the good news:
People like measurement, and they are measuring more now than they were five years ago.

The survey's biggest finding was that 8% more people say they are measuring in 2009 than said so in 2004. Also, the survey asked some general attitude questions about measurement, and found that respondents believe, unanimously and strongly that:

• Measurement is essential to demonstrate value (45% agree strongly)
• Measurement helps inform our communications (48% agree strongly)
• Measurement helps us to assign budgets (34% agree strongly)
• Communications activities are good value (32% agree strongly)

Now the bad news:
Although more than three out of four people think they're measuring, the data shows that only about half of those are actually doing true evaluations.

As we will see below, people are counting more and measuring less. And worse, they don't seem to know the difference. Consider the following results:

1. Criteria of Effectiveness are mostly just outputs

The responses to the following question are quite revealing:

What criteria do you actually use to evaluate your external communications effectiveness?

• 20% said "Hit target media"
• 16.2% said "On time, on budget"
• 16.2% "Measures of message output"
• "Awareness/image," "Client satisfaction," and "Achievement of goals" all tied for third at about 15%

According to the survey authors: "This is a multi choice question which attracted nearly 1600 responses. So respondents clearly use more than one criterion. We are a long way from a universally accepted methodology."

My interpretation: Most measurement experts, including myself, would not consider "Hit target media" and "On time, on budget" as doing real measurement. Probably not "Measures of message output," either. So what this question is telling us is that over half of the people who think they are measuring are actually only confirming that they got coverage.

2. Tools Used: Clips rule, AVEs increase in popularity

Notice the chart under "Tools Used for Evaluating External Communications Effectiveness":

• Counting clips is the most frequently used "measurement tool." Do communications professionals really still think of counting clips as a measurement tool? The notion that this survey even listed that as an option is horrifying.
• The saddest finding, however, was that advertising value equivalency (AVE), which ranked 5th in responses in 2004, now ranks 3rd. People use this discredited technique because it's easy and it provides inflated dollar values that impress people who don't know anything about proper measurement. But few measurement experts believe it qualifies as real evaluation.
• What is even more shocking, given the number of tools and firms out there, is that, as the authors say: "Media Evaluation tools have dropped down 3 places, from 2 to 5... Opinion polls are much the same, but focus groups have dropped sharply from 4th to 11th, with a corresponding drop in users from 12% to 5%. Reputation rankings, a new entrant this year, barely made it into the league table."

My interpretation: What these results reveal is that, while more people believe they are measuring, what they are actually doing is counting more and measuring less. Don't they know any better?

3. Apparently they don't know any better, because only the measurement pros have a clue

When all communicators are compared to just those closely involved in measurement, we see that measurement tool approval ratings are lower for clippings and AVEs. In other words, the people most knowledgeable about measurement do realize the limitations of these two "tools." What we can conclude here is that the rest of the people -- who think they are measuring but are using outmoded or discredited tools -- are simply not knowledgeable enough about measurement to understand what they are doing.

4. Lack of knowledge is also why people just don't measure in the first place

A substantial number of respondents to the survey said they do not measure their pubic relations programs. Their stated reasons for not measuring and the relative rank of those reasons has not changed since the last survey. In rank order the excuses given for not measuring are:

1. Cost, with 31% of non measurer respondents believing it is a "very major" barrier and 42% believing it is a "major" barrier.
2. Lack of time
3. Lack of expertise
4. Fear that others (senior management, other departments) will question the value
5. Fear of the unknown

My interpretation: In many cases, although people cited "Cost" and "Lack of time," the underlying reason may just as well be lack of knowledge. This is because, in a great many cases, affordable and less time intensive options are available. People just don't know about them. This is borne out in respondents' comments (my comments in italics):

• "... the cheap; easy options have little value ... the more sophisticated options are expensive."
  This is not always true, and implies a lack of knowledge about alternatives.
• "Client inability to agree clear method of measuring outcomes." "It is not the value of research that is questioned; rather the value of particular methodologies."
  Here we see the results of the lack of standards; and the lack of knowledge regarding the use of established methods.
• "Clients are not prepared to undertake research and prefer instead to use available budget on promotion rather than measuring."
  Shows lack of understanding, knowledge and experience within senior management when it does not ask for evaluation or does not take time to consider the value of communication.
• "Companies that use metrics have no respect for their PR function. This kind of measurement is usually only the territory of very poor PR people who have trouble impressing upon their clients the value of their work." "No demand. Mgmt values and believes in communications. Measurement of communications does not much capture the interest of C-suite execs."
  These comments seem different, but really both demonstrate a lack of knowledge of measurement.
• "Exposure of Professional Incompetence," "Fear of exposure," "Internal politics; such work seems to have to support management opinion rather than disclose reality."
  Fear of too much transparency

5. People say they like ROI, but they don't use the measurement tools that provide it? Why? Fear of ROI.

The survey found strong agreement that:

• It is possible to calculate ROI on communications
• Businesses need different ROI measures
• Demonstrable ROI means better recognition and bigger budgets
• And there is overwhelming interest in a workable ROI calculation tool

The survey authors conclude that there is great opportunity for a PR ROI tool: "the hunger for a workable ROI methodology is even greater than it was before... An opportunity if ever there was one."

My interpretation: I come to a different conclusion. If everyone is so darn hot on ROI, why aren't they measuring it? I say the reason is, again, lack of knowledge about measurement and measurement tools. Because, there are measurement tools out there that will provide ROI (albeit, not simple or easy to use). So if they really, truly wanted ROI, then they would use measurement techniques that would allow them to calculate it. But instead, they focus on exposure-based methods like clippings and AVEs, because they are afraid of actually learning what the ROI of their programs is. They might find that their programs yielded considerably less R than the I that was put into them.

The survey was conducted by Benchpoint, an online survey company specializing in employee, customer and membership surveys for large and small organizations. It uses a unique proprietary system to provide real-time, fully analyzed results. Clients include international companies, NGOs, professional associations and the UK Government. Download the survey report here. For more information contact Richard Gaunt at: gaunt@benchpoint.com

Katie Delahaye Paine's Survival-of-the-Fittest 2009 Predictions

 

Prepare your resume: Here's a dire dirty dozen things to come for public relations measurement.

1. Clipping services will become obsolete, thanks to Google.

2. There will be half as many social media tracking companies at the end of 2009 as there are now.

3. Half of the 80% of corporate America that doesn't get social media will go out of business. The other half will wake up and start listening to their customers.

4. Likewise, the 50% of corporate communications people who don't get social media will retire or get fired. (The other 50% may very well start having fun again.)

5. Twitter will rule: We'll see at least 100 new words starting with TW (as in Twinfluence, Tweetup, etc.)

6. Within organizations, social media measurement will become a function of the competitive intelligence department.

7. Some form of modeling and factor analysis will begin to be required for every measurement program, as Marketing, PR, IR, SocMed, and internal communications realize that they need to understand how one form of communications influences another.

8. The current whining about how we can't measure social media will shift to whining about measuring mobile applications.

9. Measuring YouTube, Flickr, Twitter, Facebook and Social Bookmarking sites will be a basic requirement of any PR measurement program.

10. Marketing mix modeling and integrating web analytics to traditional and social media measurement will move from fantasy to reality.

11. Measuring PR will stand for "Measuring People & Relationships" as organizations realize that the true benefit of social media and the Web 2.0 revolution is in the connections you can make with your constituencies – internal and external.

12. Due to continued volatility in the market, stock price will no longer be an adequate measure for the effectiveness of IR programs.

Six Defining Characteristics of the Successful Public Relations Professional of the Next Decade

The Future of Public Relations Measurement

PR Stars of the Future

by Katie Delahaye Paine

I had a great lunch with Tom Nicholson, Executive Director of the Arthur W Page Society, recently and, like most people these days, we talked about change. Change in the political environment, change in the communications environment, and change in the PR profession.

Tom speculates that, in another decade, the job of PR within the corporate environment will either be seen as more important than legal, finance and facilities combined, or else it will be as irrelevant as one of my old jobs now is: making sure the 35 mm slides showed up on time for the Sales Meeting presentations. (Yes, I'm old enough to have actually produced 35 mm slide presentations!)

He's right. I think that, ten years from now, many people currently occupying the office of Public Affairs, Public Relations, and/or Media Relations will find the new world order just too weird, too strange and too out of control. For those people -- who grew up thinking they could control the message, manage the media, and spin their way out of a crisis -- this new environment will ultimately be overwhelming, and they'll retire to their eBay businesses, their consultancies, or their organic goat farms.

For the others, adapting will be a rough but ultimately healthy transition, involving a lot of confrontations and personal emotional battles. For still others, they will be (and are right now) growing up amidst the new world order, and they might not notice the transition. (Or, they will be dealing with their own transitions that we can't yet predict.)

At any rate, the public relations environment of a decade from now will require talents and personality traits somewhat different from today. Here are six characteristics of the PR Stars of the Future:

1. They'll know that it is all about relationships.
A PR person who approaches the market from a relationships standpoint will win. They'll understand that the market is a conversation, and that building and maintaining healthy relationships is the foundation of any strategy.

2. They'll make decisions based on data, not gut feelings.
Yes the gut will still be a powerful tool, but in an environment that morphs faster than you can say "Utterli, Seesmic, Plurk, and Twittergrader," the gut will be a very difficult thing to read and rely upon.

3. They'll listen first, listen more, and listen more carefully than did any of their predecessors.
They'll constantly keep an ear to the ground, listening to every whisper of what the customers, media, influencers, analysts, employees, neighbors, and community are saying. They spend at least as much on listening as they do on shouting.

4. They'll design programs, strategies, and messages around the needs and perceptions of the target audience.
Not around the product features, or the whims of a boss.

5. They'll value truth and transparency above all; there will be no secrets in this new era.
Anyone can fact check anything and probably will.

6. They'll stand up for their ethics and values.
If asked to lie, to obfuscate, or to conceal, the successful PR person of the future will quit.

I realize I am not the first to write on this topic, read John Bell at the Digital Influence Mapping Project. I'm curious to know if you have other traits that should be on this list… Please feel free to email them to us.

Measurement During Tough Economic Times

The Measurement Industry

 

What is next for public relations measurement? The best and brightest minds in measurement (including yours!) predict the future of your job, your business, and your profession.

Featuring the views of Nicholas Grant, Angela Jeffrey, Gary Getto, Fraser Likely, Jim Macnamara, Katie Delahaye Paine and Mark Weiner.

Times are tough. As of this morning, the U.S. jobless rate has hit a 14-year high, and General Motors says it will go bankrupt without another tankful of cash from the government. The news is jammed with predictions of even worse economic turmoil and years of tight money ahead. What does this mean for public relations measurement?

We asked an elite corps of measurement professionals to weight in on the problem, and their responses are given below. First, you will want to read Jenny Schade's article (also in this issue of The Measurement Standard):"Captain Measurement to the Rescue! It's time to put on your cape and be a superhero." Then...

What do the blogs say?

The blogs have been buzzing about what tight money is going to mean for public relations measurement, and most seem to agree that:

1. Measurement can and will help protect PR budgets,
2. But, it needs to speak directly to business objectives.

If you haven't already, you can read them here:

• Shonali Burke at Communications Overtones: "...this chilly state of affairs does bring, it seems to me, some golden opportunities for communicators who are willing to put their money where their mouth is. The goose that's going to lay your FDIC-insured egg is measurement."
• Metrica's Kristin Wadge at Measurement Matters: "...use decent measurement to make sure every single penny is working its socks off for you. PR works well in these times, as long as it's working hard."
• Metrica's Claire at Measurement Matters: "If PRs want to protect their budgets they should make sure relevant measurement is planned and built in from the start, adhered to during execution of campaigns... and show contextual ROI on completion."
• Don Bartholomew at Proving the Value of Public Relations: "...2009 will be the year when the pendulum swings from experimentation to accountability. Buckle your seat belts."
• Kari at First Person PR: "2008 needs to be the year of PR measurement... I mean substantive analysis that measures PR programs against business objectives. Then, perhaps, we can recession proof our industry (a bit)!"

What do you think?

Before we hear from the experts, this is your chance, dear reader, to weigh in on the subject. What do you think the future of measurement holds? How do you plan to deal with these tight money times? Feel free to add your thoughts and predictions as comments to this article. To start your thinking off, here's a six-point stimulus package of options:

1. Hard times won't affect measurement:
Companies will still need to get their messages out, and the practice of proper measurement design and analysis is the same, good times or bad.

2. Hard times will mean greatly reduced demand for measurement as budgets are slashed:
Batten down the hatches, 'cause a hard rain is gonna fall.

3. Hard times will mean increased demand for measurement:
Companies will work to tend their brands and will need to know, more than ever, what works and what doesn't.

4. Hard times will mean great opportunities, but only for certain kinds of measurement:
Managers will be desperately seeking ROI to justify their existence, and so measurement practitioners will find a ready market for both innovative techniques and snake-oil.

5. Hard times will be an unpleasant but eventually beneficial dose of tough love for the measurement industry:
Practitioners will be forced to really listen to what their clients need, and respond with clear, focused, and effective products.

6. Hard times will accelerate the transition from old media to new:
Measurement will prosper to the extent that it can not just adapt, but stake it's claim to indispensability in the new media landscape.

And Now, The Experts Speak

From Nicholas Grant,
Chairman, Mediatrack Research Ltd

If you asked me to choose between your six statements (above), I'd go with #5. Where measured comms programs are in place, measurement is unlikely to be affected unless the comms program itself is reduced. The big, huge, gaping, gob-smacking, mouth-watering opportunity however is to get in hard and help PR practitioners encourage the switch into below the line!

...in a world recession, high advertising budgets are often the first to go to the wall, leaving the way open for less costly but arguably more effective methods of connecting messages to markets. The main barrier to switching to PR is the same reason those budgets were not bigger in the first place – low recognition and awareness of the availability and value of reliable, effective measurement. "Where is the proof this will work, where's the evidence, how can we be confident in such testing times?"

From Angela Jeffrey, APR
Vice President Editorial Research, VMS, and
Gary Getto
Vice President Integrated Media Intelligence, VMS

We have already seen a major client accelerate its measurement efforts in the face of challenges to its budgets. Programs that were accepted on faith and commonsense are now candidates for elimination and will be cut if measurement can't demonstrate their value.

So, in this case, hard times have increased demand for measurement. Knowing if a communications program is a cost or a high yielding investment is more important than ever.

And while it is never good to have clients under financial pressure, tough times may just be the catalyst we've needed to accelerate measurement and analysis. For several years, we've been showing the correlation between news coverage and business outcomes. And new research on integrated communications is showing that positive news coverage significantly enhances the effectiveness of paid advertising, in addition to its direct impact on outcomes. So PR measurement is more important, more far reaching, and more relevant than ever.

But despite all this good news, tight budgets put downward pressure on spending and business-as-usual won't get it done. We need to communicate more than ever that PR works, that earned media improves overall communication effectiveness, and that our measurement programs are as good and as sophisticated as that of other marketing disciplines.

From Fraser Likely
President and Managing Partner
Likely Communication Strategies Ltd.

Let's assume that 'hard times' will bring on more issues and possibly crises in the organization. Historically, PR's contribution to the successful management of issues and crises has increased the perception of its value to the organization. Traditionally though, PR does little hard measurement of its role in issue and crises management.

At the same time, let's assume that 'hard times' will limit the dollars the organization spends on bringing new product into production and thus the marketing of new - and probably existing - product. Therefore, there will be less marketing PR - and thus less marketing PR (promotion; publicity) measurement. On the whole, there will be less measurement.

From Jim Macnamara
Professor of Public Communication
University of Technology Sydney

Hard times will not hit the budgets for advertising, public relations or measurement immediately. Around the world there are few reports of major cut-backs yet. However, as budget cycles roll around, some belt tightening is likely. This means that come 1 January for calendar year budgeters, and 1 July for July-June fiscals, there may be some pain.

However, there is a flip side for measurement. Hard times mean scrutiny and accountability are increased. Therefore, measurement showing effectiveness, meeting of KPIs and ROI can help maintain programs and campaigns (Do you hear that PR people?) Measurement can grow your business; it's a gainer not a drainer.

Hard times will also increase attention being given to online media. Direct methods with less wastage and more accountability are likely to receive an increased share of marketing communication budgets. A downer in this regard is that the online measurement world is not doing a good job yet. Despite work by comScore and Nielsen, many marketers are unhappy with the metrics they are receiving and efforts are underway to establish standards for online measurement.

A postscript of that is that lack of standards in PR measurement are likely to be a dampener on growth, as clients struggle to get their heads around ad values, hocus pocus PR values, impressions, OTS, CPIs, favourability ratings, etc. Standards don't mean that vendors have to forsake competitive advantages and develop generic products. Standards could be used productively to set down the basic metrics that should be measured for PR - and what should not be measured such as PR values based on multipliers - as a minimum. Then vendors can add their proprietary 'cream' on top.

From Katie Delahaye Paine
CEO, KDPaine & Partners

This is the third major economic downturn that I've seen since starting my first company in 1987. My father was around for the first one, (and for the Crash of 1929) and he was the first to assure me that it wasn't the end of the world. And he was right. As budgets got threatened, the smarter Comms people -- my clients -- turned to research to determine what was working and not working in their programs and adjusted budgets accordingly.

This economic downturn is no different, except that with social media in the mix, the budget decisions are very different. Increasingly, we're using the benchmarking research we've done in social media to help organizations determine where they should be putting their resources. What we're seeing is that people are looking at metrics like cost per message communicated or cost per engagement to sort out the expensive programs from the cheaper ones. So traditional activities like advertising, direct mail, and sponsorships are under fire, while new media activities like twitter, blogs, etc., are increasingly attracting attention and $$.

Five ideas that might help:

1. Use existing research. SNCR and KDPP both have a ton of benchmarking research that might help you decide which types of social media are best for you.

2. Talk to your web analytics folks. They're sitting on a ton of data and I've found them more than happy to share. They also know statistical analysis, so tap into their knowledge and expertise and you could have a measurement program established in a matter of minutes.

3. If you don't have a web analytics guy in house, tap in to the Web Analytics Association (WAA) and see if they can put you in touch with someone that can help.

4. Talk to your nearest university. They're always looking for real-life case studies and increasingly are using sophisticated content analysis techniques. USC, BYU, Univ. of MD, Michigan State, Univ. of Miami and Univ. of Florida all have excellent comms research programs, for starters.

5. The simplest way to measure the effectiveness of a program is to cut it out of the budget all together and see what happens. If no one complains and no one misses it, then it probably wasn't adding value.

From Mark Weiner
CEO, PRIME Research North America

Since the dawn of marketing, it's been argued that increased spending is exactly what's needed during a difficult economy. The argument, of course, always seems to be made by the marketers whose interests are at greatest risk. Certainly, research and evaluation lead to greater efficiencies during a time when doing more for less is imperative. Similarly, research and evaluation help to prove the value of public relations when everyone is looking for validation.

Equivocations aside, here are my predictions:

- Those PR people who are devoted to measurement and have proven their abilities to apply it properly will continue to invest, although they may take a more demanding position when negotiating terms. Some will be forced to switch to lower-priced alternatives while some will be forced to go without.

- Those with an interest in beginning research and evaluation programs may postpone.

- Some research and evaluation providers will be stressed to continue operating. Those providers with the most diversified foundation will succeed because their business is spread (and their risk is mitigated) across different geographies, client industry categories and service offerings. Those services with a small number of particularly large clients will be most vulnerable as the loss of one such client can be catastrophic. What's more, emerging research and evaluation companies who are now funded by venture capital may see their backers exerting extra pressure or they may see funds dry up entirely.

Measurement Industry Predictions 2008

The Measurement Industry

by Katie Delahaye Paine

1. There will be more partnerships between technology companies and traditional research firms.
Technology companies are quickly learning that pure technology solutions are not what the customers want. The customers want technology to bring down the cost, but they still need smart people to interpret the data and tell them what to do with it. And not just any smart people, but smart people who understand data and public relations and corporate communications.

2. There will be more partnerships/mergers between European research companies and U.S. firms.
To gain market share and social media expertise, European firms will take advantage of the weak dollar and scoop up some of the smaller, still independent research firms, like Biz360, CARMA and Umbria.

3. Some content providers will merge.
If nothing else, exasperated clients will demand that content providers begin to collaborate to provide a better, unified solution. At KDPaine & Partners, we have clients who use four or more different services (Nexis, Burrelles, CyberAlert and TVEyes) to get their clips, because just one of them either misses too many or produces too many false positives.

4. Peace on one front or another will prevail.
Even though measuring engagement is where the focus of most marketer's attention is these days, the battle about how to count eyeballs correctly continues. Comcast and Nielsen are battling it out on one front; Compete, Quantcast and Alexa on another. With luck these wars (and others) will end in 2008.

5. Some unhappy marriages will end.
There's been a lot of consolidation in the industry in the last few years, and not all those relationships are entirely satisfying to either the shareholders, the employees, or the customers. I predict that at least one recent marriage will wind up in divorce court in 2008.

6. There will be more correlations, less causality.
Corporate America's love affair with marketing mix modeling shows no sign of cooling off, but with the multitude of different media to choose from it will be harder and harder to prove causality between PR and marketing results. (At least until someone figures out that they have to add social media measurement to the mix.)

7. Relationships and engagement will be the new ROI.
Marketers will finally give up on defining the ever-elusive ROI and instead will begin to measure relationships and engagement. (Okay, perhaps I've had too much Champagne and am hallucinating a bit on this one.)

8. Joe Thornley will solve the social media measurement debate once and for all.
Actually, I'm just dreaming here, but I do predict that Joe Thornley's gathering in Toronto next May will make great strides in defining some parameters for social media.