The Measurement Standard: Blog Edition

How Public Relations Measurement Can Win the War on Terror

It's not a new thing to talk about the war on terror as a war in the media for mindshare. But I've never seen it stated as explicitly as in the article "Marketing Osama" which appeared in a recent issue of The Week, and which was reprinted from a story by John Cook originally published in Radar. The point there is that Brand USA and Brand Osama are fighting it out in the media, and this marketing/PR battle is the important front in the war.

Public relations measurement evaluates exactly the sort of metrics that define the strategy that al Qaida has been using to enlarge its support: media impressions, media content, key messages, contributions and new members. Public relations should be not only measuring the war on terror, but, by virtue of tracking the most important metrics, should be defining the war on terror too. (See "Knowing the Enemy: Can social scientists redefine the "war on terror"?")

One of the more interesting aspects of the war on terror is that there is, (so far, that I know of) no way to know if we are winning or losing, or how we will ever know if we do win or lose. And that's where public relations measurement comes in: If we can use public relations metrics to define progress in the war, we can use them to define success and failure. Public relations measurement can win the war on terror by defining when the war will be over.

And here's how we do it: Let's say that al Qaida's media and marketing efforts now generate X impressions of their key messages each year, which in turn result in Y dollars of contributions and Z new members. We gather and track this data for the last ten or twenty years, and correlate it against terrorist activity of all sorts. (Yeah, it's a big job.) We then will understand what level of exposure to key messages (or Osama bin Laden videos) is required to generate what levels of recruits. Or to generate one new suicide bomber.

Then we define success in terms of the metrics. So, we could define "winning the war on terror" as when al-Qaida's impressions, contributions and recruits are dropping. And then we define victory as the point at which these metric fall low enough so that al Qaida can no longer function effectively. If Osama bin Laden doesn't get his message out, or if no one believes it enough to do anything, then we've won. --Bill Paarlberg

The Media Integrity Index

A call for weighting media coverage based on trust and credibility.

by Katie Delahaye Paine

In this article I suggest that media analysis programs would more accurately reflect articles' influence on readers if the integrity of the media outlet was taken into account. This could be achieved by developing a standard measure of media integrity which would used to rate different media outlets for different shareholder groups. The resulting Integrity Index would be used to weight coverage, thereby achieving a more accurate measure of media coverage's impact on consumers.

To understand why an Integrity Index is needed, consider:

Item #1
Why measure what no one believes?

Mazen Nawahi, in an impassioned speech at the 2008 Dubai Measurement Summit, raised the issue of journalistic integrity and the degree to which the integrity of a particular media outlet should be accounted for in any measurement program. Put another way, if:

1. Everyone knows that a given media outlet is going to print whatever a company sends it because they always do or because they're a major advertiser, and
2. Everyone also knows that they will never print stories that don't agree with the governing dogma,

then why would anyone believe anything that was printed?

And if all your media coverage in a particular outlet lacks credibility, why would you include it in your measurement results? For example, in 22 years in business I have yet to have a client want me to include The Weekly World News in its media list. So how do people justify measurement programs that include stories that do nothing to achieve their goals?

Item #2
Credibility is in the eye of the beholder.

Suppose a highly credible blogger gets into a mud-wrestling contest with the blogging equivalent of a pig. The contest attracts a lot of attention, a lot of arguments fly back and forth, and a lot of dirt gets thrown around. Now for the people that are into farm animals, the arguments of the pig might be very credible. However, people interested in learning something of value professionally will no doubt pay more attention to the words of the credible blogger. But the real question is: Do you weight coverage of them both equally in your media analysis reports?

How to correct for influence?

Taking examples of this sort into consideration, it is easy to see why some sort of authority or influence weighting of coverage would be of value. There has been a fair amount of research done in this area. For several years, Angie Jeffrey at VMS has been studying various weightings of media coverage to determine which factors most directly impact sales. Her research first found that share of discussion was effective, but she postulates that advertising rates are more telling because they reflect the importance of the various media. (See this paper and this paper.) I think it's a step in the right direction, but I generally frown on the use of advertising rates (and of AVEs), so I'm skeptical.

The Media Integrity Index: Why not rate coverage on integrity as rated by stakeholders?

I suggest that it's not the ad rate but the integrity of the publication that most determines consumer credibility and thus drives consumer action. And I'm suggesting that to rank a media outlet in importance, we should ask our stakeholders how they perceive the media outlet in terms of integrity.

I raised the question on Twitter and got some interesting responses:

• "It's all relative... What is good for one is bad for the other. It has to be specified for a purpose, customized."

• "Integrity or perceived integrity? Seems there's a difference there that's highly subjective." --Ryan Anderson

• "Integrity is the measure of perceived relationship (believability) of one-to-many listeners. So measure relationships. I would suggest looking at the way political integrity is measured. It would be a perceived metric, measured by polling audience. --Videodred

• "What's the goal? What's the client want it to say? In the case of journalists, what's the relationship?

• "Integrity has to be measured parallel to influence as audience determines credibility of source." --Mike Maney

So in one way, my proposed Integrity Index has to start with the goal of the coverage. What is it that the company or organization is trying to accomplish? Consider it this way: If you don't care who or what your brand is associated with, and you want exposure pure and simple, then you don't care about the integrity or credibility of whatever media outlet is talking about you. On the other hand, if you're trying to establish a reputation, or build brand loyalty and trust, or trying to reach an audience with certain media preferences, then the credibility of media to your audience should matter a great deal.

How to determine integrity?

So how do we determine which media is trustworthy and credible, and which journalists have or do not have integrity? The wrong way is to look at a list of media and assign weights or values to each one yourself. What you think doesn't matter. All that really matters is what your customers or stakeholders or members or constituents think. So you need to ask them. The best way to do that is to use the Grunig Relationship Instrument.

Now you probably don't need to ask them about all 150 publications on your media list. Remember: Never ask a question about something that can't be changed. If The New York Times is what your boss' boss reads every morning, there's no chance in hell you'll ever take it off you top tier media list. Even if your target audience is nine-year-old girls. So start with a list of suspect publications, blogs or any other type of media outlet.

Remember that different stakeholder groups may rate a given media outlet differently. As I pointed out above, integrity is not a fixed standard. For instance, we would expect Republican viewers to rate Fox News and The New York Times differently than Democratic viewers. It is conceivable that you will want to generate different Integrity Indexes for different stakeholder groups.

Below are some survey questions adapted from the Grunig Relationship Instrument. You may wish to develop your own, based on the specific components of relationships you wish to measure (refer to this paper). Ask your stakeholders whether they agree or disagree with each statement as it pertains to each media outlet, then use the responses to rate the outlets.

1. This media outlet treats people like me and organizations like mine fairly and justly.

2. Whenever this media outlet makes an important decision, I know it will be concerned about people like me.

3. This media outlet can be relied on to keep its promises.

4. I believe that this media outlet takes the opinions of people like me into account when making decisions.

5. I feel very confident about this media outlet's skills.

6. This media outlet has the ability to accomplish what it says it will do.

7. Sound principles seem to guide this media outlet's behavior.

8. This media outlet does not mislead people like me.

9. I think it is important to watch this media outlet closely.

10. This media outlet is known to be successful at the things it tries to do.

Industry-wide integrity standards?

Now, the logical question is, "Why aren't we doing this as an industry?" Shouldn't we be factoring in credibility based on some industry-wide standard? It certainly is a more accurate weighting factor than simple eyeballs or ad cost. But the reality is that your stakeholders aren't going to be identical to my stakeholders, and what matters is how a significantly valid sample of your stakeholders feel. It would be awfully complex and difficult to set industry-wide standards for many different stakeholder groups.

On the other hand, if the PR industry wanted to take on a project to try and accurately weight publications based on their integrity, the world would most definitely be a better place. And it is not far-fetched to imagine an industry-standard Integrity Scale or Survey that could be used to determine the Integrity Index for different stakeholder groups.

Cubic Applications Earns Platinum Research Award For Media Analysis

We are proud to note that Mike Daily (a recipient of The Measurement Standard's coveted Measurement Maven of the Month Award) and the company he works for, Cubic Applications, Inc., has won a Platinum Award in the 2007 MarCom Creative Awards. Well done, Mike and Co.! Read more here.

Measure the Media that Are Most Important to Your Customers

The Paine of Measurement

 

 

Make Good Choices
No one says you have to measure every medium on the planet.

When my cousin Caleb was into his "terrible twos" and heading for trouble, his father, who stands an imposing 6 feet 3 inches tall, would look down at him and say in his deep baritone: "CALEB, MAKE GOOD CHOICES." Invariably, Caleb would stop what he was doing, sit down and think for a minute, and then modify his behavior accordingly.

The lesson seems to have taken. Caleb is now taller than his father, and a true teenage heartthrob. With very few exceptions, he makes great choices.

Oh, that PR people had such good survival skills. Sadly, they tend to keep doing the bad stuff rather than embracing the good. Four years ago, a member of the audience at a conference on new media raised his hand and said, "l don't have time to deal with all this on-line stuff, I'm too busy as it is."

I wonder how many PR people in the 30's and 40's said the same thing about radio and television. "Woe is me, one more thing I have to learn how to do." Rather than embrace the new, PR people tend to look at new media as yet another burden to be borne. And once they start measuring and tracking one media, they find it extraordinarily hard to embrace a new one. In the last month, at conference after conference, people have whined about how hard it is to deal with social media, and why they don't have the time or resources to measure it.

My answer is: Make Good Choices. No one says you have to measure every medium on the planet. Just because it's new, it doesn't mean you have to monitor it. You can and should choose to concentrate on, and measure, the media that are most important to your customers. If they're on Facebook and YouTube, you better be monitoring them. If they're sharing ideas on Twitter and photos on Flickr, it would behoove you to sign up and listen in. On the other hand, if they're still reading hard copies of trade magazines, then that may be all you need to monitor.

Better still, measure the impact on the customer directly. As Jim Macnamara writes, the average person sees, consumes and creates dozens of different media every day. If you have an unlimited budget, you measure all of those different media. But it's so much simpler to just measure the behavior of the consumer instead. Ask them how they perceive their relationship with your organization. Follow their behavior on your website. Watch and see how engaged they are with your blog. That's all the measure you need.

As long as PR people obsess about tracking column inches and "hits," of course they are going to be overwhelmed with the possibilities and the burden of measuring them all. But if they focus on the outcome – the actual customer behavior – chances are they'll survive and thrive in this new hyper-media world.

Public Relations Measurement Technique Sheds Light on Companies' Environmental Reputations

Jim Macnamara's 'Measuring Up'

 

Media Analysis Shows Companies Need to Set
and Achieve Measurable
Climate Change Goals
Environmental performance will be a key driver of reputation.

A major new study of public debate and discussion on energy completed by my company, Media Monitors – CARMA Asia Pacific, presents some important warnings and advice for companies, organizations and governments and concludes that environmental performance is likely to increasingly be a key driver of reputation. We used content analysis to examine statements and commentary by organizations, spokespersons and consumers in leading mainstream media and blogs in Australia and major Asian countries including China, Hong Kong and Singapore.

The analysis covered discussions of nuclear, coal, wind, solar and other alternative sources appearing in more than 1,500 print news articles, radio and TV program segments and 170 blogs appearing between May and July 2007. It focused on Asia Pacific because China is fast becoming the world's leading polluter. Australia is the world's largest exporter of coal and uranium, and 16 of 29 nuclear power plants under construction or proposed worldwide are due to come into operation in Asia in the next 10-15 years.

Overall Results:
Companies must achieve measurable targets on the environment or their reputations will suffer.

The report found mild optimism that solutions will be found to balance environmental and economic interests, although serious fears and concerns also abound in relation to safety, the environment and consumer protection.

Specifically of interest to companies and organizations is a finding that they face widespread criticism if they do not implement meaningful measures to reduce carbon emissions. Environmental performance is likely to increasingly be a key driver of reputation, the report says.

The report concluded that companies adopting policies and planning measures to address climate change need to set and achieve significant measurable targets. "Broad unspecific policies and 'aspirational goals' are not going to cut it and may backfire leading to public criticism. There are signs of this already happening," the report warns.

Also:

• Positive messages outweighed negative messages about energy production and use – albeit the margin was narrow. (The analysis examined discussion and comment from the perspective of a balanced position between protecting the environment and maintaining economic development and standards of living.)
• Nuclear energy is the most discussed form of energy and is mired in controversy and near evenly divided opinion for and against, the analysis found. Most opposition and concerns in relation to nuclear power are about safety.
• Solar and wind power are being actively promoted by environmentalists, but are described in most discussions as not being able to make a substantial contribution to energy needs.

Carbon Trading:
Despite a number of initiatives, governments are generally not seen to be doing enough.

Carbon trading is one initiative that is being welcomed by a majority of spokespersons and commentators and is set to become a multi-billion dollar industry over the next few years. However, some are warning that it is still uncertain whether carbon trading will stimulate a net reduction in carbon emissions. Some energy producers and consumer groups have pointed to lessons in Europe where too many permits were issued, resulting in a price collapse below levels necessary to stimulate investment in carbon reducing initiatives.

Also, concerns are being expressed that carbon trading is insufficiently regulated and could lead to scams and fraudulent schemes. One leading environment group has warned that some companies are selling trees without accreditation and that their claims of carbon credits or neutrality are not substantiated. Consumer groups are also expressing concern that consumers may be paying levies and surcharges with no guarantee that their money will be used to address climate change.

The report says:

"While the efforts of some companies to reduce their carbon footprint may be commendable and genuine, there is considerable concern over a lack of safeguards for consumers on how extra charges will be used and whether they will reduce carbon emissions... With carbon trading set to become the next big thing in the financial world, strict regulation of the carbon trading industry, including accreditation and monitoring of carbon offset providers, is seen as necessary to avoid scams and corruption... and to ensure that offsets are real and effective."

Clean coal technology (CCT) is also creating some optimism that a balance can be achieved between reducing greenhouse gases and continuing to exploit vast coal reserves. However, environmental groups and some commentators are cynical, pointing out that the technology is still unproven.

Blog Analysis

Blogs were found to be more unfavorable than mainstream media coverage (48% unfavorable compared with 30% of mainstream media content unfavorable). However, 30% of blog discussion was favorable. While much of this was supportive of renewable forms of energy, some bloggers support clean coal technology and carbon trading – albeit with the same concerns as mainstream media commentators and spokespersons.

Analysis also showed that bloggers often include scientists and technical experts and, therefore, blogs need to be recognized as an increasingly important medium, reflecting the viewpoints of influencers and thought-leaders.

Dr. Jim Macnamara BA, MA, PhD is Group Research Director with Media Monitors – CARMA Asia Pacific which conducts media monitoring, media analysis, e-surveys, reputation research and sponsorship analysis. He has a 30-year career in the media and PR in Asia Pacific and is the author of 11 books on media and communication. In August 2007 he was appointed Professor of Public Communication at the University of Technology, Sydney.

When Is An Impression Not An Impression: How Does Public Relations Measurement Take News Timing Into Account?

The other day I was re-reading Ed Moed's blog post about how much media coverage depends on what other events happen to be happening at the same time. His point, in part, was that events get pushed out of the media by other events, and what we think of as news actually depends greatly on what other news is also being covered.

And this got me to thinking about how public relations measurement typically measures media outputs with media content analysis, but -- to my knowledge -- does not typically take the news environment into account.

My point is that Article X might result in ten zillion impressions, but if it was published on a day with really big other news (an extreme example would be 9/11/01) then those impressions just aren't the same as they'd be on a slow news day.

We all recognize that the news hole shrinks when something else big is hogging all the coverage. But what we're talking about here is not whether or not there's room for an article to get published, but whether or not there is room in the reader's head for another article. Or conversely, if there is no other news in a reader's head, does an article somehow gain extra importance?

I asked Katie Paine about this and she said she recognized the phenomenon from experience. As to how to deal with it, measurement-wise, she emphasized the importance of human experience behind data analysis: "That's actually why [KDPaine & Partners'] human intervention and hybrid approach is so important. The automated systems just give you numbers, we give you an explanation."

Does anyone out there adjust or modify their media analysis results to take into account the news environment? (I mean formally, with some kind of multiplier or variable that corrects raw impressions data for the impact of other news activity on the reader.)

Some might argue that correcting for the news environment doesn't really belong in media analysis, that it is part of some later stage of public relations' impact on the public's mind. If we put that in terms of outputs, outtakes, and outcomes, then what we are talking about here happens somewhere between outputs and outtakes: Between the outputs and outtakes lies the news environment, which may enhance or inhibit the results of impressions.

To my knowledge, measurement does not formally take this "news environment" phenomenon into account. How does the news environment affect PR's effectiveness? How does PR affect a person differently according to what other news they are experiencing?

In "Measurement's Empty Head: Measurement ignores the most complex part of PR" I wrote something that's pertinent here:

Contemporary PR measurement treats the mind of the media consumer like it's some sort of an empty box to be filled: The media dumps in impressions and we measure the outtakes and outcomes. But human beings aren't just empty heads, and they don't consume media in a simple and rational fashion... Measuring impressions is a whole lot easier than measuring what happens to those impressions once they're inside a person's head... What happens there is probably the most complex part of how PR happens, yet state-of-the-art PR measurement doesn't take it into account.

--Bill Paarlberg

A Public Relations Measurement Index Number Based on Ideal Coverage

The Paine of Measurement

Optimum Content Score: An Index Number to Love?
Here's a quick and easy way to design your customized article content score.

For years I've ranted about why a single Index number or a standard PR metric won't work (see "The Problem of PR Indexes: Magic Number or Big Headache?"). My argument has been that each PR program is different, with different goals, different audiences, and different measures of success. So how can you possibly compare them all with one number?

Embrace the Differences

However, I am beginning to believe that there might be a solution to this problem, and that is to embrace the differences--to build the differences into the measure. Here at KDPaine & Partners we now offer our clients a measure that is tailored to match their individual program goals: the Optimum Content Score (OCS). It gauges the success of articles, based on what the client considers to be the ideal article. With the OCS, we distill a potentially large number of variables into one number, based on the relative concept of “ideal" coverage. That way, we can tell you whether your coverage was more ideal than last year (comparing to benchmark), or less ideal than you wanted it to be (comparing to objectives).

What Is the Ideal Article?

For any particular client, we define an ideal article based on their coverage goals at the present time. So, for instance, an ideal article for Client X might be one that:

• contains several key messages,
• leaves a reader more likely to purchase,
• mentions the brand in the headline, and
• appears in the Wall Street Journal.

If an article includes all these attributes, we give it the maximum score, ten out of ten. To determine the score for any given article, points are deducted from the maximum based on how far an article is from ideal, taking points off, for instance, for negative positioning and minor mentions.

The beauty of this system is that the company that is looking to keep out of the headlines can gauge its effectiveness as well as a company that is trying to get into the headlines. For instance, in the middle of a crisis, ideal press might be a minor mention that contains a key message and is balanced.

I realize we're still measuring outputs here (and our long term goal is to measure outcomes), but this sort of customized-to-goals score is a big improvement over imposing one score across a wide variety of organizations and industries that may not have a lot in common. See Ed Moed's "Measuring Up" blog post on the subject here.

OCS allows you to look at those things that are important to your own business, insuring that you are measuring the most relevant elements of what you do. For instance, Sabrina Steele at Raytheon has used it to make better decisions on supporting trade and air shows, using spokespeople, and comparing her effectiveness with her peers.

Another advantage of OCS is that it can be compared to other marketing data to determine what does, in fact, drive outcomes like Web traffic, product preference and consideration. Factor analysis using OCS can tell you exactly which program elements and media efforts are having the most impact on whatever are your ultimate marketing measures of success (typically sales, sales leads or market share).

Design Your Own Article Content Score

You can easily get started with your own version of an optimum content score. Below is a sample list of questions--and you may wish to remove some or add your own--that you can use to determine the characteristics of the ideal article for your own program. Then rate your articles on how closely they come to your ideal.

Wishing you large measures of success,

Sample questions to determine ideal article content:

1. What is the goal of the program? (pick only one):

a. Increase exposure

b. Keep bad news to a minimum

c. Disseminate key messages

d. Increase preference

e. Generate awareness for spokespeople

2. How important is brand or sub-brand visibility?

a. Extremely important

b. Somewhat important

c. Not important at all

3. How important is it that the story be exclusively about your organization or brand?

a. Extremely important

b. Somewhat important

c. Not important at all

4. How visible do you want your spokespeople to be?

a. Highly visible

b. Somewhat visible

c. Doesn't matter

d. Invisible

5. How important is competitive positioning?

a. Extremely important

b. Somewhat important

c. Not important at all

6. How important is the tone of the coverage?

a. Extremely important

b. Somewhat important

c. Not important at all

 

 

AVE Gets A Makeover, But It's Still Just Tarted-Up Output Data

We were thrilled to see Erica Iacono's article on AVEs in PRWeek. Ms. Iacono has done her homework carefully and provides a nicely balanced and up-to-date summary of the AVE controversy. And now some of the other blogs are weighing in on the matter, including PRomulgator (who provides a copy of the article).

(As regular Measurement Standard readers know (see our newsletter here), we have not looked favorably on AVEs. Katie Delahaye Paine, TMS's publisher and probably the most prolific writer in the world of PR measurement, has written plenty about their evils, as have others, including Jim Macnamara, and Bruce Jeffries-Fox.)

The gist of new developments is that if you tidy up your media analysis data, "taking into account a media placement's reach, type and prominence of the story, and positioning," then use a conservative ad rate that reflects rates actually paid by advertisers, then what you come out with is a number that is somewhat more accurate and sports a sheen of respectability.

Our take:

1.) AVEs are popular because they appear to be an easy and cheap way to measure outcomes.
A number with a dollar sign in front of it allows somebody somewhere to believe that they can compare PR's results with those of some other marketing effort. They think they are comparing outcomes, but they are only looking at tarted-up output data. There are ways to make this sort of comparison, but for now they are usually much more complicated and expensive. That won't be the case forever. When tools come along that allow quick and easy direct measurement of the effects of any communications or marketing effort on sales (or whatever your preferred outcome), who's gonna care about AVEs?

2.) On the other hand, you can't argue with success.
Angela Jeffrey is one person who has found some darn good results correlating a type of AVE with business outcomes. Hey, if it works, it works. We don't care if your regression equation includes eye of newt and toe of frog: If it works, consistently, then it could be a valuable measurement tool. Here's the text of an article by Paul-Mark Rendon from Marketing Magazine with more on how AVEs can be useful.

3.) Why should PR waste its time using AVEs to mud-wrestle with advertising, when it could be sitting at the boardroom table doing some clean and serious strategizing?
Jim Grunig is the most prominent proponent of the view that PR should set its sights higher than just being a tool to produce sales. PR can provide extremely valuable services that are qualitatively different than those of other marketing tools, but unless we focus on measuring those, we'll never achieve our true potential. From this point of view, AVEs might very well be a costly distraction.

--Bill Paarlberg, editor