The Measurement Standard: Blog Edition

The Media Integrity Index

A call for weighting media coverage based on trust and credibility.

by Katie Delahaye Paine

In this article I suggest that media analysis programs would more accurately reflect articles' influence on readers if the integrity of the media outlet was taken into account. This could be achieved by developing a standard measure of media integrity which would used to rate different media outlets for different shareholder groups. The resulting Integrity Index would be used to weight coverage, thereby achieving a more accurate measure of media coverage's impact on consumers.

To understand why an Integrity Index is needed, consider:

Item #1
Why measure what no one believes?

Mazen Nawahi, in an impassioned speech at the 2008 Dubai Measurement Summit, raised the issue of journalistic integrity and the degree to which the integrity of a particular media outlet should be accounted for in any measurement program. Put another way, if:

1. Everyone knows that a given media outlet is going to print whatever a company sends it because they always do or because they're a major advertiser, and
2. Everyone also knows that they will never print stories that don't agree with the governing dogma,

then why would anyone believe anything that was printed?

And if all your media coverage in a particular outlet lacks credibility, why would you include it in your measurement results? For example, in 22 years in business I have yet to have a client want me to include The Weekly World News in its media list. So how do people justify measurement programs that include stories that do nothing to achieve their goals?

Item #2
Credibility is in the eye of the beholder.

Suppose a highly credible blogger gets into a mud-wrestling contest with the blogging equivalent of a pig. The contest attracts a lot of attention, a lot of arguments fly back and forth, and a lot of dirt gets thrown around. Now for the people that are into farm animals, the arguments of the pig might be very credible. However, people interested in learning something of value professionally will no doubt pay more attention to the words of the credible blogger. But the real question is: Do you weight coverage of them both equally in your media analysis reports?

How to correct for influence?

Taking examples of this sort into consideration, it is easy to see why some sort of authority or influence weighting of coverage would be of value. There has been a fair amount of research done in this area. For several years, Angie Jeffrey at VMS has been studying various weightings of media coverage to determine which factors most directly impact sales. Her research first found that share of discussion was effective, but she postulates that advertising rates are more telling because they reflect the importance of the various media. (See this paper and this paper.) I think it's a step in the right direction, but I generally frown on the use of advertising rates (and of AVEs), so I'm skeptical.

The Media Integrity Index: Why not rate coverage on integrity as rated by stakeholders?

I suggest that it's not the ad rate but the integrity of the publication that most determines consumer credibility and thus drives consumer action. And I'm suggesting that to rank a media outlet in importance, we should ask our stakeholders how they perceive the media outlet in terms of integrity.

I raised the question on Twitter and got some interesting responses:

• "It's all relative... What is good for one is bad for the other. It has to be specified for a purpose, customized."

• "Integrity or perceived integrity? Seems there's a difference there that's highly subjective." --Ryan Anderson

• "Integrity is the measure of perceived relationship (believability) of one-to-many listeners. So measure relationships. I would suggest looking at the way political integrity is measured. It would be a perceived metric, measured by polling audience. --Videodred

• "What's the goal? What's the client want it to say? In the case of journalists, what's the relationship?

• "Integrity has to be measured parallel to influence as audience determines credibility of source." --Mike Maney

So in one way, my proposed Integrity Index has to start with the goal of the coverage. What is it that the company or organization is trying to accomplish? Consider it this way: If you don't care who or what your brand is associated with, and you want exposure pure and simple, then you don't care about the integrity or credibility of whatever media outlet is talking about you. On the other hand, if you're trying to establish a reputation, or build brand loyalty and trust, or trying to reach an audience with certain media preferences, then the credibility of media to your audience should matter a great deal.

How to determine integrity?

So how do we determine which media is trustworthy and credible, and which journalists have or do not have integrity? The wrong way is to look at a list of media and assign weights or values to each one yourself. What you think doesn't matter. All that really matters is what your customers or stakeholders or members or constituents think. So you need to ask them. The best way to do that is to use the Grunig Relationship Instrument.

Now you probably don't need to ask them about all 150 publications on your media list. Remember: Never ask a question about something that can't be changed. If The New York Times is what your boss' boss reads every morning, there's no chance in hell you'll ever take it off you top tier media list. Even if your target audience is nine-year-old girls. So start with a list of suspect publications, blogs or any other type of media outlet.

Remember that different stakeholder groups may rate a given media outlet differently. As I pointed out above, integrity is not a fixed standard. For instance, we would expect Republican viewers to rate Fox News and The New York Times differently than Democratic viewers. It is conceivable that you will want to generate different Integrity Indexes for different stakeholder groups.

Below are some survey questions adapted from the Grunig Relationship Instrument. You may wish to develop your own, based on the specific components of relationships you wish to measure (refer to this paper). Ask your stakeholders whether they agree or disagree with each statement as it pertains to each media outlet, then use the responses to rate the outlets.

1. This media outlet treats people like me and organizations like mine fairly and justly.

2. Whenever this media outlet makes an important decision, I know it will be concerned about people like me.

3. This media outlet can be relied on to keep its promises.

4. I believe that this media outlet takes the opinions of people like me into account when making decisions.

5. I feel very confident about this media outlet's skills.

6. This media outlet has the ability to accomplish what it says it will do.

7. Sound principles seem to guide this media outlet's behavior.

8. This media outlet does not mislead people like me.

9. I think it is important to watch this media outlet closely.

10. This media outlet is known to be successful at the things it tries to do.

Industry-wide integrity standards?

Now, the logical question is, "Why aren't we doing this as an industry?" Shouldn't we be factoring in credibility based on some industry-wide standard? It certainly is a more accurate weighting factor than simple eyeballs or ad cost. But the reality is that your stakeholders aren't going to be identical to my stakeholders, and what matters is how a significantly valid sample of your stakeholders feel. It would be awfully complex and difficult to set industry-wide standards for many different stakeholder groups.

On the other hand, if the PR industry wanted to take on a project to try and accurately weight publications based on their integrity, the world would most definitely be a better place. And it is not far-fetched to imagine an industry-standard Integrity Scale or Survey that could be used to determine the Integrity Index for different stakeholder groups.

A Public Relations Measurement Index Number Based on Ideal Coverage

The Paine of Measurement

Optimum Content Score: An Index Number to Love?
Here's a quick and easy way to design your customized article content score.

For years I've ranted about why a single Index number or a standard PR metric won't work (see "The Problem of PR Indexes: Magic Number or Big Headache?"). My argument has been that each PR program is different, with different goals, different audiences, and different measures of success. So how can you possibly compare them all with one number?

Embrace the Differences

However, I am beginning to believe that there might be a solution to this problem, and that is to embrace the differences--to build the differences into the measure. Here at KDPaine & Partners we now offer our clients a measure that is tailored to match their individual program goals: the Optimum Content Score (OCS). It gauges the success of articles, based on what the client considers to be the ideal article. With the OCS, we distill a potentially large number of variables into one number, based on the relative concept of “ideal" coverage. That way, we can tell you whether your coverage was more ideal than last year (comparing to benchmark), or less ideal than you wanted it to be (comparing to objectives).

What Is the Ideal Article?

For any particular client, we define an ideal article based on their coverage goals at the present time. So, for instance, an ideal article for Client X might be one that:

• contains several key messages,
• leaves a reader more likely to purchase,
• mentions the brand in the headline, and
• appears in the Wall Street Journal.

If an article includes all these attributes, we give it the maximum score, ten out of ten. To determine the score for any given article, points are deducted from the maximum based on how far an article is from ideal, taking points off, for instance, for negative positioning and minor mentions.

The beauty of this system is that the company that is looking to keep out of the headlines can gauge its effectiveness as well as a company that is trying to get into the headlines. For instance, in the middle of a crisis, ideal press might be a minor mention that contains a key message and is balanced.

I realize we're still measuring outputs here (and our long term goal is to measure outcomes), but this sort of customized-to-goals score is a big improvement over imposing one score across a wide variety of organizations and industries that may not have a lot in common. See Ed Moed's "Measuring Up" blog post on the subject here.

OCS allows you to look at those things that are important to your own business, insuring that you are measuring the most relevant elements of what you do. For instance, Sabrina Steele at Raytheon has used it to make better decisions on supporting trade and air shows, using spokespeople, and comparing her effectiveness with her peers.

Another advantage of OCS is that it can be compared to other marketing data to determine what does, in fact, drive outcomes like Web traffic, product preference and consideration. Factor analysis using OCS can tell you exactly which program elements and media efforts are having the most impact on whatever are your ultimate marketing measures of success (typically sales, sales leads or market share).

Design Your Own Article Content Score

You can easily get started with your own version of an optimum content score. Below is a sample list of questions--and you may wish to remove some or add your own--that you can use to determine the characteristics of the ideal article for your own program. Then rate your articles on how closely they come to your ideal.

Wishing you large measures of success,

Sample questions to determine ideal article content:

1. What is the goal of the program? (pick only one):

a. Increase exposure

b. Keep bad news to a minimum

c. Disseminate key messages

d. Increase preference

e. Generate awareness for spokespeople

2. How important is brand or sub-brand visibility?

a. Extremely important

b. Somewhat important

c. Not important at all

3. How important is it that the story be exclusively about your organization or brand?

a. Extremely important

b. Somewhat important

c. Not important at all

4. How visible do you want your spokespeople to be?

a. Highly visible

b. Somewhat visible

c. Doesn't matter

d. Invisible

5. How important is competitive positioning?

a. Extremely important

b. Somewhat important

c. Not important at all

6. How important is the tone of the coverage?

a. Extremely important

b. Somewhat important

c. Not important at all