The Measurement Standard: Blog Edition

Xigaware's PR Measurement Interview with Katie Delahaye Paine

In case you missed it, here's Xigaware's interview with KDPaine & Partners' CEO Katie Delahaye Paine that addresses some fundamental measurement questions, including:

-- What are the newest social media measurement tools?

-- What are some of the biggest mistakes you see companies making when it comes to digital PR and marketing?

-- What are three tangible social media measurement methods I can use today , show my boss tomorrow, and use in decision-making?

-- Can we track social media hits on Twitter and Facebook the same as any other hit in print or on radio/TV?

Easy Strategies to Get Started Measuring Your Public Relations

 

Quick and simple techniques to start measuring your public relations programs and impress your boss.

by Katie Delahaye Paine and Bill Paarlberg

If you're reading this and you're not actually measuring your public relations programs yet, then the only New Year's resolution that really counts is for you to get started measuring. Which, as it turns out, is going to be a whole lot easier than those other resolutions you made a couple weeks ago and probably flunked out of already.

Measurement doesn't have to be a big deal. We've rounded up plenty of ways you can get started without spending much time or money at all. In fact, for several of our tips below, the service is free and the data gets delivered right to your in-box. Of course, if you want to get beyond the newbie stuff you might have to click around the Web some, do some reading, maybe break a sweat thinking about what your data means. (Relax -- that's a different article.)

Here is a baker's dozen of quick and painless techniques to get something started.

1. Get with Google:

• Sign up for Google Alerts for your organization or brand, and for at least three of your competitors or peers. Then tell your boss you have installed a crisis alert monitoring system.
• Put Google Analytics on every website that you control. (Yeah, the instructions need a serious usability makeover, but once you get the code, you just paste it into all your pages. Cake.) Then tell your boss you don't need no stinking WebTrends. (Actually, you probably do need WebTrends if you have to do advanced analyses -- it's much more robust and customizable -- but we're talking quick and easy here.)

2. Time for Twitter:

• Go ahead and get yourself on Twitter. See what it's all about, in 140 characters or less.
• And even if you don't want to start Tweeting yourself, you can still set up an RSS feed for a Twitter Search on your brand/s, your competition, and anything else you want to keep track of. Tell your boss you've implemented a Twitter monitoring system.
• And for you heavy Twitter users, see how influential you -- or your boss -- is on Twitter: Go to www.twinfluence.com.

3. Educate yourself:

• Stay on top of social media every day, with a click or two: socialmediabiz.com will send you daily email updates.
• Get to know the Institute for Public Relations website, especially the research and papers on measurement.
• Check out any of the numerous YouTube videos that talk about how to measure public relations and Social Media, (especially, in all modesty, the ones that feature our Ms. Paine).
• Read Katie Delahaye Paine's book How to Measure Public Relationships. OK, it's not free, but measurement gurus got to make a living. And it's the best $29.95 you'll ever spend on measurement. Click here to buy it now.

4. New media meets old school:

• Set up a free Survey Monkey account and start surveying your constituencies.
• Back to basics: Use Survey Monkey to measure your relationships. Send out a survey to assess the level of trust, commitment and satisfaction they feel about your brand.

5. More easy online stats:

• Get yourself a Facebook account and, at least once a week, conduct searches on your brand and your competition. Then tell your boss you are monitoring Facebook. (For more on measuring Facebook, see "Measuring Community on Social Networking Sites: A detailed how-to guide for measuring MySpace and Facebook.")
• Go to www.xinureturns.com and see how your URLs stack up.
• Go to www.compete.com and see how you stack up against the competition.
• Go to kineda.com and use their widget to see how your "bloglebrity status" ranks.

6. All together now:

• Now put together your own dashboard. Whatever metrics from above you actually have, put them in an Excel spread sheet and set up Key Performance Indicators. Repeat monthly. Progress confidently toward success.

Research Bias: Another example of why you must consider the source of research.

Biased research results can creep into even the most carefully designed and executed studies. That's why medical research is often double-blind, so, hopefully, the experimenter's bias can't affect the results. In public relations research, bias is endemic because most every measurement project may affect someone's job or budget, and every measurement vendor wants their results to justify their fees.

So here's a review of cell-phone health research to remind all of us just how bias can affect results ("Source of Funding and Results of Studies of Health Effects of Mobile Phone Use: Systematic Review of Experimental Studies" by Anke Huss, Matthias Egger, Kerstin Hug, Karin Huwiler-Müntener, and Martin Röösli.) (Or see a summary here.) It analyzed research on the biologic effects of cell phone use, and found that industry-funded studies were far less likely to identify negative consequences than studies funded by governments and non-profits. Researchers analyzed 57 studies that appeared in the academic literature between 1995 and 2005. Only a third of the industry-funded studies identified a biologic effect with possible health consequences from exposure to cell phone radio waves, while 82% of the government- or non-profits-funded studies found such effects, as did 77% of the studies whose funding source was not identified.

How can public relations measurement identify and reduce research bias? --WTP

Taking Public Relations Measurement to the Next Step

Measurement Strategy

 

Examine the events, actions, strategies and tactics that drive coverage and conversations.

by Katie Delahaye Paine

Having been at this measurement stuff for some 22 years now, I find it remarkably refreshing and rewarding that more and more communications managers are actually measuring results, as opposed to just counting clips. Yes, the vast majority of folks out there are still just counting, but more and more I'm hearing people talk about taking measurement to the next level.

For me, that next level is deriving more meaning from the data. In other words, it's not enough to just look at a trend chart and say, "There's a big spike in June." You need to look at the variables that caused that big spike, or that big slump.

The variable might be the activities that a PR department or agency has undertaken, or it might be the actions or strategies of the competition, or it might be external news events that take over the news hole or the conversation. The point is that for true measurement to occur, you need to be taking these things into account.

That doesn't mean obsess over them so much you don't act. Too often I've had clients get into "Variable Paralysis," a condition in which they are so concerned with accounting for every possible variable, that they don't measure anything.

How to Pick the Right Variables

The one things we at KDP&P have learned is that variables can be specific to the market space or industry you are in. For example:

• In the non-profit sector, the ultimate goal for communications is typically to either generate donations, recruit volunteers, build membership, or spread a message. The variables we track are typically events, -- major high visibility programs or initiatives that drive media coverage and ultimately drive Web traffic and membership or donations. What's important to remember about using events as a variable is that the competition is also doing similar events and programs, and you need to account for them as well.
• In the business-to-business space, the end result may be to change positioning or reputation. Is perception changing over time? Are more of your messages getting out there? In which case the variables are more people- and product-based. What thought leaders are being quoted? What products or programs are being discussed? And are we getting into that conversation as often as the competition? So you're not just looking at the volume of coverage, but of the visibility of that coverage, how it positions your organization, and who is framing the discussion.
• In the educational marketplace variables tend to focus on news events. Whatever is hot in the traditional news cycle tends to drive actions in academia. So when a bridge collapses in Minnesota, engineering departments around the country are tapped for their expertise. Or if alternative energy becomes a major news focus, professors and universities who have published books or papers on the topic become widely quoted. So a key variable in academia is the degree to which proactive PR efforts in response to uncontrolled media events have impacted your visibility.

The point in all of this is that we all know how to monitor and count our coverage. Many of you also know how to count and monitor conversations going on amongst your customers. But now we have to look at the events, actions, strategies and tactics that are driving those conversations.

Five Tips for Integrating Public Relations Measurement of Social Media and Traditional Media

Integrated Public Relations Measurement

Your customers read all sorts of media, and so should you.

by Katie Delahaye Paine

For years we at KDPaine & Partners have had a pretty big wall between our traditional MSM (main stream media) analysis projects and our social media or CGM (Consumer Generated Media) projects. However, we've recently had a revelation: Our client's customers consume both types of media to form their opinions, so why shouldn't we measure both types of media and integrate our results?

We know that consumers are increasingly getting their information from online sources and chances are, they don't spend a lot of time worrying about whether the source is "main stream" or "social" or whether its coming from an RSS feed or an email or a YouTube video or Twitter. They seek out the news source, and they read it.

Customers Read Everything...
Geepers, Maybe We Should Too!

So we revamped our approach, and, boy, did we learn a lot. For starters, it's amazing how conversations taking place between customers differ from the articles in traditional MSM. Sometimes they're more positive, and sometimes they're more negative, but the big Aha! was that they focused on different things.

For example, in one recent discussion of a merger between two high-tech companies, the traditional and trade media focused on the financial and organizational implications, but in CGM the customers were furious that the new owners were now going to charge for an open source application that used to be free. They even dubbed it "closed source," a term that soon showed up in traditional media coverage. The disparity in coverage offered our client a large opportunity to stress their positioning in all their exposure, which was that their "open source" product would remain truly open source.

Further, we had noticed that the traditional media wasn't picking up on customer stories from our client, favoring the competition instead. However, in CGM, there were a number of happy customer stories that offered great fodder for future applications stories. Had we not compared the different forms of media, we would not have identified the difference, and might well have missed the opportunity.

How can you integrate measurement of CGM and MGM to improve your measurement? Here are five tips to get you started. Let me know how you do and if you come up with more ideas. (And, [shameless self-promotion ahead] if you decide you need help integrating the measurement of traditional and social media, give Mike Layton at KDPaine & Partners a call at 603-696-6098.)

Five Tips for Integrating
CGM and MSM Measurement

1. Look for obvious differences in tone.

• a. Notice the negative, and heed the warning.
  If CGM is 30% negative, and traditional and/or trade media is 3% negative, that sends up an immediate red flag. If customers are saying more negative things than the media, it's just a matter of time before the media starts to hear those customer complaints. Count yourself lucky you have some warning and do your best to prepare to weather the storm.
• b. Accentuate the positive.
  On the other hand, if CGM is much more positive than traditional media outlets, you could have a big win on your hands. There's nothing better than third party or customer endorsements. So if the users out there are very happy, dig into those stories to find opportunities for good application or research stories.

2. Compare your share.
Compare your share of conversation or coverage between CGM and MSM, and if there is a big difference, find out why. Suppose, for instance, you're winning (compared to your competition) in CGM, but losing in MSM. So look at what topics the customers are focusing on in their conversations. Those are the "hot" relevant topics in your marketplace. Now perhaps you will want to refocus your traditional MSM pitches around what is already being discussed, rather than trying to change or force the conversation.

3. Find out what the market cares about.
Traditional media is far more influenced by corporate PR than is CGM. CGM more strongly reflects the actual consumer marketplace, and CGM conversations tend to deal with practical issues at hand -- like something that doesn't work, or finding a solution that works beautifully. These issues could be the key to what sells --or doesn't sell -- your products. Get into the conversation and you'll find you'll learn a lot more. Take a tip from MSM reporters, who are increasingly searching CGM for stories and angles.

4. Look at who's doing the talking.
Are the same spokespeople/influencers or thought leaders showing up in both places? Probably not, and therein lies opportunity. You might, for instance, discover a whole new slate of influencers in social media that may well be more receptive to your outreach than your old MSM contacts. On the other hand, you also might find that they're talking about topics in which you are irrelevant, and therefore you should move on and find people who are talking about topics relevant to you.

5. Tie back to the Web.
Correlate activity in both forms of media back to Web traffic (leads, inquiries or revenue) and you'll have a much clearer picture of what drives customer action. For one of our clients, it turned out that social media was a much better source of online donations than traditional media. While traditional media focused on the sensational and celebrity aspects of a story, consumers, in this case pet owners, were using social media to discuss how to keep their pets healthy and/or how to rescue pets in trouble. These issues were much more likely to prompt donations than were the sensationalized aspects of the news covered by traditional media. Once our client realized this, it was able to refocus its messages and increase donations.

How to Measure YouTube and Other Video Sharing Sites

"Evolution of Dance" is the most watched YouTube video ever.

Set up a simple YouTube measurement program, decide if particular videos are really problems, and prepare for the worst case scenario.

by Peter Kowalski

Quick: How many people have watched the most popular YouTube video? Now, how many people watched the last Superbowl?

If you answered 87 million for the first and 98 million for the second, then you probably already know that video is the most popular type of shared content in Consumer Generate Media outlets. And with video technology becoming simpler to use and bandwidth growing, we can expect huge video sharing growth in the future. So, if you are not measuring YouTube now, then you will be sooner or later.

Actually, YouTube is only the most popular of at least 40 or so video-sharing websites. You will want to check most of these to see if the content is relevant to your organization or products. And maybe you will want to monitor some of them. As a practical matter, however, if you keep an eye on YouTube, you've pretty much got the field covered. Here's how to get started.

Welcome to YouTube

If you are absolutely new to YouTube, then first just go there and look around and watch some videos. Be amazed at a few of them. Be amazed at the inanity of many of them. Notice that some of the videos have been viewed 30 million times, and some have been viewed 30 times. There is even a movie being made about the YouTube video "Battle at Kruger."

Do a few searches. Do a basic search for your name and your teenage daughter's name and the name of your ex. Take a look at the Channels page, and at the TestTube page. And if you want to really get to know the place, create an account and upload your own video. Here's a hot tip: If you are going to upload that video of you doing the Wild Chicken Dance at the company party, don't put your own name on it.

Why Would You Want To Measure YouTube?

Now let's get a little more complicated. There are three basic reasons why you would want to measure YouTube:

1. You want to measure other peoples's videos about you, and need to decide if you should really care about them.
2. You are thinking of making a video, and you want to know best practices.
3. You already made a video and you want to know if it achieved your objectives.

We will leave #2 above to a separate article. But in the meantime there are some resources on the Web that might help. Just google "How to make a good YouTube video."

Measuring Other People's Videos About You, and Should You Care About Them?

Your Five Minute YouTube Measurement Program
First off, here is the quick and easy way to stay on top of YouTube. Do a search for your company and product names every day or every week. If you come up with nothing, great: You can tell your boss there are no videos posted, and you are monitoring YouTube. For each video you do find, decide: "Would we approve this video, or would we not approve this video?" And so put all the videos into those two files. Now you can report on YouTube activity whenever you need to.

(Note that these searches will pick up all text in the headlines, descriptions and user tags [it doesn't search the actual video itself]. Knowing user tag behavior for your organization is key on YouTube and elsewhere for this reason.)

When Should You Get Excited about a Video?
If you find an number of videos that mention or concern your organization or product, then you will want to do a more careful analysis. Do you appear in the title, description, tag, or just in a very quick glimpse? What are the positive points? What are the negative points?

Thumbs Up? The videos that are favorable can often teach you something about your products or image that you didn't know. Keep track of them, and see below for techniques to examine them more closely.

Thumbs Down? The only ones to worry about are the negative ones, and even then, don't go getting all excited until you examine the situation carefully. How many people are actually watching the negative video(s)? Did the video(s) get low ratings and bad comments, or good ratings and lots of links? Maybe a negative video actually proved strong brand loyalty when viewers gave it negative comments.

You can deal with the problem videos on a case-by-case basis, and/or compare them to see if there is a pattern or a larger problem that needs to be addressed.

How to Survive the Worst Case Scenario
So, it comes to pass that there is a video out there that says strongly negative things about your product or company and lots of people are watching it, commenting on it and passing it on. What to do? First, recognize that you have a full blown crisis on your hands; any video that gets really strong response indicates a major problem that needs serious attention. Respond to the video as you would respond to any crisis: Monitor the press, get your replies together and get your messages out. Details depend on the situation. Maybe you do a response video, maybe you do a press release. But the good news is you have plenty of opportunity to learn what people's problems are and, by analyzing comments, to learn specific ways to fix them.

Indepth YouTube Measurement: Basic Metrics

If you find a number of videos about your products or organization, you may decide you want to examine them more closely. If so, here are the basic metrics that you want to collect for YouTube or any video site:

Most important:

• How many times was a video watched?
• How many times was it rated and what were the ratings?
• How many comments did it get?

Also record:

• How many times has it been favorited?
• How many times has it been embedded?
• How many times has it been linked to?

See below for ideas on how to compare coverage to other media, and how to analyze comments. Be aware that it is often very complicated to understand the impact of a video that mentions your product or company. (For instance, KDPaine & Partners looks specifically at what characteristics of brand strength or organizational reputation are affected.)

So You Made a YouTube Video:
Did it Achieve Your Objectives?

The key here is that you want to be able to show that you communicated what you were supposed to. So, right from the start set your video design objectives so that the video expresses what your bosses want to communicate about the company or product. O.K., yes, it seems elementary, but if everyone agrees that your video says to the world what you (and your boss, board or board room) want to say about your organization, then you have a video you can measure. If there is an uncertainty about what your video should communicate, then all the metrics in the world are not going to tell you how effective it was.

Once your video is posted, you want to collect the basic metrics described above. Measure at standard times from posting: say, one hour, one day, a week, two weeks, a month, and then once a month thereafter. After six months your video has probably finished its YouTube lifecycle, but check at one year just to see.

Now compare your video's metrics to those of your competition, or to your videos in the past. Are you doing better or worse? Which stats are different and why? What do comments reveal about each video and what do they teach you about how to make your next video?

Comparisons to Other Media

Your YouTube metrics will be put in context if you can compare them to other media. So, for instance, you can provide some meaning by comparing number of views to newspaper circulation, or TV viewership. It's not exactly apples to apples -- you can make a case for YouTube views as being much more powerful than print or TV impressions, for instance -- but it is a place to start.

For the Advanced Student: Comment Analysis

If you have plenty of comments to analyze, then things get much more interesting. If a viewer leaves a comment, you can argue that the video had a greater impact on that viewer than did some other video on which the viewer was not moved to comment. So for two videos with equal viewership, the one with the most comments can be said to have the greatest impact.

You will find, however, that many comments are not relevant to your objectives or relate in any obvious way to your product or organization. So, analyze comments based on to what extent they achieve your original objectives. Also check to see if they indicate any other important sentiments or reactions that you did not anticipate:

• Positive or negative responses to your organization, brand or products
• Demonstrations of increased brand strength of loyalty
• Unusual engagement with product or company

More advanced analyses, like that those available from KDPaine & Partners, include teasing out exchanges of particular sorts, debates on particular topics, and threads on topics of choice.

How to Impress your Boss and Justify Your Existence, Video-wise

Here are two quick ways you can demonstrate that your video has been particularly effective:

1. Compare your video against other media (ads, newsletters, etc.) for transmission of key messages or brand reputation.

2. Use comment analysis to show that people are interacting with your brand(s) and are more engaged with your company and products. Remind your boss that non-social media -- like advertisements -- can't encourage or measure that sort of engagement at all.

Extra Credit: Do focus group or survey analysis of video viewers to determine if they, after watching the video, are more likely to buy your product or visit your theme park or whatever. If you survey all customers, ask "Do you watch YouTube videos, and did you see ours?"

Peter Kowalski is Director of Research Strategy at KDPaine & Partners. His research interests include international inter-media influence, agenda setting and network analysis. He holds a Bachelor of Arts in Public Relations from the USC Annenberg School of Journalism.

Measuring Naked Relationships: How to evaluate the impact of a social media program on your public relationships.

Your step-by-step guide to using relationship metrics to evaluate the success of your social media program.

by Katie Delahaye Paine

I've talked a lot here -- also in my speeches, in my blog, and of course in my recently published book Measuring Public Relationships -- about the importance of measuring relationships. I believe that without factoring in the impact that your social media program has on your relationships (with employees, community and constituencies) you are undervaluing your efforts.

So, how does one actually measure relationships? Well, now that you asked, my book explains this with regard to most types of public relations programs. Social media being all the rage right now, I thought it would be appropriate to provide a step-by-step guide as to how to actually do it for social media programs.

Whatever type of program you want to measure, the basic technique is similar: You conduct a survey of your audience using a special set of questions designed to specifically measure the different components of relationships. You do this before and after your social media program is in place, and you do it for your organization and as many competing organizations as you can afford to. Then you compare the data before and after, and between your organization and the others, and then you know where your relationship with your audience stands and where you need to go.

The Grunig/Hon Relationship Research

Before we get to the nuts and bolts, here's a bit of background. A decade or so ago, University of Maryland Professors Jim and Laurie Grunig and Linda Hon synthesized communications and sociology research and theory into a paper published by the Institute for PR called "Guidelines for Measuring Relationships in Public Relations." Their feeling was that, amidst all the brouhaha about quantifying public relations, we were forgetting the essential truth that PR was about relationships. And so you needed to measure the impact that your efforts were having on those relationships.

Their research isolated six fundamental components of relationships -- Trust, Satisfaction, Commitment, Control Mutuality and Exchange/Communal -- and they designed and tested 75 statements to measure those components. These statements are of the sort: "This organization treats people like me fairly and justly," and, "I feel a sense of loyalty to this organization." You can see them all, and copy them for your own use, on my blog or in the Grunig/Hon paper. For an example of these statements successfully used to measure public relationships in a non-social media context, see this research by Forrest W. Anderson and Paul Raab.

Nine Steps to Measure Your Social Media Relationships

Before we start, remember that to isolate the effect of your social media program, you must begin your measurement before you launch that program. Then you'll have a benchmark to compare against later: Before Social Media vs. After Social Media. Without a Before benchmark you won't know how your social media program has changed your relationships.

Of course, you can always begin measuring after your social media program has started, and by doing so you will be able to do ongoing evaluation of your relationships. Which is a good thing. But a better thing is to isolate the effect of social media, and to do that you must compare before and after.

So use Steps 1 through 9 below to survey your audience before your program begins, and thus establish your benchmark. Then begin your social media program, and after an appropriate time, say three or six months (depending on your situation), re-survey to see what has changed.

To be sure that whatever change you find is solely attributable to the social media program, you must hold constant any other PR programs that affect your social media audience's perception of your organization. Yes, it's tricky, and it's not always an ideal world. But if you are trying to measure the effect of your new blog at the same time as the Promotions Department decides to give away A Free Cadillac With Every Purchase, then you can kiss your results goodbye.

Step 1: Define the audience for your social media program.
Social media is about conversation and engagement, so decide with whom you want to converse and engage. If you're starting an internal blog, your audience is your employees. If you're starting an IdeaStorm-type customer community, your audience is anyone likely to buy or recommend your product. If your mission is advocacy, your audience might be voters, politicians or industry influencers.

Step 2: Get a list of your audience's email addresses and/or phone numbers.
To get a representative sample you will need at least 500 names for each organization you are testing (more on that later). If you already have a list of your members, subscribers or customers, then you are ahead of the game.

If you have to purchase your list, then potential vendors vary with the type of sample you're looking for (mail, phone, web). Most lists are sorted based on demographic or title data. There are a lot more resources out there for mail addresses and those resources do not necessarily need to be survey sample companies. For email addresses, some reputable firms include Survey Sampling, e-Rewards and Zoomerang.

Step 3: Pick a survey methodology.
The Grunigs would recommend in-person surveys for the best results, but most researchers find that to be very expensive. Phone surveys are fast and provide very accurate results, but again, depending on the audience, may be cost-prohibitive. Email surveys are an increasingly accepted methodology, and for social media can be appropriate and highly reliable, since, presumably, your audience is all on email.

You may be able to piggyback on an existing survey going out to your community. If marketing, customer satisfaction, business development or anyone else in your organization is doing a survey, see if you can add a few of the Grunig/Hon statements to it.

Step 4: Select which of the Grunig/Hon statements are most appropriate to your organization.
You can probably only impose on someone for 7-10 minutes of their time, so you need to pick which statements you will include. Grunig and Hon suggest that if you want to shorten the survey, you use only the boldfaced items. Not all statements are appropriate for all organizations, so pick and chose the ones that will be most meaningful to your audience.

Step 5: Prepare your survey.
If you are using an electronic survey system like Survey Monkey, you need to create an introductory screen that explains what you are doing and how the scoring works. For instance:

In order to better understand the needs and perceptions of our marketplace, we'd like to ask you some questions. Please tell us whether you agree or disagree with the following statements as they apply to X company/organization.

Explain that 1 means "totally disagree" and 7 means "completely agree," and give them an option for "no opinion." You also need to ask them the same questions about a competing company or organization, so you have comparable data on the competition to compare to.

Step 6: Send out the survey.

Step 7. Resend the survey.
Depending on your audience it may take several tries and an incentive to get sufficient responses (I'll do just about anything for an Amazon or Starbucks card). How many is sufficient? Well, it depends on how you plan to break down your analysis, but in general plan to resurvey until you get at least a 10% return. If in doubt, talk to your local survey expert.

Step 8. Analyze and learn from the results.
Calculate a mean score for each relationship component. There are both positive and negative statements in the survey instrument, so make sure you take that into account. Compare your mean on each score to the competition. (And of course to your earlier survey results, if this survey is not the benchmark.)

Step 9. Implement your program and measure again.
If this is your pre-social media program (benchmark) survey, then implement your program now, and measure again in six months. Or, if your program has been running for a while and your analysis indicates you need to make changes, then make the changes now and let them work on your audience for six months before you measure again.

Good luck, and let me know how things go.

How to Measure Relationships with Voters, Legislators and Other Political Constituencies

by Katie Delahaye Paine

Ah Springtime! When a young man's thoughts turn to love, or basketball, or, if you live in New Hampshire, politics. Given that New Hampshire has the third largest legislative body in the English speaking world (after the British Parliament and the US Congress), and that once a year in March a million or so average citizens get together in town meetings to determine the fate of the state, you can understand why the term "March Madness" takes on a whole new meeting up in these parts.

When measuring relationships with voters, as with any measurement program, the type of system you put in place depends on your objectives and your role. Below is a list of tools you need for each type of program. One important point first...

Timing is more critical than usual
Because the length and success of voter measurement programs are so strictly defined by the election cycle, you can't afford to waste time or opportunities. Probably the most important element in any political measurement program is to have the data on hand when you need it. And you need it whenever there's a decision to be made about how much money to spend or what tactics to use. You don't need it after the votes have been counted.

Measuring PR Efforts on Behalf of a Candidate

My company, KDPaine & Partners, measured political PR for one of the country's best political PR pros, Doug Hattaway, former spokesperson for Al Gore and Tom Daschle. Doug was working on Chellie Pingree's effort to unseat Susan Collins as Senator from Maine. What Doug needed was a measurement system that would tell him which tactics were working or not working in a campaign. The challenge was to do it fast enough so the data would be useful, and cheap enough so the campaign could afford it.

The good news about working with political campaigns is that volunteers are generally plentiful, and in this case the volunteers were charged with collecting the clips from the local publications. If you don't have volunteers to do the clipping, you need to hire a local clipping company or someone like Bacons or Burrelle's to collect the clips for you. Small-town newspapers are just too important to miss. Electronic data aggregators like Nexis and Factiva do not work on local or statewide campaigns; they do a terrible job collecting clips from anything smaller than the Boston Globe. Even Cyberalert and Custom Scoop miss far too many of the small town weeklies.

One we had the clips about both Pingree and Collins in hand, we began to analyze them for positioning on key issues, tone of coverage, type of article, subject of the article and who was quoted. We compiled them monthly and issued a report that included charts like this (click the charts to see them bigger):

 

 

 

 

We could quickly identify the tactics that were most successful. For example, a visit by Hilary Clinton in support of Pingree was far more successful in garnering visibility for Pingree than any other action. However, Pingree's visit to Washington in support of her health care reform legislation, while it didn't get anywhere near the visibility, was more successful in positioning her as "the health care candidate."

The analysis also revealed weaknesses in Pingree's campaign and strengths in Collins' campaign. While Pingree was almost entirely focused on her health care message, Collins was making points with the voters on the environment and campaign finance reform.

Additionally, the research enabled the campaign to track the correlation between visibility and contributions. By tracking Pingree's share of exposure over time against the contributions over time, the campaign could determine the level of additional exposure necessary to generate the requisite contributions.

In the end, the earned media that was generated by Hattaway's PR efforts narrowed incumbent Collins' lead substantially.

Measuring Lobbying Efforts

We ran into a government relations person for a major organization who wanted to know how to measure the impact of his efforts. He was trying to get a bill passed that would be highly advantageous to his company. We began to design his "dashboard," and asked him what success meant. He got as far as "If the bill passes, I'm successful." We pointed out that if the bill failed it would be too late to do anything about it.

So what he really needed was to monitor the tone of conversation around the bill in question so that he could tweak his PR program according to its progress. We recommended that he establish a system that gives him feedback all along the way.

There are several ways to do that. The easiest method is to track who is saying what to the media or in local speeches about the bill. Are key influentials supporting it or trashing it? Or is no one talking about it at all? More important to track, however, is the bill's progress through the legislative process. Is it being heard, tabled, or moved? Who are the cosigners to the bill and are they actively promoting it? Is the opposition gaining or weakening? What do bloggers say? What's being discussed in news groups? Are there any list serves you need to track?

Some of these data points can be gathered via media analysis, others should be part of your regular legislative tracking process. Still others may require a poll of the constituency to determine the level of support, and the perceptions of the voters and/or legislators.

If at all possible, a once-a-year survey of your key legislators is recommended to test and evaluate the health of the relationship between your organization and the elected or appointed officials you are striving to measure.

 

Measuring your Relationship with Publics, Both Active and In-Active

Your constituencies come in several forms, from students to senior citizens. You should measure relationships with as many different stakeholders or constituencies groups as you can afford. It may be useful to segment the different publics by gender, age, length of time in the area and political leanings so you can identify any pockets of opportunities or threat within the community.

Clearly active stakeholders who are most likely to protest, boycott or otherwise cause trouble are the most important to understand. However, inactive stakeholders that are clueless about your programs and therefore don't care and won't get engaged can be just as dangerous.

We recommend using the Grunig relationship model to test your relationships and in particular the level of trust that each public feels towards you. Trust is a key component of success in any political campaign. If it's not there, or if it is recently lost, you will have a significantly harder time achieving success. Alternatively, if your constituents trust you, as do those of Alan Greenspan or John McCain, you have a lot more leeway with those publics. Therefore it is critical to establish a benchmark level of trust to begin with. Subsequently you should conduct regular trust/relationship measurement studies to gauge the level of trust and engagement over time. Conduct these studies as often as possible so you can tie any changes in the trust and relationship scores to actual actions you may have taken.

The Ultimate Measure: Votes

There's a ton of great data in any set of election results. The challenge is to use it to make more informed decisions. When I ran for town council here in Durham five years ago, I knew that my efforts to go door-to-door in specific communities paid off since the number of votes I received from those communities far exceeded my expectations. I also knew that my signage (bright purple and white with the slogan "No Paine, No Gain") was successful since I had the highest name recognition, and ultimately the highest vote count of all the candidates.

In larger elections, there's far more granular data to be gathered, and much of it is readily available online. Using this data you can frequently identify the specific areas or pockets or demographics you've targeted and thus measure your success in getting those groups to vote and/or vote your program or candidate.

The Media Integrity Index

A call for weighting media coverage based on trust and credibility.

by Katie Delahaye Paine

In this article I suggest that media analysis programs would more accurately reflect articles' influence on readers if the integrity of the media outlet was taken into account. This could be achieved by developing a standard measure of media integrity which would used to rate different media outlets for different shareholder groups. The resulting Integrity Index would be used to weight coverage, thereby achieving a more accurate measure of media coverage's impact on consumers.

To understand why an Integrity Index is needed, consider:

Item #1
Why measure what no one believes?

Mazen Nawahi, in an impassioned speech at the 2008 Dubai Measurement Summit, raised the issue of journalistic integrity and the degree to which the integrity of a particular media outlet should be accounted for in any measurement program. Put another way, if:

1. Everyone knows that a given media outlet is going to print whatever a company sends it because they always do or because they're a major advertiser, and
2. Everyone also knows that they will never print stories that don't agree with the governing dogma,

then why would anyone believe anything that was printed?

And if all your media coverage in a particular outlet lacks credibility, why would you include it in your measurement results? For example, in 22 years in business I have yet to have a client want me to include The Weekly World News in its media list. So how do people justify measurement programs that include stories that do nothing to achieve their goals?

Item #2
Credibility is in the eye of the beholder.

Suppose a highly credible blogger gets into a mud-wrestling contest with the blogging equivalent of a pig. The contest attracts a lot of attention, a lot of arguments fly back and forth, and a lot of dirt gets thrown around. Now for the people that are into farm animals, the arguments of the pig might be very credible. However, people interested in learning something of value professionally will no doubt pay more attention to the words of the credible blogger. But the real question is: Do you weight coverage of them both equally in your media analysis reports?

How to correct for influence?

Taking examples of this sort into consideration, it is easy to see why some sort of authority or influence weighting of coverage would be of value. There has been a fair amount of research done in this area. For several years, Angie Jeffrey at VMS has been studying various weightings of media coverage to determine which factors most directly impact sales. Her research first found that share of discussion was effective, but she postulates that advertising rates are more telling because they reflect the importance of the various media. (See this paper and this paper.) I think it's a step in the right direction, but I generally frown on the use of advertising rates (and of AVEs), so I'm skeptical.

The Media Integrity Index: Why not rate coverage on integrity as rated by stakeholders?

I suggest that it's not the ad rate but the integrity of the publication that most determines consumer credibility and thus drives consumer action. And I'm suggesting that to rank a media outlet in importance, we should ask our stakeholders how they perceive the media outlet in terms of integrity.

I raised the question on Twitter and got some interesting responses:

• "It's all relative... What is good for one is bad for the other. It has to be specified for a purpose, customized."

• "Integrity or perceived integrity? Seems there's a difference there that's highly subjective." --Ryan Anderson

• "Integrity is the measure of perceived relationship (believability) of one-to-many listeners. So measure relationships. I would suggest looking at the way political integrity is measured. It would be a perceived metric, measured by polling audience. --Videodred

• "What's the goal? What's the client want it to say? In the case of journalists, what's the relationship?

• "Integrity has to be measured parallel to influence as audience determines credibility of source." --Mike Maney

So in one way, my proposed Integrity Index has to start with the goal of the coverage. What is it that the company or organization is trying to accomplish? Consider it this way: If you don't care who or what your brand is associated with, and you want exposure pure and simple, then you don't care about the integrity or credibility of whatever media outlet is talking about you. On the other hand, if you're trying to establish a reputation, or build brand loyalty and trust, or trying to reach an audience with certain media preferences, then the credibility of media to your audience should matter a great deal.

How to determine integrity?

So how do we determine which media is trustworthy and credible, and which journalists have or do not have integrity? The wrong way is to look at a list of media and assign weights or values to each one yourself. What you think doesn't matter. All that really matters is what your customers or stakeholders or members or constituents think. So you need to ask them. The best way to do that is to use the Grunig Relationship Instrument.

Now you probably don't need to ask them about all 150 publications on your media list. Remember: Never ask a question about something that can't be changed. If The New York Times is what your boss' boss reads every morning, there's no chance in hell you'll ever take it off you top tier media list. Even if your target audience is nine-year-old girls. So start with a list of suspect publications, blogs or any other type of media outlet.

Remember that different stakeholder groups may rate a given media outlet differently. As I pointed out above, integrity is not a fixed standard. For instance, we would expect Republican viewers to rate Fox News and The New York Times differently than Democratic viewers. It is conceivable that you will want to generate different Integrity Indexes for different stakeholder groups.

Below are some survey questions adapted from the Grunig Relationship Instrument. You may wish to develop your own, based on the specific components of relationships you wish to measure (refer to this paper). Ask your stakeholders whether they agree or disagree with each statement as it pertains to each media outlet, then use the responses to rate the outlets.

1. This media outlet treats people like me and organizations like mine fairly and justly.

2. Whenever this media outlet makes an important decision, I know it will be concerned about people like me.

3. This media outlet can be relied on to keep its promises.

4. I believe that this media outlet takes the opinions of people like me into account when making decisions.

5. I feel very confident about this media outlet's skills.

6. This media outlet has the ability to accomplish what it says it will do.

7. Sound principles seem to guide this media outlet's behavior.

8. This media outlet does not mislead people like me.

9. I think it is important to watch this media outlet closely.

10. This media outlet is known to be successful at the things it tries to do.

Industry-wide integrity standards?

Now, the logical question is, "Why aren't we doing this as an industry?" Shouldn't we be factoring in credibility based on some industry-wide standard? It certainly is a more accurate weighting factor than simple eyeballs or ad cost. But the reality is that your stakeholders aren't going to be identical to my stakeholders, and what matters is how a significantly valid sample of your stakeholders feel. It would be awfully complex and difficult to set industry-wide standards for many different stakeholder groups.

On the other hand, if the PR industry wanted to take on a project to try and accurately weight publications based on their integrity, the world would most definitely be a better place. And it is not far-fetched to imagine an industry-standard Integrity Scale or Survey that could be used to determine the Integrity Index for different stakeholder groups.

Ten No-Cost Ways to Measure Online Engagement

by Katie Delahaye Paine

1. Set up Google Analytics on your blog to find out how many repeat visitors you have. How many pages per visit do they check out? How many go back more than 3 times a week? How many go back and spend more than a second or two on the site?

2. Post a vizu poll on your blog and see how many people respond.

3. Go to xinure and enter the URL of your choice to find out how well it is doing in search engines, links, social bookmarks and a whole bunch of other stats.

4. With many of the leading blog providers like TypePad, check your stats to find out how many people have subscribed, and how many visits per day you're receiving.

5. What's the Conversation Index (the ratio of postings to comments)? In the blogosphere any comment is a good comment because it shows that people are engaged enough in what you are saying to take the time to respond.

6. If you have posted a video on YouTube, or a photo on Flickr, check to see how many people have rated it, and/or commented on it.

7. If you have a presence on Facebook, how many people have joined your group?

8. Ask a question on Facebook and see how many people respond.

9. If you're on Twitter, how many followers do you have? More importantly, how many responses do you get when you ask them a question?

10. Set up a Technorati, Sphere or Icerocket search to find out whether people are writing or talking about you.