The Measurement Standard: Blog Edition

Don Bartholomew on The Digitization of Research and Measurement in Public Relations

Read this article about a new model of PR/social media measurement by Don Bartholomew, author of the MetricsMan blog, over at Social Media Explorer: "The Digitization of Research and Measurement in Public Relations." Here is the crucial chart from it:

Now You Can Judge Customer Purchase Intent with Mouse Click Metrics from Omniture!

Katie Delahaye Paine's New Product Launch Measurement Checklist

 
 

Your step-by-step guide to demonstrating solid results from a product launch.

Click here to download Katie Delahaye Paine's Product Launch Measurement Checklist now.

Back in January of this year, we presented Katie Delahaye Paine's Social Media Measurement Checklist and it was the most popular article and download we've ever offered. (You can read that article here, or just go ahead and download the checklist.) So we thought we'd stick with a good thing and offer another Checklist, this time for product launches.

----------Advertisement: Katie Delahaye Paine and her company KDPaine & Partners can help you measure your product launch, or any other traditional PR or social media program you might need help with. Email Katie Delahaye Paine now, or visit www.kdpaine.com for more information. -------------------

As before, Katie Delahaye Paine's Product Launch Measurement Checklist lays it all out for you -- just about every decision you need to make and every thing you need to do to handle your product launch in a way that will allow you to prove its success and learn from your experience. You can use it to plan your measurement program, and to keep track of your progress, too.

Remember that every program is a little different, so you will have to adapt the Checklist to your own situation. Notice, for instance, that right near the beginning you've got to define your Key Performance Indicators (KPIs). Sometimes not an easy thing.

(And if you need some help, take a look at Katie Delahaye Paine's book "Measuring Public Relationships," which will walk you step by step through most any measurement problem or program you need to deal with.)

So look over the Checklist, adapt it to your own situation, and get your product launch started. And please let us know how we can improve it. Click here to download Katie Delahaye Paine's Product Launch Measurement Checklist now.

Make Data Your Bitch: Dominate your data in ten easy steps.

 by Katie Delahaye Paine

Yes, it’s a great slogan. (Thanks to Gavin O'Malley and Pointroll.) I overheard it at OMMA, a conference that primarily focused on using analytics to measure online marketing. It was full of data wonks and stat geeks who cheerfully claimed credit for all kinds of stuff without taking into account any of the contribution from PR and social media.

Which, of course, got me thinking. The reason these guys (and gals) have all the power and credibility in their organizations is because they do use data to achieve their ends. PR people, on the other hand, have a tendency to let the data push them around. You’ve heard PR’s feeble data wrangling skills expressed at meetings, as in these real-life quotes I've had to deal with: 

• “I really don't believe that PR data, I just know that direct mail works.” 
• Or, “There’s really no data that shows that social media/PR was effective.” 
• Better yet, “You really can’t put any data behind that relationship/reputation stuff.” 

Arrgh!
 

And why do we let this happen? Because we aren’t familiar with statistical analysis -- or are afraid of it, or don’t know how to do it, or haven’t thought of using it. We let everyone else throw their data at us and we just duck under the table. Then we can’t figure out why we aren’t getting a seat at the table.

 

So how do we turn this around? Make data your bitch:

 Use data to achieve your goals. Don't let it push you around. 

 Take control of your data and make it work for you. Don't let it just arrive in waves and pour over your desktop. 

 Understand your data’s strengths and weaknesses. Leverage them so that you win in the end. 

 Don’t be afraid of data. Look it full in the face and say, “Bring it on!” 

Here are ten easy steps to dominating your data. 

1. Trust your data.
Make sure you’re getting data you can have confidence in. If you aren’t sure of your data, you won't use it like the incredibly powerful tool it is. 

• If you’re using an outside vendor, ask to see their inter-coder reliability scores. Ask them for their confidence levels and do periodic checks to make sure the data is coded the way it should be. 
• If you’re doing the data analysis yourself, ask a colleague -- or better yet a graduate student at a local university -- to check it for you.

2. Know your data.
Know it backwards and forwards, inside out and upside down. Take the time to learn the details. If you’re presenting, get prepared by having a friend or colleague ask you the really tough questions ahead of time. Have the answers at your fingertips. 

3. Draw conclusions.
Don’t just say, “There was a big spike in June,” or, “The numbers went up 10%.” Provide the “So what?” Like this, for instance: "The numbers went up 10%, which was significant not just for the increase, but because we accomplished it with half the budget. Therefore we recommend…” 

4. Tease out the really interesting stuff.
Here's an example: We were recently reporting on a thought leadership program in which one organization consistently dominated the industry. Normally, we look at the comparative share of thought leader quotes for each organization. In this case we noticed that the dominating institution didn’t just have the greatest share of quotes, they also had the highest number of individuals that were quoted. We dug into the data to discover that it was the depth and breadth of their program that enabled them to dominate in every area. If their leading expert on swine flu was busy with another interview, they had 4 or 5 others waiting in the wings all with equally fancy titles. The lesson here is that if we hadn't really dug into the data, we never would have discovered what the competitive advantage was.

5. Don’t just do the standard comparison.
Everyone always compares data month-on-month or year-on-year. We recommend looking at data over a thirteen month window to spot long term trends. 

6. Take a good, hard look at the bad stuff.
You’ll learn a lot more from your failures (and look like a hero for stopping a dog of a program) than you will from fixating on a small improvement in performance. Even more interesting: Look at what the competition is doing right. Where are they beating you? Finding out what customers like about the competition will yield much more insight than just listening to them complain about your own company. 

7. Beat the bushes.
Run correlations on anything you find interesting. So it doesn’t correlate, no harm done. Move on. Although, sometimes you can learn more from what doesn’t correlate than what does. 

8. If something doesn’t make sense, pounce.
Now that you really trust your data, trust yourself when something looks a little off. Get down into the dirt and figure out why. Nine times out of ten, it's not the data that wrong, it’s the program. 

9. Increase the depth and breadth of your data.
Beg, borrow and steal data from throughout your organization. Get it from online sources. Take your Market Research department to lunch and see what they’ve got. Track down your competitive intelligence gurus and take them to a ball game or dinner. (They’re the ones who really have the data. And the budgets.) Walk into your corporate library with a batch of fresh homemade chocolate chip cookies and find out what they have access to.

10. Once you’ve tamed the data bitch, dress it up and take it out.
Make it pretty and presentable. All the pie charts in the world don’t tell you nearly as much as a good spider chart. 
 
 

Measuring Engagement on the Web by How Often Visitors Cut and Paste

This is really cool. Zachary M. Seward at Nieman Journalism Lab reports on Tracer.

How Can We Take Office Politics Into Account When Doing Public Relations Measurement?

This is the third of three daily posts on the general topic of politics and public relations measurement. The crux of the problem is: If office politics is important enough to bury your measurement report, then shouldn't you take it into account from the start? 

For instance, suppose you know that a manager will hide your report if it shows results that are not flattering to him. And suppose you actually give him such a report, and then he does bury it and no action is taken as a result of your work. So where are you? Did you measure the wrong thing? Or did you just do your job?

A real-life story: Almost twenty years ago now, I was a bright-eyed, idealistic, and naive youngster, with some experience in publishing, and with a graduate degree earned by doing experiments on human beings. I applied for a job writing research reports for one of the major public relations research vendors. During the interview, the Director of Research explained the job: "Every measurement project has a story behind it. Every report we write means someone's ego or budget or job or career is on the line. I have to know what situation we are getting into, and we have to take that into account when we present our data."

Well, I was pretty much appalled, and refused the job. Didn't want to distort what I saw as my objective integrity. But today, of course, I see The Director's point and would love to be able to go back and ask some questions: How to take the backstory into account? What kind of research or snooping is necessary to find it out? How is it possible to present the research data and at the same time allow for the bruised egos or budgets that might result? --Bill Paarlberg, Editor, The Measurement Standard

Office Politics and Public Relations Measurement: Are we measuring the wrong thing?

 

Yesterday I wrote about public relations measurement and office politics, and brought up the notion that measurement, no matter how solid the data, will not be effective if people have a strong incentive to misrepresent the results. We have all heard of or experienced measurement reports that have been ignored or buried because the results were not as good or as flattering or as useable as someone hoped they would be. I would very much like to hear from anyone doing research on this: How often do public relations measurement reports get ignored because of politics? For what reasons do the reports get ignored?

Some measurement vendors will say, "Hey, that's not our problem. They hired us to tell them what results their program was getting, and if they don't like what we tell them, well, that's just office politics that we can't control. So what if our report got buried somewhere? We still did our job."

But here's the crux of the problem: Maybe we are measuring the wrong thing. What good does it do to present your gold-plated research results when they don't fit into the politics or office culture that is going to use them? If your measurement report was doomed to be ignored, then not only was the measurement not effective, but you measured something that was of no value in that situation. You measured the wrong thing.

Wouldn't it have been better to somehow take the politics into account, and actually produce a report that had a chance of being used, of being effective? People must do this already to some extend. Scope out the political lay of the land before they dive into a project. But no one ever talks about it as a vital component of the measurement process. So, how do you do that? What's the technique or procedure or system to plan or deal with this? -- Bill Paarlberg, Editor, The Measurement Standard

Concerning "The Wire" and Measurement: How does politics affect public relations measurement?

So I just finished watching the fourth season of The Wire. (The Wire is an HBO TV series about Baltimore. I watch it on DVD, and it's got to be about the best TV ever. Right up there with The Sopranos and Deadwood.)

One of the themes of the fourth season was "juking the stats," the way in which the police and education departments distort their results to make them appear more effective than they actually are. For instance, the cops would purposely not investigate homicides, in order to keep the official city homicide rate low. And the schools would classify as "proficient readers" those kids who were reading two years or less behind their actual grade level. 

The big point here is that measurement, no matter how solid the data, will not be effective if the people who are doing it have some strong incentive to misrepresent the results. There must be a lot of public relations measurement programs that never get reported, or get distorted in the way they are reported, because the results are inconvenient or embarrassing or otherwise unwanted. So, what good is it to have the best measurement tools and the most accurate measurement program, when the results are destined by office politics to be ignored or distorted?

Here at The Measurement Standard we spend a lot of time and effort trying to convince people that measurement can improve the effectiveness of their public relations, and showing them better and more accurate ways to collect and analyze data. But we never talk about the politics behind measurement as an important aspect of the process. (Well, Katie and I do mention "getting buy-in from everyone involved" in her book, but only very briefly. As if all you have to do to get around awkward politics is to hand people a signoff sheet or something.)

So my purpose here is to propose that maybe we have been ignoring a very important part of the process. Maybe the office politics behind measurement can be just as important as actually getting the right data. (I notice I am not the first to bring this up. See Victor Tan Chen at In The Fray.)

So how can we get a handle on this? Who's doing research on this? How can we get an idea of how many measurement programs have been scuttled, or languish on a shelf somewhere, because the results were not quite what someone, or everyone, had hoped?  -- Bill Paarlberg, Editor, The Measurement Standard

The All-Purpose, One-Size-Fits-All Public Relations Measurement Program Checklist

Measurement Tools 101

 

Everything you need to know before you start any public relations measurement program. Don't leave home without it.

This basic information checklist could be the handiest, most convenient, and most valuable tool in your public relations measurement quiver. No matter what project or media you are working with, these are all the things you should know -- all the ducks to get in a row -- before you jump in the water.

Some of these are easy, some are hard, and for some you've got to think, maybe make some tough decisions. But, by the time you're done your brain -- and your project -- will be up to speed. Plus -- Bonus! -- it will be obvious to your client or your boss that you've not only got a clue, you've got the measurement bull by the horns. And, yes, you can download the Checklist here as a pdf.

The Measurement Program Checklist

Step 1: What are your objectives?

1. What are your organization's key goals for this year?

2. What are your department's key goals for this year?

3. What do you hope to accomplish with your measurement report?

• Get budget approved
• Increase budget
• Increase head count
• Get more internal support
• Get more external support
• Justify my existence
• Get a raise
• Get promoted
• Change the world

Step 2: What audiences are you targeting?

List all that apply, not just the following common ones. (If in doubt, put all the communications people in your organization in a room and ask them.)

• media
• prospects
• customers
• partners
• employees
• governments
• NGOs
• communities
• investors
• thought leaders
• international community
• other

How does a good relationship with your various target audiences benefit your organization?

1. Increases sales

2. Increases attendance

3. Increases donations

4. Increases likelihood of desirable legislation passing

5. Increase preference

6. Gets messages out

7. Improves employee retention

8. Improves employee loyalty

9. Improves customer retention

10. Improves customer loyalty

11. Improves likelihood of purchase

12. Attracts new customers

13. Attracts new prospects

14. Attracts new donors/potential donors

15. Increases amount of purchase

16. Increases frequency of purchase

17. Boosts stock price

18. Increases profitability

19. Reduces turnover

20. Decreases time to market

21. Decreases number of complaints

22. Decreases absenteeism

Step 3: Set priorities

Prioritize your audiences with this exercise: You have a total of 100 points to allocate. Award those points to the audiences that you have identified in order of their importance to your organization, based on your answers to questions in Step 2.

Step 4: Determine a benchmark

Who or what keeps your boss/client up at night? In other words, what are the competitive threats or perceived competitive threats to your organization? Select from the following lists to determine what you will be comparing your results to:

• Competitor(s) / Peer companies
• Industry benchmarks
• Yourself over time
• Last year's results

Step 5: Select the right measurement tool

If your objectives (Step 1 above) include increase awareness, attitude change or education, you will need to conduct a survey. Do you have email addresses for all those you want to survey? Do you have telephone numbers for all those you want to survey?

If you are seeking to measure sales and leads, you should be tracking web site traffic. If you are measuring media relations you will need to consider the following criteria in your measurement efforts:

• Tone of coverage
• Share of voice vs. the competition
• Share of discussion vs. the competition
• Share of spokesperson visibility
• Degree to which you have communicated key messages
• Degree to which you are favorably or unfavorably positioned on key issues
• Audience reached
• Prominence of coverage – the visibility of your brand within the story
• Dominance of coverage – the extent to which your brand dominates the coverage or is subservient to another brand
• Proactive vs. reactive media
• Key analysts quoted
• Type of analyst quoted
• Key reporters
• Key publications
• Key topics or subjects
• Type of article (check all that apply)
  • ____Letter to the editor
  • ____Editorial
  • ____Product review
  • ____Product brief
  • ____Industry round up
  • ____Application article
  • ____Article about the competition
  • Other _____________________________________
• Key messages, list below:

  1._________________________________________________________

  2. _________________________________________________________

  3. _________________________________________________________

  4.__________________________________________________________

5. __________________________________________________________

• Industry issues:

• __Employer of choice
• __Investment of choice (financial strength)
• __Vendor of choice (good/best value for the money)
• __Good corporate citizen
• __Environmentally responsible
• __Global leadership
• __Other: _______________________________________________________
• Prominence
  • Organization is mentioned in the headline, photograph or caption
  • Organization is mentioned in the top 20% of the article
  • Organization is mentioned in the bottom 80% of the article
• Dominance
• Organization is the focal point
• Organization is one of several mentioned
• Organization is only mentioned in passing
• Other _________________________________________
• Subjects (note all that apply)
  • Financials
  • Management changes
  • New product launch (specify product names)____________________________
  • Coverage of crisis or scandal (specify) ________________________________
  • Merger or acquisition
  • Customer
  • Corporate philanthropy
  • Industry
  • Other _________________________________________
• Influencers and key spokespeople
  • Company spokespeople (provide names)

_______________________________________________________________________

_______________________________________________________________________

_______________________________________________________________________

_______________________________________________________________________

_______________________________________________________________________

_________________________________________________________________

• Competitors' spokespeople (provide names if available)

_________________________________________________________________

_________________________________________________________________

_________________________________________________________________

_________________________________________________________________

• Industry analysts
• Financial analysts
• Government officials
• Educators
• Other

Other considerations:

• Approximately how many clips a month appear about you and the competition? (If you are currently receiving more than 250 articles per month you may want to consider computer-aided content analysis.)
• How frequently do others in your company report results?
• How often would you/your boss/your client like to see a PR measurement report?

###

Easy Strategies to Get Started Measuring Your Public Relations

 

Quick and simple techniques to start measuring your public relations programs and impress your boss.

by Katie Delahaye Paine and Bill Paarlberg

If you're reading this and you're not actually measuring your public relations programs yet, then the only New Year's resolution that really counts is for you to get started measuring. Which, as it turns out, is going to be a whole lot easier than those other resolutions you made a couple weeks ago and probably flunked out of already.

Measurement doesn't have to be a big deal. We've rounded up plenty of ways you can get started without spending much time or money at all. In fact, for several of our tips below, the service is free and the data gets delivered right to your in-box. Of course, if you want to get beyond the newbie stuff you might have to click around the Web some, do some reading, maybe break a sweat thinking about what your data means. (Relax -- that's a different article.)

Here is a baker's dozen of quick and painless techniques to get something started.

1. Get with Google:

• Sign up for Google Alerts for your organization or brand, and for at least three of your competitors or peers. Then tell your boss you have installed a crisis alert monitoring system.
• Put Google Analytics on every website that you control. (Yeah, the instructions need a serious usability makeover, but once you get the code, you just paste it into all your pages. Cake.) Then tell your boss you don't need no stinking WebTrends. (Actually, you probably do need WebTrends if you have to do advanced analyses -- it's much more robust and customizable -- but we're talking quick and easy here.)

2. Time for Twitter:

• Go ahead and get yourself on Twitter. See what it's all about, in 140 characters or less.
• And even if you don't want to start Tweeting yourself, you can still set up an RSS feed for a Twitter Search on your brand/s, your competition, and anything else you want to keep track of. Tell your boss you've implemented a Twitter monitoring system.
• And for you heavy Twitter users, see how influential you -- or your boss -- is on Twitter: Go to www.twinfluence.com.

3. Educate yourself:

• Stay on top of social media every day, with a click or two: socialmediabiz.com will send you daily email updates.
• Get to know the Institute for Public Relations website, especially the research and papers on measurement.
• Check out any of the numerous YouTube videos that talk about how to measure public relations and Social Media, (especially, in all modesty, the ones that feature our Ms. Paine).
• Read Katie Delahaye Paine's book How to Measure Public Relationships. OK, it's not free, but measurement gurus got to make a living. And it's the best $29.95 you'll ever spend on measurement. Click here to buy it now.

4. New media meets old school:

• Set up a free Survey Monkey account and start surveying your constituencies.
• Back to basics: Use Survey Monkey to measure your relationships. Send out a survey to assess the level of trust, commitment and satisfaction they feel about your brand.

5. More easy online stats:

• Get yourself a Facebook account and, at least once a week, conduct searches on your brand and your competition. Then tell your boss you are monitoring Facebook. (For more on measuring Facebook, see "Measuring Community on Social Networking Sites: A detailed how-to guide for measuring MySpace and Facebook.")
• Go to www.xinureturns.com and see how your URLs stack up.
• Go to www.compete.com and see how you stack up against the competition.
• Go to kineda.com and use their widget to see how your "bloglebrity status" ranks.

6. All together now:

• Now put together your own dashboard. Whatever metrics from above you actually have, put them in an Excel spread sheet and set up Key Performance Indicators. Repeat monthly. Progress confidently toward success.

Katie Delahaye Paine's Top Ten Public Relations Measurement Tool Wishes for 2009

Public Relations Measurement Tools We'd Like to See

1. An operant conditioning device
...attached to a sensitive body part of all corporate spokespeople that gives them increasingly painful sensations when they stray off message, and increasingly pleasurable sensations the more on message they are.

2. One source...
...that accurately, and completely gathers items from newspapers, magazines, TV, radio, blogs, Twitter, online publications, websites, YouTube, Facebook, Flickr, etc. I know I've been asking for this for five years, but Santa, I've been really really good and really, really patient and I want it now! I can't be the only one: Who in their right minds wants to filter through eNR, Cision, Meltwater, Burrelles, Vocus, VMS, Radian6, Google News, TVEyes, Critical Mention, and god knows what else just to make sure you're tracking the content of all your key publications?

3. A charting program that gives results like this...
...for PR measurement that will automatically take the data from a media content analysis and create a chartogram to indicate where our clients have the most visible presence, the most favorable presence, the most negative presence, and where the most messages are appearing.

4. 100% accuracy...
...from whatever source we use for content collection.

5. SPSS style statistical analysis...
...for Google Analytics. I want the ability to integrate Google Analytics with things like Twinfulence, my blog stats and other analytic tools so I can actually figure out whether my obsession with Twitter really is driving business to my website. It looks that way, but I want some factor analysis to make it real.

6. Google predictive analysis...
...I want to be able to use Google Analytics to tell me what I should be doing more of. Google Analytics is great at doing stuff like that if you're selling widgets, but I want the same capability for services.

7. Another charting program...
...that automatically puts little bubbles on the chart to signify major events that are driving a change in the results. It's a little thing but it makes me nuts to have to manually add explanations to our client's charts.

8. An interactive application for iPhones and Android phones...
...that allows people to instantly give feedback on what they are seeing, reading, watching or hearing directly into our database.

9. A better way than just tags...
...to search for brand mentions on YouTube and Flickr.

10. A required course on how to measure results correctly...
...for everyone who graduates with a degree in Public Relations or Mass Communications. (And, of course, it would be nice if my book, Measuring Public Relationships, were required reading.)

Fascinating Correlation Research on SAT Scores vs. GPA in Predicting College Performance

In public relations measurement, we are often trying to investigate correlations between variables in an effort to determine what effect public relations efforts have on something else: 

• Does greater use of social media result in more votes in an election? 
• Does more effort on building relationships with stakeholders result in lower future legal costs?
• What predicts sales better: The number of positive impressions or advertising sales equivalencies? 

As we all know, it is often a very tricky business to get clear answers in this sort of correlation research. In particular, it is often difficult to isolate the effects of one variable from another.

 

Well, here's an example of how this type of research can provide very useful results. In today's NYTimes, this article, "The Test Passes, the Colleges Fail," tells the story of how the impacts of SAT scores and high school GPAs were separated to determine their ability to predict college success at SUNY. Thanks to a serendipitous combination of admission policy changes and the structure of the university system, the effects of the two variables were teased apart to yield some strong and very interesting results.

Here's a summary of results from the paper:

“...we find that among a group of SUNY campuses with very different missions and admissions standards, and at which the high school grade-point averages of enrolling freshmen improved by the same modest amount (about 2 percent to 4 percent), only those campuses whose incoming students’ SAT scores improved substantially saw gains in graduation rates.”

 

It's not often, of course, that events drop a ready-made, real-life experiment into one's lap. But in complicated social research like predicting student success, or public relations measurement, these situations can tell a marvelous story.  --Bill Paarlberg

Update: An interesting follow-up study from Johnson C. Smith University.

Counting Words (Hey, isn't that what PR measurement does?)

In today's NYTimes there is an article about word counting ("He Counts Your Words (Even Those Pronouns)"), and what the frequency of use of different types of words can imply about their author.

Maybe this analysis could be applied to PR measurement. If increased use of causal words indicates health improvements in individual people, then perhaps companies could tailor the subtle subtext of their press releases by using more or less of them.

The article includes a word use analysis of the political candidates that has some counterintuitive results, to my thinking: The Democrats’ use is “concrete and restrained,” whereas the Republicans’ is “cheerfully unrestrained, future-focused.” Very interesting. --Bill Paarlberg

KDPaine & Partners Debuts MarketFramer

New Measurement Tools

A new social media-based tool to help marketers better frame messages for their markets

KDPaine & Partners launched a new public relations measurement tool last month to help organizations quickly get a handle on what their marketplace is saying and thinking, enabling them to craft their messages accordingly. Based on KDPaine & Partners' 22 years of media analysis and research experience, the new tool, MarketFramer, takes fundamental principles of sociology, public relations measurement and survey research and applies them to social media, blogs and other consumer generated media, to order to produce a report that answers questions like:

• What are the hot issues and battles being discussed in the market?
• How are people responding to significant developments in your market?
• What are the key phrase and search terms that people are using to describe your business space?
• What's the best way to position your organization or products in a new marketplace?
• What's the best way to respond to a developing issue or crisis?
• What does the market think about your most recent product release or announcement?

"Ever since we started measuring consumer generated media in 1995, people have been asking us if there's a way to get answers to questions like these without doing a large-scale survey research project," explained Katie Delahaye Paine, CEO of KDPaine & Partners. "In part they have budget concerns, and they also question the results of phone surveys when many consumers only have mobile phones. But, they're mostly concerned about the time it takes and the complexity of the programs. People don't have time to wait for survey results, they need answers right away," Paine added.

Conversations are driving choices in the marketplace

"At KDPaine & Partners, we know that conversations are driving choices in the marketplace. Because they're based on what people have experienced, we believe that those conversations are every bit as accurate a predictor as traditional survey research. We thought why not analyze and aggregate those conversations to answer the questions clients are asking?"

Paine explains MarketFramer's simple 3-step process. "It starts with a conversation about which markets are important, which brands are important, and who or what might be influencing the conversation," Paine explains. "Armed with that information, we use a variety of social media analysis tools to search the online environment for conversations that relate to the client's issues, market or questions. We then gather and analyze that data to determine trends and issues and deliver to the client a simple, clear, easy-to-understand report that answers the specific questions the client is asking. Additionally, our clients can access their data online 365/24/7."

KDPP has already used MarketFramer to help a networking company figure out a positioning strategy, and to help a non-profit figure out its messaging. Pricing for MarketFramer starts at $10,000 per marketplace and typical turnaround time is about two weeks.

Measuring Naked Relationships: How to evaluate the impact of a social media program on your public relationships.

Your step-by-step guide to using relationship metrics to evaluate the success of your social media program.

by Katie Delahaye Paine

I've talked a lot here -- also in my speeches, in my blog, and of course in my recently published book Measuring Public Relationships -- about the importance of measuring relationships. I believe that without factoring in the impact that your social media program has on your relationships (with employees, community and constituencies) you are undervaluing your efforts.

So, how does one actually measure relationships? Well, now that you asked, my book explains this with regard to most types of public relations programs. Social media being all the rage right now, I thought it would be appropriate to provide a step-by-step guide as to how to actually do it for social media programs.

Whatever type of program you want to measure, the basic technique is similar: You conduct a survey of your audience using a special set of questions designed to specifically measure the different components of relationships. You do this before and after your social media program is in place, and you do it for your organization and as many competing organizations as you can afford to. Then you compare the data before and after, and between your organization and the others, and then you know where your relationship with your audience stands and where you need to go.

The Grunig/Hon Relationship Research

Before we get to the nuts and bolts, here's a bit of background. A decade or so ago, University of Maryland Professors Jim and Laurie Grunig and Linda Hon synthesized communications and sociology research and theory into a paper published by the Institute for PR called "Guidelines for Measuring Relationships in Public Relations." Their feeling was that, amidst all the brouhaha about quantifying public relations, we were forgetting the essential truth that PR was about relationships. And so you needed to measure the impact that your efforts were having on those relationships.

Their research isolated six fundamental components of relationships -- Trust, Satisfaction, Commitment, Control Mutuality and Exchange/Communal -- and they designed and tested 75 statements to measure those components. These statements are of the sort: "This organization treats people like me fairly and justly," and, "I feel a sense of loyalty to this organization." You can see them all, and copy them for your own use, on my blog or in the Grunig/Hon paper. For an example of these statements successfully used to measure public relationships in a non-social media context, see this research by Forrest W. Anderson and Paul Raab.

Nine Steps to Measure Your Social Media Relationships

Before we start, remember that to isolate the effect of your social media program, you must begin your measurement before you launch that program. Then you'll have a benchmark to compare against later: Before Social Media vs. After Social Media. Without a Before benchmark you won't know how your social media program has changed your relationships.

Of course, you can always begin measuring after your social media program has started, and by doing so you will be able to do ongoing evaluation of your relationships. Which is a good thing. But a better thing is to isolate the effect of social media, and to do that you must compare before and after.

So use Steps 1 through 9 below to survey your audience before your program begins, and thus establish your benchmark. Then begin your social media program, and after an appropriate time, say three or six months (depending on your situation), re-survey to see what has changed.

To be sure that whatever change you find is solely attributable to the social media program, you must hold constant any other PR programs that affect your social media audience's perception of your organization. Yes, it's tricky, and it's not always an ideal world. But if you are trying to measure the effect of your new blog at the same time as the Promotions Department decides to give away A Free Cadillac With Every Purchase, then you can kiss your results goodbye.

Step 1: Define the audience for your social media program.
Social media is about conversation and engagement, so decide with whom you want to converse and engage. If you're starting an internal blog, your audience is your employees. If you're starting an IdeaStorm-type customer community, your audience is anyone likely to buy or recommend your product. If your mission is advocacy, your audience might be voters, politicians or industry influencers.

Step 2: Get a list of your audience's email addresses and/or phone numbers.
To get a representative sample you will need at least 500 names for each organization you are testing (more on that later). If you already have a list of your members, subscribers or customers, then you are ahead of the game.

If you have to purchase your list, then potential vendors vary with the type of sample you're looking for (mail, phone, web). Most lists are sorted based on demographic or title data. There are a lot more resources out there for mail addresses and those resources do not necessarily need to be survey sample companies. For email addresses, some reputable firms include Survey Sampling, e-Rewards and Zoomerang.

Step 3: Pick a survey methodology.
The Grunigs would recommend in-person surveys for the best results, but most researchers find that to be very expensive. Phone surveys are fast and provide very accurate results, but again, depending on the audience, may be cost-prohibitive. Email surveys are an increasingly accepted methodology, and for social media can be appropriate and highly reliable, since, presumably, your audience is all on email.

You may be able to piggyback on an existing survey going out to your community. If marketing, customer satisfaction, business development or anyone else in your organization is doing a survey, see if you can add a few of the Grunig/Hon statements to it.

Step 4: Select which of the Grunig/Hon statements are most appropriate to your organization.
You can probably only impose on someone for 7-10 minutes of their time, so you need to pick which statements you will include. Grunig and Hon suggest that if you want to shorten the survey, you use only the boldfaced items. Not all statements are appropriate for all organizations, so pick and chose the ones that will be most meaningful to your audience.

Step 5: Prepare your survey.
If you are using an electronic survey system like Survey Monkey, you need to create an introductory screen that explains what you are doing and how the scoring works. For instance:

In order to better understand the needs and perceptions of our marketplace, we'd like to ask you some questions. Please tell us whether you agree or disagree with the following statements as they apply to X company/organization.

Explain that 1 means "totally disagree" and 7 means "completely agree," and give them an option for "no opinion." You also need to ask them the same questions about a competing company or organization, so you have comparable data on the competition to compare to.

Step 6: Send out the survey.

Step 7. Resend the survey.
Depending on your audience it may take several tries and an incentive to get sufficient responses (I'll do just about anything for an Amazon or Starbucks card). How many is sufficient? Well, it depends on how you plan to break down your analysis, but in general plan to resurvey until you get at least a 10% return. If in doubt, talk to your local survey expert.

Step 8. Analyze and learn from the results.
Calculate a mean score for each relationship component. There are both positive and negative statements in the survey instrument, so make sure you take that into account. Compare your mean on each score to the competition. (And of course to your earlier survey results, if this survey is not the benchmark.)

Step 9. Implement your program and measure again.
If this is your pre-social media program (benchmark) survey, then implement your program now, and measure again in six months. Or, if your program has been running for a while and your analysis indicates you need to make changes, then make the changes now and let them work on your audience for six months before you measure again.

Good luck, and let me know how things go.

Should We Trust Web-Based Studies? (This research says Yes.)

This comparative study of Web-based survey samples and paper-and-pencil surveys (by Samuel D. Gosling, Simine Vazire, Sanjay Srivastava and Oliver P. John and which appeared in American Psychologist) is now four years old, but its conclusions are probably still valid: Yes, Web-based self-report and self-selected samples can be trusted for surveys. From the study's summary:

"Internet data collection methods, with a focus on self-report questionnaires from self-selected samples, are evaluated and compared with traditional paper-and-pencil methods. Six preconceptions about Internet samples and data quality are evaluated by comparing a new large Internet sample (N
361,703) with a set of 510 published traditional samples. Internet samples are shown to be relatively diverse with respect to gender, socioeconomic status, geographic region, and age. Moreover, Internet findings generalize across presentation formats, are not adversely affected by nonserious or repeat responders, and are consistent with findings from traditional methods."

Get the whole study here.

Ten No-Cost Ways to Measure Online Engagement

by Katie Delahaye Paine

1. Set up Google Analytics on your blog to find out how many repeat visitors you have. How many pages per visit do they check out? How many go back more than 3 times a week? How many go back and spend more than a second or two on the site?

2. Post a vizu poll on your blog and see how many people respond.

3. Go to xinure and enter the URL of your choice to find out how well it is doing in search engines, links, social bookmarks and a whole bunch of other stats.

4. With many of the leading blog providers like TypePad, check your stats to find out how many people have subscribed, and how many visits per day you're receiving.

5. What's the Conversation Index (the ratio of postings to comments)? In the blogosphere any comment is a good comment because it shows that people are engaged enough in what you are saying to take the time to respond.

6. If you have posted a video on YouTube, or a photo on Flickr, check to see how many people have rated it, and/or commented on it.

7. If you have a presence on Facebook, how many people have joined your group?

8. Ask a question on Facebook and see how many people respond.

9. If you're on Twitter, how many followers do you have? More importantly, how many responses do you get when you ask them a question?

10. Set up a Technorati, Sphere or Icerocket search to find out whether people are writing or talking about you.

The Interpretation of Dreams: Dreams as Public Relations Measurement -- I Dream of Hillary, I Dream of Barack

Many public relations professionals find that media analysis or survey research are tools sufficient to measure their programs. The more adventurous bring in external data and more esoteric measures of success like donations or number of new memberships or even lives saved. In politics, KDPaine and Partners has recently used YouTube video counts to predict primary results, and our Ms. Paine has ruminated on the possibility of using the number of political lawn signs as an election predictor. Not until now, however, has anyone used dreams to measure political progress.

In this week's New Yorker, Ben McGrath writes about Sheila Heti and her new blog "I Dream of Hillary... I Dream of Barack." This blog, a repository of reader-submitted dreams about the candidates, can be interpreted as a rough poll of interest in those candidates. As Mr. McGrath says, "...what if the recurrences of Presidential candidates in people's dreams were meaningful in the aggregate?" As of the writing of the article, "..Obama's edge in the over-all dream count... was roughly equivalent to his lead in the latest Gallup poll."

Ms. Heti's blog has been so successful she plans to open an I Dream of McCain section, so in the future we will have bipartisan dream data to go on. --Bill Paarlberg, Editor, The Measurement Standard

Why Public Relations Measurement Is Like Phrenology, or... Just because it is easy to measure, doesn't mean it's the best thing to measure.

PR measurement and marketing measurement often find themselves in a situation analogous to the 19th-century practice of phrenology. Way back then, some people used to think that the shape of your skull indicated your personality or intelligence. It was a very easy thing to measure, so they did. And they hung on to it as a serious area of study for much longer than they should have, because it was far more difficult to measure personality or intelligence by other, more effective, means.

Click-throughs aren't quite as useless as bumps on the head, but still... In the news today is a study by Starcom USA, behavioral targeting firm Tacoda, and comScore that suggests that clickthroughs might not be a very effective measure of the usefulness of online ads. See here for the news and here for Sam Harrelson's ReveNews discussion.

The obvious point here is that clickthroughs, while easy to measure, probably are not very effective at measuring what we would like to measure. The more interesting point is: Why aren't we measuring what we really want to know?

I have noticed that PR and marketing metrics are often determined by what technology makes easily possible, rather than by what we would really like to know. Thus people tend to measure what technology drops in their laps (the easy things like click-throughs or AVEs) rather than something more relevant but difficult to measure (like, "Did the ad with the red type result in more sales among wealthy women?"). Then at some point they realize that their easy-but-less-relevant metric is not very effective, and then they spend a lot of time asking why the metric doesn't work.

What they should be asking themselves is: "What made us think that that metric would work in the first place?"

Instead of focussing on what technology is available (in this case click-throughs), wouldn't it be more productive to think about human thought and behavior first? (I've written about this before: "Measurement's Empty Head: Measurement ignores the most complex part of PR.") First think about what it is that might make PR work -- why it affects people's thoughts and behaviors -- and then decide what might make a good metric. First determine what we really want to measure (regardless of the technology or data available), then figure out a way to measure it. -- Bill Paarlberg, editor, The Measurement Standard

Great Things About A Public Relations Measurement Dashboard

Measurement Tools

13 Reasons Why You Need A Public Relations Measurement Dashboard
And if you have one already, here's a baker's dozen ways in which your dashboard can help you do your job better every day.

1. Your dashboard provides data for making better strategic decisions.
Whether you're being asked to "Put out a press release," or "Put together a press tour," you need to know what's worked and what hasn't worked in the past. The trend data in your dashboard will tell you not just what has worked in terms of generating exposure, but also whether the tactic resulted in more or less positive exposure, more or less message communication, and the impact the effort had on new user registrations.

2. Think of your dashboard as a continuous improvement tool at your finger tips.
A dashboard is essentially a live, searchable, continuous quality improvement tool. You can instantly determine which of your efforts resulted in better performance. By weeding out the tactics that didn't work, you are ensured of increasing effectiveness.

3. Your dashboard tells you if you are winning or losing key battles.
In any PR department, winning or losing the major positioning battles is critical. How you spend your day may depend on whether you're ahead or behind on a key initiative. Your dashboard can tell you right now how you're doing on any given battle, so you'll know where you need to place your resources. It also provides a long term trend view of whether or not you're winning or losing the battles. Furthermore, it helps you figure out why.

4. Your dashboard helps you figure out if your resources--budgets as well as time--are being spent effectively.
These days, chances are time is even scarcer than money in most PR departments, so where you spend your time becomes a critical decision. Checking your results on a regular basis--to determine if your agency is being effective and if your efforts are paying off--enables you to allocate resources more effectively.

5. Your dashboard helps you figure out which of your actions has the biggest impact on business.
By integrating media results with signups, retention, and other key data, you can make decisions based on the expected impact on the business, not just on media exposure.

6. When you're asked for goals and objectives, your dashboard helps you set realistic expectations.
Whenever you go into a meeting these days, people expect you to be able to set specific, numeric objectives. Your dashboard provides the data at your fingertips with which you can confidently set those benchmarks.

7. Your dashboard can tell you which messages are resonating, which are falling on deaf ears, and why.
Despite the best laid plans, there are times when a key message just isn't getting picked up, either because the media doesn't care or because your spokespeople aren't delivering it. Your dashboard helps you determine which messages are or aren't getting into the media. And better yet, it can help you determine the cause.

8. When a reporter calls, you can look up, instantly, how they've been covering you, what they've covered, and whether or not they "get" your key messages.
Before you answer that call from a reporter, you should know if he's been hostile or friendly, whether he gets your key messages or not, and what topics he's been covering. All that information is available at your fingertips in your dashboard.

9. When you're trying to decide on the best spokesperson, your dashboard can tell you who will be most and least effective.
To quickly determine who the best spokesperson is for any particular interview, use your dashboard to look up who is being quoted in your coverage, then note the tone (positive, negative or neutral) of the article, and the extent to which it conveys a key message.

10. Your dashboard will help you make better decisions about which publications to pitch for which story.
Suppose you have an exciting announcement, but you don't know which media to pitch first. Check your dashboard and determine which publications are most likely to cover the topic or product category. If it's not something you've ever talked about before, do a quick Google News search. Compare the top publications that come up with the data in your dashboard to see which ones are most likely to give you favorable coverage.

11. You can make better decisions about which bloggers to engage in a conversation.
Bloggers and other social media influencers are a key element of many communications programs. By checking your dashboard to see who is blogging about which issues and how favorable they are towards you, you'll have a better understanding of which bloggers you need to start a conversation with.

12. Your dashboard enables you to find influential spokespeople and analysts relevant to the issues.
The dashboard will help you not just identify who are your most influential analysts, but also who are the analysts and experts most relevant to your issues, and which ones are talking about the issues most often.

13. Your dashboard gives you ammo to push back against dumb ideas.
Every communications department is asked to take on a wide variety of projects. Some are wonderful, and some are dogs, but without data it's hard to push back against the latter. Your dashboard can save you countless wasted hours by demonstrating what hasn't worked in the past, and providing you with welcome proof that not all ideas are good ones.