The Measurement Standard: Blog Edition

Pornography, Public Relations Measurement and Anytown, USA

There are few things in life as exciting as solving a measurement problem with the innovative use of some new source of data. Followers of this blog and The Measurement Standard newsletter probably share our enthusiasm about Freakonomics (see our review here, and another example on beauty premiums here), and know how much we enjoy coming up with new ways of gaining insight from data. (Heck, it is in large part the theme of Katie Delahaye Paine's new book Measuring Public Relationships, and is the topic of a couple of her recent blog posts: "Another alternative to the Ad Value nightmare" and her Optimum Content Score proposal for media measurement. [She is the publisher of The Measurement Standard.])

So we encourage you to head right over and read an article in today's NY Times about using Google Trends data to demonstrate community standards in a pornography trial: "What’s Obscene? Google Could Have an Answer." It seems that these legal proceedings often face difficulty in establishing just what is or isn't obscene. And, wouldn't you know, Google has data on that: if Anytown, USA, searches as often for "orgy" as it does for "apple pie," then maybe some conclusions can be made about community standards. Or maybe not. Read the article. --Bill Paarlberg, editor

Measuring Naked Relationships: How to evaluate the impact of a social media program on your public relationships.

Your step-by-step guide to using relationship metrics to evaluate the success of your social media program.

by Katie Delahaye Paine

I've talked a lot here -- also in my speeches, in my blog, and of course in my recently published book Measuring Public Relationships -- about the importance of measuring relationships. I believe that without factoring in the impact that your social media program has on your relationships (with employees, community and constituencies) you are undervaluing your efforts.

So, how does one actually measure relationships? Well, now that you asked, my book explains this with regard to most types of public relations programs. Social media being all the rage right now, I thought it would be appropriate to provide a step-by-step guide as to how to actually do it for social media programs.

Whatever type of program you want to measure, the basic technique is similar: You conduct a survey of your audience using a special set of questions designed to specifically measure the different components of relationships. You do this before and after your social media program is in place, and you do it for your organization and as many competing organizations as you can afford to. Then you compare the data before and after, and between your organization and the others, and then you know where your relationship with your audience stands and where you need to go.

The Grunig/Hon Relationship Research

Before we get to the nuts and bolts, here's a bit of background. A decade or so ago, University of Maryland Professors Jim and Laurie Grunig and Linda Hon synthesized communications and sociology research and theory into a paper published by the Institute for PR called "Guidelines for Measuring Relationships in Public Relations." Their feeling was that, amidst all the brouhaha about quantifying public relations, we were forgetting the essential truth that PR was about relationships. And so you needed to measure the impact that your efforts were having on those relationships.

Their research isolated six fundamental components of relationships -- Trust, Satisfaction, Commitment, Control Mutuality and Exchange/Communal -- and they designed and tested 75 statements to measure those components. These statements are of the sort: "This organization treats people like me fairly and justly," and, "I feel a sense of loyalty to this organization." You can see them all, and copy them for your own use, on my blog or in the Grunig/Hon paper. For an example of these statements successfully used to measure public relationships in a non-social media context, see this research by Forrest W. Anderson and Paul Raab.

Nine Steps to Measure Your Social Media Relationships

Before we start, remember that to isolate the effect of your social media program, you must begin your measurement before you launch that program. Then you'll have a benchmark to compare against later: Before Social Media vs. After Social Media. Without a Before benchmark you won't know how your social media program has changed your relationships.

Of course, you can always begin measuring after your social media program has started, and by doing so you will be able to do ongoing evaluation of your relationships. Which is a good thing. But a better thing is to isolate the effect of social media, and to do that you must compare before and after.

So use Steps 1 through 9 below to survey your audience before your program begins, and thus establish your benchmark. Then begin your social media program, and after an appropriate time, say three or six months (depending on your situation), re-survey to see what has changed.

To be sure that whatever change you find is solely attributable to the social media program, you must hold constant any other PR programs that affect your social media audience's perception of your organization. Yes, it's tricky, and it's not always an ideal world. But if you are trying to measure the effect of your new blog at the same time as the Promotions Department decides to give away A Free Cadillac With Every Purchase, then you can kiss your results goodbye.

Step 1: Define the audience for your social media program.
Social media is about conversation and engagement, so decide with whom you want to converse and engage. If you're starting an internal blog, your audience is your employees. If you're starting an IdeaStorm-type customer community, your audience is anyone likely to buy or recommend your product. If your mission is advocacy, your audience might be voters, politicians or industry influencers.

Step 2: Get a list of your audience's email addresses and/or phone numbers.
To get a representative sample you will need at least 500 names for each organization you are testing (more on that later). If you already have a list of your members, subscribers or customers, then you are ahead of the game.

If you have to purchase your list, then potential vendors vary with the type of sample you're looking for (mail, phone, web). Most lists are sorted based on demographic or title data. There are a lot more resources out there for mail addresses and those resources do not necessarily need to be survey sample companies. For email addresses, some reputable firms include Survey Sampling, e-Rewards and Zoomerang.

Step 3: Pick a survey methodology.
The Grunigs would recommend in-person surveys for the best results, but most researchers find that to be very expensive. Phone surveys are fast and provide very accurate results, but again, depending on the audience, may be cost-prohibitive. Email surveys are an increasingly accepted methodology, and for social media can be appropriate and highly reliable, since, presumably, your audience is all on email.

You may be able to piggyback on an existing survey going out to your community. If marketing, customer satisfaction, business development or anyone else in your organization is doing a survey, see if you can add a few of the Grunig/Hon statements to it.

Step 4: Select which of the Grunig/Hon statements are most appropriate to your organization.
You can probably only impose on someone for 7-10 minutes of their time, so you need to pick which statements you will include. Grunig and Hon suggest that if you want to shorten the survey, you use only the boldfaced items. Not all statements are appropriate for all organizations, so pick and chose the ones that will be most meaningful to your audience.

Step 5: Prepare your survey.
If you are using an electronic survey system like Survey Monkey, you need to create an introductory screen that explains what you are doing and how the scoring works. For instance:

In order to better understand the needs and perceptions of our marketplace, we'd like to ask you some questions. Please tell us whether you agree or disagree with the following statements as they apply to X company/organization.

Explain that 1 means "totally disagree" and 7 means "completely agree," and give them an option for "no opinion." You also need to ask them the same questions about a competing company or organization, so you have comparable data on the competition to compare to.

Step 6: Send out the survey.

Step 7. Resend the survey.
Depending on your audience it may take several tries and an incentive to get sufficient responses (I'll do just about anything for an Amazon or Starbucks card). How many is sufficient? Well, it depends on how you plan to break down your analysis, but in general plan to resurvey until you get at least a 10% return. If in doubt, talk to your local survey expert.

Step 8. Analyze and learn from the results.
Calculate a mean score for each relationship component. There are both positive and negative statements in the survey instrument, so make sure you take that into account. Compare your mean on each score to the competition. (And of course to your earlier survey results, if this survey is not the benchmark.)

Step 9. Implement your program and measure again.
If this is your pre-social media program (benchmark) survey, then implement your program now, and measure again in six months. Or, if your program has been running for a while and your analysis indicates you need to make changes, then make the changes now and let them work on your audience for six months before you measure again.

Good luck, and let me know how things go.

Should We Trust Web-Based Studies? (This research says Yes.)

This comparative study of Web-based survey samples and paper-and-pencil surveys (by Samuel D. Gosling, Simine Vazire, Sanjay Srivastava and Oliver P. John and which appeared in American Psychologist) is now four years old, but its conclusions are probably still valid: Yes, Web-based self-report and self-selected samples can be trusted for surveys. From the study's summary:

"Internet data collection methods, with a focus on self-report questionnaires from self-selected samples, are evaluated and compared with traditional paper-and-pencil methods. Six preconceptions about Internet samples and data quality are evaluated by comparing a new large Internet sample (N
361,703) with a set of 510 published traditional samples. Internet samples are shown to be relatively diverse with respect to gender, socioeconomic status, geographic region, and age. Moreover, Internet findings generalize across presentation formats, are not adversely affected by nonserious or repeat responders, and are consistent with findings from traditional methods."

Get the whole study here.

The Interpretation of Dreams: Dreams as Public Relations Measurement -- I Dream of Hillary, I Dream of Barack

Many public relations professionals find that media analysis or survey research are tools sufficient to measure their programs. The more adventurous bring in external data and more esoteric measures of success like donations or number of new memberships or even lives saved. In politics, KDPaine and Partners has recently used YouTube video counts to predict primary results, and our Ms. Paine has ruminated on the possibility of using the number of political lawn signs as an election predictor. Not until now, however, has anyone used dreams to measure political progress.

In this week's New Yorker, Ben McGrath writes about Sheila Heti and her new blog "I Dream of Hillary... I Dream of Barack." This blog, a repository of reader-submitted dreams about the candidates, can be interpreted as a rough poll of interest in those candidates. As Mr. McGrath says, "...what if the recurrences of Presidential candidates in people's dreams were meaningful in the aggregate?" As of the writing of the article, "..Obama's edge in the over-all dream count... was roughly equivalent to his lead in the latest Gallup poll."

Ms. Heti's blog has been so successful she plans to open an I Dream of McCain section, so in the future we will have bipartisan dream data to go on. --Bill Paarlberg, Editor, The Measurement Standard

KDPaine & Partners Releases New Book on Measuring PR and Corporate Communications

Your Measurement Reading List

Measuring Public Relationships Makes Measurement Simple

Measuring Public Relationships, The Data-Driven Communicator's Guide to Success by Katie Delahaye Paine, 228 pages, paperback. ISBN 978-0-9789899-0-3. Order online or by phone: 603-319-1047.

BERLIN, NH, Dec 12, 2007-- The first how-to book on measuring relationships for public relations professionals was released today by KDPaine & Partners, the Berlin, NH-based leader in social media and public relations measurement. Measuring Public Relationships, The Data-Driven Communicator's Guide to Success was written by KDPaine & Partners' CEO, Katie Delahaye Paine.

Paine takes a "cookbook" approach, providing specific steps to measure all forms of public relationships, from social media measurement to tracking relationships with local communities, industry analysts and social networks. It relies heavily on the relationship theories of James and Laurie Grunig and Linda Hon but translates the theory into an easy to read practical guidebook. The book has been well received by academics and professionals alike who have called it "the must-have practical guide to hands-on PR measurement." Emphasizing the role and evaluation of relationships, this book provides every public relations professional with step-by-step research procedures for measuring programs, improving results, and managing relationships.

"It really is an excellent book," said Professor James Grunig, calling it "lively, engaging, accessible, wise, and candid... a remarkable compendium..."

"Katie Paine is the most practical consultant and educator on public relations research that I know. She is brave enough to link PR to return on investment demands and smart enough to provide marketers and PR professionals with a wide range of useful research tools. If there is cost effective way to do research, Katie Paine will tell you about it," said Clark Caywood of the Department of Integrated Marketing Communications at the Medill School at Northwestern University.

"By force of prose and experience, Katie Paine level-sets the practical and now critical disciplines of media and communication measurement. A student of Grunig theory and a champion of the almighty relationship, Paine plops on every communicator's desk the best new reference for measuring the intangible. Read it, and you'll count your successes," said Alan Kelly, CEO & Founder, The Playmaker's Standard, LLC and author of The Elements of Influence. Adding that it's "...the best new reference for measuring the intangible."

Measuring Public Relationships is available at a 25% discount until January 31st, 2008 when ordered directly online or by calling the KDPaine & Partners' office at 603-319-1047.

How To Evaluate Events and Sponsorships

Sponsorship Measurement

Six basic steps, five final lessons and four additional resources.

By Katie Delahaye Paine

"Live from the KDPaine & Partners/Verizon Wireless Arena we have the KDPaine & Partners' Battle of the Sponsorship Suppliers, underwritten by Orville Redenbacher's Microwave Popcorn, the official snack food of KDPaine & Partners; by Starbucks, the official fuel of KDPaine & Partners; and with additional support from Staples, the official office supply company; FedEx, the official overnight delivery system; Diet Coke, the official diet drink; and Catalina, the official pet of KDPaine & Partners."

Perhaps not these exact words, but something like them has passed your ears at least a dozen times in the last few years. From Little League teams to the World Cup, from naming arenas to naming road races, sponsorship of events is now one of the marketing methods of choice. As communications is increasingly impersonal and electronic, the chance to actually touch, feel and experience the brand is ever more important. Increasingly, purchasing is neatly divided into two categories -- on-line and experiential. More and more decisions are based on familiarity with the brand and ease of purchase.

Now, I'm an avid Amazon fanatic, but I still go into Borders' brick and mortar store and end up buying something. Why? Because I get to touch and feel and browse and listen to products. If I need something in a hurry, I go online. If I want "retail therapy," I go to Borders. That's experiential marketing. Marketing that transcends the product and the media and allows direct interaction with the brand.

A 2006 study by Jack Morton Worldwide showed that live event marketing -- experiences where consumers interact with products, brands or "brand ambassadors" face-to-face -- are among the most effective ways to influence coveted consumer audiences. The study, an online survey of 2,574 consumers ages 13-65 in the top 25 US markets, confirmed that this increasingly important marketing medium resonates strongly across all demographic and product categories.

Which is why total spending on experiential marketing is expected to reach $15 billion by the end of 2007, an 11% increase over last year, according to the latest IEG research report. Companies are now allocating on average 17% of their budgets to sponsorships, compared to just 13% last year. However, 76% of companies spend less than 1% of their total sponsorship budgets on research, and in a shaky economy, if you don't measure it, you can't show ROI. And if you can't show ROI, chances are good that the program will be killed.

Another factor in the measurement of experiential marketing is that simply "reaching large numbers of eyeballs" is no more relevant a goal than is "increasing market share in the buggy whip market." Therefore, new alternative metrics -- like relationship measures, engagement indices and cost per touch -- are increasingly demanded.

We suggest that there are six basic steps to developing a solid measurement program for your events and sponsorships.

Step 1: Be clear about what outcomes your communications program is designed to achieve.

As with any effort, you can't start to measure success until you know what success means for you. For any given event the objectives might be:

• Percent of attendees more likely to purchase
• Percent of attendees remembering the brand
• Number of qualified sales leads generated
• Conversion rate of attendees
• Total potential sales = [number of attendees] x [conversion rate] x [average sale]
• For press events: number of key editors and analysts attending, percent of attendees writing or quoted on the issue, total exposure of key messages in resulting press.

Each objective, of course, requires a different type of measurement. Some types isolate the impact of PR from other communications efforts better than others. And the best objectives are specific and measurable.

Step 2: Determine criteria -- quality as well as quantity

Once you've agreed upon your objectives, establish the specific criteria of success that you will measure. If your objective is awareness, the criterion might be the percentage increase of unaided awareness of brand or product. If your objective is to sell product, the criterion might be the incremental sales after a particular PR or promotional program took place. Consider those numbers that best reflect the health of your business, or that best represent characteristics that most affect your business:

• Increase in awareness
• Increase in preference
• Increase in purchase intent
• Increase in customer loyalty
• Percent improvement in customer experience

To a certain extent, your choice of criteria is dependent on the type of event you are evaluating. If the customer experience you are measuring takes place at a trade show or exhibit booth, you might choose as key criteria the percentage of new visitors or the cost per minute spent with a client in the booth. If the experience takes place at an event such as a concert, you need to count how many people were exposed to your brand or the brand experience you were offering. Don't trust the promoter's numbers; do your own counting. Here's why:

I once met with a major auto manufacturer who wanted to measure the effectiveness of different sponsorships. It was spending many millions sponsoring car races, golf tournaments, antique car auctions and a variety of other events. According to the event organizers, attendance and therefore results were always better than the year before. But then I asked them what they were trying to achieve by these efforts. The first response was essentially: "We are always a major sponsor of these types of events." After about two hours of discussion, we agreed that the business objective was to drive potential customers into dealer showrooms.

We began a series of surveys at each event to determine first if the attendees remembered the sponsors, and secondly if they were more or less likely to test drive and/or buy the sponsor's car. We collected names at the event itself and called attendees two weeks after the event. Our results showed that on average 50% of all attendees were more likely to test drive a "sponsor" vehicle after attending the event.

After we had measured three different events, we were able to compare and contrast the cost effectiveness ($$ per person reached) of different events. The results enabled the sponsor to better understand the ROI from each event. By looking at the percent of people more likely to go to a dealership, they could determine the number of potential buyers. By subtracting the cost of the sponsorship from the profit and the projected number of car sales, they determined a projected ROI from the event. These results led the sponsor to dramatically alter their sponsorship strategies.

It's important to look beyond simple quantitative data, especially if your objective is exposure, to assess the quality of your communication. When companies clearly define their objectives it becomes a relatively simple matter to define a set of criteria against which to measure the relationships the company establishes with its constituencies in different events. Then it's a matter of comparing and contrasting results consistently cross a number of different events to ensure that the company has the most effective sponsorship program possible.

Step 3: Decide upon a benchmark

The key point to remember about any evaluation program is that measurement is a comparative tool. You need to compare one set of results to something else. The most meaningful comparisons are between different events, or between you and your peer sponsors at one event. If your fellow sponsors aren't direct competitors you can learn a lot by sharing information and determining what best practices really moved the needle.

4. Select a measurement instrument

Do not rely on data gathered at the event itself. It is rarely useful or actionable. Consider the following instead:

• Post-show awareness survey
  In our opinion, the most reliable way to measure relationships with your customers is to collect names at the event and interview them afterwards to see what they can remember of about the experience. Do they remember being in your booth? Did they even know you were a sponsor? Did they remember your brand, and the brand benefits or brand positioning you were trying to convey? Did they leave more likely to purchase or to recommend?
• Web analytics
  Because customer actions are so easily trackable from website activity, more and more companies are relying on web analytics to measure the impact of sponsorships. Determining not just the increase in traffic but the actual conversion rate is a relatively easily task using Google Analytics, WebTrends or Visual Sciences.
• Cost per touch
  As a way of comparing different marketing opportunities, some organizations are replacing the old "CPM" or cost per contact with Cost per Touch. In other words what does it cost to make contact (by phone, email, in person, or however you define it for your research) with one potential customer?
• Cost per minute spent with prospect
  If the purpose of the event or sponsorship is to collect sales leads you may want to calculate the value of your effort in monetary terms. If you have a product or service that has a long sales cycle and needs time to explain, this is a particularly good way to evaluate marketing efforts. Several years ago, the pharmaceutical company Glaxo figured out that it cost them $300 to get a salesperson into a doctor's office for about five minutes. That's $60 per minute spent in front of a prospect. Now suppose that by giving a speech you get 60 minutes in front of a qualified audience of 100 people. That's $0.83 per minute spent with each potential prospect. Relatively, this appears to be very efficient.
  
  Once you have expressed it in that form, you can compare value to other marketing efforts. So, for example, if a 20-second underwriting spot on NPR costs $5,000, and assuming it communicates your key message and reaches 500,000 listeners, that's $.01 per opportunity to hear a key message and $0.03 per minute spent with a prospect.
  
  You should also monitor the media coverage around the event to see if additional messages are communicated to your audience via the media surrounding the event. You need to look at the reach of the publication and also determine whether the article contained a key message. You can then take the cost of the program and divide it by the total number of opportunities to see or hear a key message to determine the cost per opportunity to see a key message. That way you can decide if the event is more or less efficient than other programs at getting your messages across and achieving your goals

A relationship study

You can also use the Grunig relationship survey instrument (downloadable for free from the IPR web site) to test the health and strength of your customers' relationships to your brand. The Grunig instrument has been thoroughly tested and shown to be an extremely effective measure of how customers perceive their relationships with an organization.

The key thing about whatever measurement tool you use is to make sure that you are tying results back to the desired business outcome. A number of years ago, Country Music Television (CMT) hired tractor trailers to set up country music events in Wal-Mart parking lots. Their first measure was to ask the truck drivers how many people showed up. Realizing that those answers didn't tell them anything about what those potential customers were actually doing when they went home, CMT decided to hire my company to do follow-up studies with the attendees. The stated purpose of the events was to convince attendees to call their local cable company and request CMT or to take some other action on behalf of CMT. CMT used a sweatshirt giveaway at the events to collect names and addresses of attendees, whom we called two weeks after the event. We agreed that the common criteria against which we should measure all events was an ad hoc "level of engagement." Specifically, we asked people to rate how much the event experience affected their relationship with CMT, and then we established the mean level of interest for each event. We further defined the success criterion as the percentage of people willing to take action on behalf of CMT. Our surveys showed that 93 percent of attendees were willing to take some action and 89% were willing to make a phone call to their local cable company. We then compared results between different cities so CMT could determine where to expand the program the following year.

Five final lessons for sponsorship measurement:

1. You become what you measure, so be very clear internally on the objectives of your sponsorship or event that you will measure.

2. Match the event, the speaker, and the metrics to the objectives.

3. Document any immediate or tangible results: media coverage, phone calls or inquiries, messages delivered to key audience, etc.

4. Share and leverage results internally.

5. Repeat your measurement consistently.

Other resources:

www.instituteforpr.org offers several white papers on measuring events, including:

• Measuring the Effectiveness of Speakers Programs
  By Marianne Eisenmann and Katie Paine
• A Guide for Measuring Event Sponsorships
  By Bruce Jeffries-Fox
• Guidelines For Measuring Relationships in Public Relations
  By Linda Childers Hon and James E. Grunig

My just-published book Measuring Public Relationships includes at least a couple chapters that bear on event measurement. You can purchase the book from the KDP&P's office at 603-319-1047 (25% off the list price of $29.95 until 1/31/08), or you can download the pre- press draft at no charge right here.

Hitting on Guys in Bars and Other Tools of the Trade for Obtaining Customer Feedback

Jenny Schade's
Making It Count

 

Research at the Scene of the Dime

In a sports bar in downtown Chicago, my colleague and I approach a twenty-something guy. I'm doing the talking and she's ready with her clipboard.

"Excuse me," I begin. "Can we speak to you for a few minutes?"

"Yeah, just a second," he responds. He leans over to one of his buddies and whispers, "I can't believe this, we just got here and already chicks are hitting on me!"

He returns to us. "Hi!"

"We'd like to ask you some questions about the video games here,"I explain. "It's market research. Our interview will take about 15 minutes and we'll give you $25 for making time for us."

"Oh," he replies. "Well, I guess that sounds okay. What are your questions?"

Our interview begins:

Q: "Why do you come to this bar in particular?"

A: "My friends and I probably come here two or three nights a week. I don't have to plan anything – it's all here for me. All I have to do is show up."

Q: "Do you ever play the video games they offer here? "Why or why not?"

A: "More often than not, we play video games. It's something to do and after playing one a few times, you can actually get better at it.

Q: "What do you look for in a good video game?"

A: "Something happens on the screen and I have to do the right thing. It makes me feel kind of good when I do well. A couple of weeks ago, I was winning and this one game made all sorts of noise. Pretty soon, I had a crowd gathered around me. It was this game right here – let me show you what I mean…"

His responses exemplify why doing these interviews on-site is so valuable for obtaining practical information for our video game manufacturing client. We've quickly covered a lot of ground in this interview and it's all very important to our client:

• The idea that this sports bar patron is a "regular" and looks to the bar he frequents to provide easy entertainment.
• The fact that he develops a sense of competency as he plays a video game.
• The emotional reward he receives when he interacts with a game and it signals to him and to others that he is succeeding.

When we multiply this single interview times the 30 video game players we will talk with this week, we start to recognize some patterns. Product innovation and marketing recommendations emerge almost effortlessly. For example, our research immediately suggests:

• Developing new games that offer increased interaction and lights/sound acknowledgment of winning players.
• Offering bar promotions that provide patrons with added entertainment and rewards such as video game tournaments and play-offs.
• Capitalizing on players' desire for feeling competent by publicly recognizing winners through a bar "Wall of Fame."

Research At the Scene of the Dime

By being on-site for these interviews, we are able to talk directly to our client's customers and observe them in their natural surroundings – their habitat, so to speak. We're catching them off-guard and encouraging them to share how they use our client's products as well as what they value about them.

This case study about interviewing video game players in sports bars is an example of going directly to your customers and conducting research in the field rather than having them come to a research facility to learn about their opinions. Traditional qualitative market research that takes place in a focus group facility can be very informative, however, it is invaluable to observe and talk with customers of certain kinds of products and services right where they make the purchase decision or, as I like to call it, "Right At the Scene of the Dime."

By talking with customers as they are pulling out their wallets to plug dimes – or dollars – into video games, we can capture their top-of-mind thoughts and emotions, gaining insights into their needs and connections with our products.

Jenny Schade's Three Questions for Determining Interview Location

So how do you identify when conducting research in the field would be advantageous over a more traditional setting? Ask yourself Jenny Schade's Three Questions for Determining Interview Location:

1. Is there a benefit to observing the people we want to interview in the setting where they actually use our products?
2. Would it be easier/more cost effective to go to our respondents rather than trying to recruit them to come talk to us?
3. Do others influence how our customers interact with our products?

In the case of our video game machine producing client, our responses to all three questions were affirmative.

1. It was tremendously beneficial to observe the bar patrons on-site and understand the other attractions that competed for their attention.
2. It was actually quite easy to recruit respondents in the bar, rather than paying for recruits that might not actually show up (probably because they were in the bar that night!).
3. Only by observing our respondents were we able to understand the role their friends and other patrons played in their choice to play video games and keep playing throughout the evening.

Following are examples of other research initiatives that we deemed more appropriate to conduct in the field rather than in a professional research facility:

• Interviewing construction workers at their sites regarding the qualities they seek in the truck boxes that house their tools in the back of their pickup trucks. Doing the interviews at the work sites enabled the workers to show us the truck box features they preferred and respond to product ideas under consideration.
• Running "rap groups"of teens in their after-school clubs to identify the program offerings they found most appealing. Holding the sessions at the schools greatly increased participation and allowed us and our client to understand the situational pressures faced by club members.
• Interviewing bakery employees right on the assembly line as they frosted and put together cookies and cakes. This setting helped us see directly the logistics involved in the workers' responsibilities and understand their reactions to corporate initiatives that they felt were insensitive to their needs. (This was also memorable as the only time I ever conducted interviews while wearing a hair net! Just picture the "The Lucy Show.")

When you really need to see your customers in action, there's nothing as effective as conducting research in the field. Whether that means putting on your construction hat or a hair net, you're sure to gain an in-depth perspective on what motivates your customers to invest financially – and emotionally – in your products.

Jenny Schade is president of JRS Consulting, Inc., a firm that helps organizations build leading brands and efficiently attract and motivate employees and customers. Subscribe to the free JRS newsletter on www.jrsconsulting.net/newsletter.html
© JRS Consulting, Inc. 2007

New Book on How to Measure Relationships is Now in Final Proofing and Available for Download at No Charge

The new book "Measuring Public Relationships" by Katie Delahaye Paine is a practical guide to hands-on public relations measurement. As public relations professionals realize, PR measurement is a multi-disciplinary combination of communications, research and statistical methods, social psychology, and office politics. Katie Paine's new book adds the study of human relationships to the mix—a theme that’s always been present, but that is now far more accessible thanks to the recent development of practical survey techniques for measuring relationships. Katie's book is now almost ready to be printed, and interested readers can download it as a pdf file here.

Great NYTimes Article on Research Difficulties

In yesterday's NYTimes Magazine there's a great article by Gary Taubes on the difficulties of doing epidemiology and medical research that will interest anyone doing public relations measurement or empirical research in general. Although we in public relations research don't have to wrestle with healthy-user bias or placebo effects, there is plenty here on causation vs. correlation and other problems familar to our field. We certainly can empathize with this quote from Jerry Avorn: “The wonder and horror of epidemiology... is that it’s not enough to just measure one thing very accurately. To get the right answer, you may have to measure a great many things very accurately.” --Bill Paarlberg

Sex, Surveys and Faulty PR Measurement!

Sex, surveys and faulty PR measurement: Can an article about public relations research get any more exciting than that? PR measurement depends an awful lot on surveys for its raw data, and "The Myth, the Math, the Sex" (in yesterday's New York Times) shows we can't trust that data. I'm as wicked keen about about this article as I was about Freakonomics. And it's about sex, too, did I mention that? --WTP

Can the Votes of a Panel Measure Trust, Reputation, and Other Tricky Things?

How do you measure trust? I just read about an innovative social media approach as applied to news media, (thanks to PR Watch.org). NewsTrust, now still in beta, is "...a social network model which uses the intellect of the masses to rate all manner of news content and news sources..." So, as I take it, news sources will be rated by many readers to result in an overall score that roughly translates to "trustworthiness."

And that brings up an interesting thought: If you can derive a useful measure of trustworthiness by having a bunch of people just vote on it, then why couldn't we measure all kinds of tricky things by having people vote on them? Could we compare the trustworthiness of companies or politicians just by combining ratings from enough people?

And if so, then why go to all the media analysis effort of compiling a Reputation Index for big companies when you could just get a bunch of people to rate the companies? Hey, maybe there's already a social media site called YourRep.com or something where everyone can rank companies to provide an overall reputation score.

I guess if we can use Wikipedia to provide accurate information on, say, Total Quality Management (which I happen to have looked up there just a little while ago, and I feel more or less confident that what I read was accurate), then perhaps we can use a similar consensus-of-many approach to defining (or at least getting a handle on) more nebulous concepts as well. --WTP

How To Really Mess up Your Public Relations Measurement Program

Measurement Mistakes You Don't Want To Make

How To Really Mess Up Your PR Measurement Program
Ten fatal PR research mistakes and how to avoid them.

Despite the best laid plans, public relations measurement programs can sometimes go awry. You can't always anticipate how everything will go, and your elegant research design rarely seems to play out quite like you planned. Let's face it, unforeseen problems and errors can creep in here and there and part of your job is to figure out how to get the job done anyway.

But there are certain errors your program just won't survive. These mistakes will ruin your data or analysis and leave you with no options but to learn an expensive lesson and to start over. Here are ten fatal research errors that now you won't have to learn about the hard way (and, yes, these will be on the exam):

1. Clipping systems that miss clips
We won't name names, but you should regularly test your provider. Do what we call a "Pub/Month" check: Look back over the stats for the last year and see on average how many articles you get in your key publications. If you are below that for the current month, or if you have zero clips for the month, someone's probably missing your clips.

2. Dirty data from your content provider
This means errors like not differentiating between nytimes.com and The New York Times. Again, check the data on a monthly basis to make sure that it includes what it's supposed to.

3. Bad circulation figures (impressions)
It really doesn't matter if it's off by 10 or even 100. But we've seen cases where providers have moved commas and made the NY Times circulation 14 million instead of 1.4 million. Do a reality check.

4. Corporate articles that end up in product categories and vice versa
This needs to be checked monthly or even weekly for the first six months to make sure that it reflects reality.

5. An unclear definition of tonality
Ask three people what a positive article is and you'll get three different answers. We define it as "leaves the reader more likely to do business with, invest in, or go to work for the company." How you define it is your own business, just make sure it's consistent.

6. An unclear understanding of key messages
Again, do a monthly reality check.

7. Not comparing apples to apples in a competitive analysis
This includes errors like looking at your own local coverage but not the local coverage of your competition.

8. Not being clear about the universe of publications
Make up a written list of search terms as well as a list of the publications/universe to be covered.

9. Not having total control of the names and mailing list for your survey
Beware of merging lists: You can end up with two surveys in one household just because the middle initial is left off one name but not the other.

10. Not being clear about what social media you want to measure
Are you interested in user reviews, Facebook, MySpace, list serves, blogs, or all of the above?

Using Ad Data To Estimate PR's Contribution To Sales

Measurement Tip of the Month

Using Ad Data To Estimate PR's Contribution to Sales

Warning: This technique is a quick and dirty estimate only, see below.

Advertising people sometimes use measurement techniques that give themselves all the credit for sales. The general approach is, "We created X ad impressions this month, and then the next month the company sold Y widgets, so now we know that each ad impression will result in Y/X sales." Crude? Yes, but when done year after year with products that are sold in large numbers (toothpaste or cat food, for instance) this approach can result in useful predictions of how many sales will result from a typical ad campaign.

How could we apply this technique to PR? Let's assume that if earned media (coverage that results from PR efforts) includes photos and key messages, then it ought to get sales results more or less like an ad that includes the same ingredients. So we can use earned media impressions instead of ad impressions in the above measurement technique.

The trick here is that if your consumer company or organization has an ongoing ad budget, you can probably figure out how much media exposure is necessary to generate a certain number of sales or responses. (Your ad department may have those numbers already.) Then use that ratio on your PR impressions. Here's the process step-by-step...

Step 1:
How many impressions did your ads make?
Gather the last two year's worth of advertising media schedules and calculate the reach of your ads (impressions) on a quarterly basis.

Step 2:
How have impressions influenced sales?
Gather the last two years worth of sales figures and use them to calculate, for each quarter, the overall ratio between reach and product sold (or whatever response you are trying to get). That is, how many ad impressions does it take to sell one unit of product? (Your ad department may have already done Steps 1 & 2 for you.)

Step 3:
How many impressions did your PR make?
Determine how many impressions you've generated via PR that contain the same message elements (brand benefits, recommendations, favorable positioning) that advertising includes.

Step 5:
What is PR's contribution to sales?
Now use the ratio (do it quarter by quarter, or take the average over all the quarters you had data for) from Step 2 to estimate how many sales you have influenced with your PR: Multiply the number of PR impressions times the ratio of sales to ad impressions:
PR Impressions x (Sales/Ad Impressions) =

Sales from PR, approximately

Warning: This is a quick and easy exercise that will result in a crude estimate only. It's fun to do if you can get the necessary data easily, but don't go thinking you've found the Holy Grail, or spending time on this that you ought to be spending on more valid measurement techniques.

If you are really excited by this sort of analysis, there are far more sophisticated ways to do it. Take a look at this post on the metricsman blog, and then look up "marketing mix modeling" if you still want more.

Variability of Human Judgements Illustrated by US Immigration Courts: Does your media analysis vendor do intercoder reliability assessment?

Today's NYTimes has an article about the huge disparities in U.S. immigration courts, with some surprising data on how widely asylum decisions vary between court locations and between judges within the same court. In a system that is meant to apply uniform judgements across cases, there is some serious variablity in decision-making: "...Colombians had an 88 percent chance of winning asylum from one judge in the Miami immigration court and a 5 percent chance from another judge in the same court." And: "...someone who has fled China in fear of persecution and asks for asylum in immigration court in Orlando, Fla., has an excellent — 76 percent — chance of success, while the same refugee would have a 7 percent chance in Atlanta."

We mention this here as a warning to those who do PR measurement and depend on human judgements for their data or analysis. Suppose, for instance, you were doing media content analysis and your human coders had as much variablity as those judges? Good luck getting useful results there.

Of course, everyone doing media analysis guards against that sort of bias by doing intercoder reliability assessment, right? Right?

Wrong, apparently. In fact, CARMA claims that: "CARMA Asia Pacific is the only commercial media analysis firm that carries out intercoder reliability assessment."

Doesn't that make you wonder about the reliability of your data? --WTP

The Lure of Being "Just A Little Bit Bad:" How to Avoid the Seven Deadly Sins of Research

Jenny Schade's
Making It Count

 

The Lure of Being
"Just a Little Bit Bad":
How to Avoid the Seven Deadly Sins of Research

The research business can be precarious at times. Studies have to be completed at breakneck speed, clients push for particular outcomes and trendy colleagues promote the latest fads. Or perhaps it's difficult to muster enthusiasm for yet another presentation.

In such situations, it is critical to avoid the lure of being "just a little bit bad" and conducting mediocre research. The following are Jenny Schade's Seven Deadly Sins of Research: