The Measurement Standard: Blog Edition

Research Bias: Another example of why you must consider the source of research.

Biased research results can creep into even the most carefully designed and executed studies. That's why medical research is often double-blind, so, hopefully, the experimenter's bias can't affect the results. In public relations research, bias is endemic because most every measurement project may affect someone's job or budget, and every measurement vendor wants their results to justify their fees.

So here's a review of cell-phone health research to remind all of us just how bias can affect results ("Source of Funding and Results of Studies of Health Effects of Mobile Phone Use: Systematic Review of Experimental Studies" by Anke Huss, Matthias Egger, Kerstin Hug, Karin Huwiler-Müntener, and Martin Röösli.) (Or see a summary here.) It analyzed research on the biologic effects of cell phone use, and found that industry-funded studies were far less likely to identify negative consequences than studies funded by governments and non-profits. Researchers analyzed 57 studies that appeared in the academic literature between 1995 and 2005. Only a third of the industry-funded studies identified a biologic effect with possible health consequences from exposure to cell phone radio waves, while 82% of the government- or non-profits-funded studies found such effects, as did 77% of the studies whose funding source was not identified.

How can public relations measurement identify and reduce research bias? --WTP

Taking Public Relations Measurement to the Next Step

Measurement Strategy

 

Examine the events, actions, strategies and tactics that drive coverage and conversations.

by Katie Delahaye Paine

Having been at this measurement stuff for some 22 years now, I find it remarkably refreshing and rewarding that more and more communications managers are actually measuring results, as opposed to just counting clips. Yes, the vast majority of folks out there are still just counting, but more and more I'm hearing people talk about taking measurement to the next level.

For me, that next level is deriving more meaning from the data. In other words, it's not enough to just look at a trend chart and say, "There's a big spike in June." You need to look at the variables that caused that big spike, or that big slump.

The variable might be the activities that a PR department or agency has undertaken, or it might be the actions or strategies of the competition, or it might be external news events that take over the news hole or the conversation. The point is that for true measurement to occur, you need to be taking these things into account.

That doesn't mean obsess over them so much you don't act. Too often I've had clients get into "Variable Paralysis," a condition in which they are so concerned with accounting for every possible variable, that they don't measure anything.

How to Pick the Right Variables

The one things we at KDP&P have learned is that variables can be specific to the market space or industry you are in. For example:

• In the non-profit sector, the ultimate goal for communications is typically to either generate donations, recruit volunteers, build membership, or spread a message. The variables we track are typically events, -- major high visibility programs or initiatives that drive media coverage and ultimately drive Web traffic and membership or donations. What's important to remember about using events as a variable is that the competition is also doing similar events and programs, and you need to account for them as well.
• In the business-to-business space, the end result may be to change positioning or reputation. Is perception changing over time? Are more of your messages getting out there? In which case the variables are more people- and product-based. What thought leaders are being quoted? What products or programs are being discussed? And are we getting into that conversation as often as the competition? So you're not just looking at the volume of coverage, but of the visibility of that coverage, how it positions your organization, and who is framing the discussion.
• In the educational marketplace variables tend to focus on news events. Whatever is hot in the traditional news cycle tends to drive actions in academia. So when a bridge collapses in Minnesota, engineering departments around the country are tapped for their expertise. Or if alternative energy becomes a major news focus, professors and universities who have published books or papers on the topic become widely quoted. So a key variable in academia is the degree to which proactive PR efforts in response to uncontrolled media events have impacted your visibility.

The point in all of this is that we all know how to monitor and count our coverage. Many of you also know how to count and monitor conversations going on amongst your customers. But now we have to look at the events, actions, strategies and tactics that are driving those conversations.

Pornography, Public Relations Measurement and Anytown, USA

There are few things in life as exciting as solving a measurement problem with the innovative use of some new source of data. Followers of this blog and The Measurement Standard newsletter probably share our enthusiasm about Freakonomics (see our review here, and another example on beauty premiums here), and know how much we enjoy coming up with new ways of gaining insight from data. (Heck, it is in large part the theme of Katie Delahaye Paine's new book Measuring Public Relationships, and is the topic of a couple of her recent blog posts: "Another alternative to the Ad Value nightmare" and her Optimum Content Score proposal for media measurement. [She is the publisher of The Measurement Standard.])

So we encourage you to head right over and read an article in today's NY Times about using Google Trends data to demonstrate community standards in a pornography trial: "What’s Obscene? Google Could Have an Answer." It seems that these legal proceedings often face difficulty in establishing just what is or isn't obscene. And, wouldn't you know, Google has data on that: if Anytown, USA, searches as often for "orgy" as it does for "apple pie," then maybe some conclusions can be made about community standards. Or maybe not. Read the article. --Bill Paarlberg, editor

Measuring Naked Relationships: How to evaluate the impact of a social media program on your public relationships.

Your step-by-step guide to using relationship metrics to evaluate the success of your social media program.

by Katie Delahaye Paine

I've talked a lot here -- also in my speeches, in my blog, and of course in my recently published book Measuring Public Relationships -- about the importance of measuring relationships. I believe that without factoring in the impact that your social media program has on your relationships (with employees, community and constituencies) you are undervaluing your efforts.

So, how does one actually measure relationships? Well, now that you asked, my book explains this with regard to most types of public relations programs. Social media being all the rage right now, I thought it would be appropriate to provide a step-by-step guide as to how to actually do it for social media programs.

Whatever type of program you want to measure, the basic technique is similar: You conduct a survey of your audience using a special set of questions designed to specifically measure the different components of relationships. You do this before and after your social media program is in place, and you do it for your organization and as many competing organizations as you can afford to. Then you compare the data before and after, and between your organization and the others, and then you know where your relationship with your audience stands and where you need to go.

The Grunig/Hon Relationship Research

Before we get to the nuts and bolts, here's a bit of background. A decade or so ago, University of Maryland Professors Jim and Laurie Grunig and Linda Hon synthesized communications and sociology research and theory into a paper published by the Institute for PR called "Guidelines for Measuring Relationships in Public Relations." Their feeling was that, amidst all the brouhaha about quantifying public relations, we were forgetting the essential truth that PR was about relationships. And so you needed to measure the impact that your efforts were having on those relationships.

Their research isolated six fundamental components of relationships -- Trust, Satisfaction, Commitment, Control Mutuality and Exchange/Communal -- and they designed and tested 75 statements to measure those components. These statements are of the sort: "This organization treats people like me fairly and justly," and, "I feel a sense of loyalty to this organization." You can see them all, and copy them for your own use, on my blog or in the Grunig/Hon paper. For an example of these statements successfully used to measure public relationships in a non-social media context, see this research by Forrest W. Anderson and Paul Raab.

Nine Steps to Measure Your Social Media Relationships

Before we start, remember that to isolate the effect of your social media program, you must begin your measurement before you launch that program. Then you'll have a benchmark to compare against later: Before Social Media vs. After Social Media. Without a Before benchmark you won't know how your social media program has changed your relationships.

Of course, you can always begin measuring after your social media program has started, and by doing so you will be able to do ongoing evaluation of your relationships. Which is a good thing. But a better thing is to isolate the effect of social media, and to do that you must compare before and after.

So use Steps 1 through 9 below to survey your audience before your program begins, and thus establish your benchmark. Then begin your social media program, and after an appropriate time, say three or six months (depending on your situation), re-survey to see what has changed.

To be sure that whatever change you find is solely attributable to the social media program, you must hold constant any other PR programs that affect your social media audience's perception of your organization. Yes, it's tricky, and it's not always an ideal world. But if you are trying to measure the effect of your new blog at the same time as the Promotions Department decides to give away A Free Cadillac With Every Purchase, then you can kiss your results goodbye.

Step 1: Define the audience for your social media program.
Social media is about conversation and engagement, so decide with whom you want to converse and engage. If you're starting an internal blog, your audience is your employees. If you're starting an IdeaStorm-type customer community, your audience is anyone likely to buy or recommend your product. If your mission is advocacy, your audience might be voters, politicians or industry influencers.

Step 2: Get a list of your audience's email addresses and/or phone numbers.
To get a representative sample you will need at least 500 names for each organization you are testing (more on that later). If you already have a list of your members, subscribers or customers, then you are ahead of the game.

If you have to purchase your list, then potential vendors vary with the type of sample you're looking for (mail, phone, web). Most lists are sorted based on demographic or title data. There are a lot more resources out there for mail addresses and those resources do not necessarily need to be survey sample companies. For email addresses, some reputable firms include Survey Sampling, e-Rewards and Zoomerang.

Step 3: Pick a survey methodology.
The Grunigs would recommend in-person surveys for the best results, but most researchers find that to be very expensive. Phone surveys are fast and provide very accurate results, but again, depending on the audience, may be cost-prohibitive. Email surveys are an increasingly accepted methodology, and for social media can be appropriate and highly reliable, since, presumably, your audience is all on email.

You may be able to piggyback on an existing survey going out to your community. If marketing, customer satisfaction, business development or anyone else in your organization is doing a survey, see if you can add a few of the Grunig/Hon statements to it.

Step 4: Select which of the Grunig/Hon statements are most appropriate to your organization.
You can probably only impose on someone for 7-10 minutes of their time, so you need to pick which statements you will include. Grunig and Hon suggest that if you want to shorten the survey, you use only the boldfaced items. Not all statements are appropriate for all organizations, so pick and chose the ones that will be most meaningful to your audience.

Step 5: Prepare your survey.
If you are using an electronic survey system like Survey Monkey, you need to create an introductory screen that explains what you are doing and how the scoring works. For instance:

In order to better understand the needs and perceptions of our marketplace, we'd like to ask you some questions. Please tell us whether you agree or disagree with the following statements as they apply to X company/organization.

Explain that 1 means "totally disagree" and 7 means "completely agree," and give them an option for "no opinion." You also need to ask them the same questions about a competing company or organization, so you have comparable data on the competition to compare to.

Step 6: Send out the survey.

Step 7. Resend the survey.
Depending on your audience it may take several tries and an incentive to get sufficient responses (I'll do just about anything for an Amazon or Starbucks card). How many is sufficient? Well, it depends on how you plan to break down your analysis, but in general plan to resurvey until you get at least a 10% return. If in doubt, talk to your local survey expert.

Step 8. Analyze and learn from the results.
Calculate a mean score for each relationship component. There are both positive and negative statements in the survey instrument, so make sure you take that into account. Compare your mean on each score to the competition. (And of course to your earlier survey results, if this survey is not the benchmark.)

Step 9. Implement your program and measure again.
If this is your pre-social media program (benchmark) survey, then implement your program now, and measure again in six months. Or, if your program has been running for a while and your analysis indicates you need to make changes, then make the changes now and let them work on your audience for six months before you measure again.

Good luck, and let me know how things go.

Should We Trust Web-Based Studies? (This research says Yes.)

This comparative study of Web-based survey samples and paper-and-pencil surveys (by Samuel D. Gosling, Simine Vazire, Sanjay Srivastava and Oliver P. John and which appeared in American Psychologist) is now four years old, but its conclusions are probably still valid: Yes, Web-based self-report and self-selected samples can be trusted for surveys. From the study's summary:

"Internet data collection methods, with a focus on self-report questionnaires from self-selected samples, are evaluated and compared with traditional paper-and-pencil methods. Six preconceptions about Internet samples and data quality are evaluated by comparing a new large Internet sample (N
361,703) with a set of 510 published traditional samples. Internet samples are shown to be relatively diverse with respect to gender, socioeconomic status, geographic region, and age. Moreover, Internet findings generalize across presentation formats, are not adversely affected by nonserious or repeat responders, and are consistent with findings from traditional methods."

Get the whole study here.

The Interpretation of Dreams: Dreams as Public Relations Measurement -- I Dream of Hillary, I Dream of Barack

Many public relations professionals find that media analysis or survey research are tools sufficient to measure their programs. The more adventurous bring in external data and more esoteric measures of success like donations or number of new memberships or even lives saved. In politics, KDPaine and Partners has recently used YouTube video counts to predict primary results, and our Ms. Paine has ruminated on the possibility of using the number of political lawn signs as an election predictor. Not until now, however, has anyone used dreams to measure political progress.

In this week's New Yorker, Ben McGrath writes about Sheila Heti and her new blog "I Dream of Hillary... I Dream of Barack." This blog, a repository of reader-submitted dreams about the candidates, can be interpreted as a rough poll of interest in those candidates. As Mr. McGrath says, "...what if the recurrences of Presidential candidates in people's dreams were meaningful in the aggregate?" As of the writing of the article, "..Obama's edge in the over-all dream count... was roughly equivalent to his lead in the latest Gallup poll."

Ms. Heti's blog has been so successful she plans to open an I Dream of McCain section, so in the future we will have bipartisan dream data to go on. --Bill Paarlberg, Editor, The Measurement Standard

KDPaine & Partners Releases New Book on Measuring PR and Corporate Communications

Your Measurement Reading List

Measuring Public Relationships Makes Measurement Simple

Measuring Public Relationships, The Data-Driven Communicator's Guide to Success by Katie Delahaye Paine, 228 pages, paperback. ISBN 978-0-9789899-0-3. Order online or by phone: 603-319-1047.

BERLIN, NH, Dec 12, 2007-- The first how-to book on measuring relationships for public relations professionals was released today by KDPaine & Partners, the Berlin, NH-based leader in social media and public relations measurement. Measuring Public Relationships, The Data-Driven Communicator's Guide to Success was written by KDPaine & Partners' CEO, Katie Delahaye Paine.

Paine takes a "cookbook" approach, providing specific steps to measure all forms of public relationships, from social media measurement to tracking relationships with local communities, industry analysts and social networks. It relies heavily on the relationship theories of James and Laurie Grunig and Linda Hon but translates the theory into an easy to read practical guidebook. The book has been well received by academics and professionals alike who have called it "the must-have practical guide to hands-on PR measurement." Emphasizing the role and evaluation of relationships, this book provides every public relations professional with step-by-step research procedures for measuring programs, improving results, and managing relationships.

"It really is an excellent book," said Professor James Grunig, calling it "lively, engaging, accessible, wise, and candid... a remarkable compendium..."

"Katie Paine is the most practical consultant and educator on public relations research that I know. She is brave enough to link PR to return on investment demands and smart enough to provide marketers and PR professionals with a wide range of useful research tools. If there is cost effective way to do research, Katie Paine will tell you about it," said Clark Caywood of the Department of Integrated Marketing Communications at the Medill School at Northwestern University.

"By force of prose and experience, Katie Paine level-sets the practical and now critical disciplines of media and communication measurement. A student of Grunig theory and a champion of the almighty relationship, Paine plops on every communicator's desk the best new reference for measuring the intangible. Read it, and you'll count your successes," said Alan Kelly, CEO & Founder, The Playmaker's Standard, LLC and author of The Elements of Influence. Adding that it's "...the best new reference for measuring the intangible."

Measuring Public Relationships is available at a 25% discount until January 31st, 2008 when ordered directly online or by calling the KDPaine & Partners' office at 603-319-1047.

How To Evaluate Events and Sponsorships

Sponsorship Measurement

Six basic steps, five final lessons and four additional resources.

By Katie Delahaye Paine

"Live from the KDPaine & Partners/Verizon Wireless Arena we have the KDPaine & Partners' Battle of the Sponsorship Suppliers, underwritten by Orville Redenbacher's Microwave Popcorn, the official snack food of KDPaine & Partners; by Starbucks, the official fuel of KDPaine & Partners; and with additional support from Staples, the official office supply company; FedEx, the official overnight delivery system; Diet Coke, the official diet drink; and Catalina, the official pet of KDPaine & Partners."

Perhaps not these exact words, but something like them has passed your ears at least a dozen times in the last few years. From Little League teams to the World Cup, from naming arenas to naming road races, sponsorship of events is now one of the marketing methods of choice. As communications is increasingly impersonal and electronic, the chance to actually touch, feel and experience the brand is ever more important. Increasingly, purchasing is neatly divided into two categories -- on-line and experiential. More and more decisions are based on familiarity with the brand and ease of purchase.

Now, I'm an avid Amazon fanatic, but I still go into Borders' brick and mortar store and end up buying something. Why? Because I get to touch and feel and browse and listen to products. If I need something in a hurry, I go online. If I want "retail therapy," I go to Borders. That's experiential marketing. Marketing that transcends the product and the media and allows direct interaction with the brand.

A 2006 study by Jack Morton Worldwide showed that live event marketing -- experiences where consumers interact with products, brands or "brand ambassadors" face-to-face -- are among the most effective ways to influence coveted consumer audiences. The study, an online survey of 2,574 consumers ages 13-65 in the top 25 US markets, confirmed that this increasingly important marketing medium resonates strongly across all demographic and product categories.

Which is why total spending on experiential marketing is expected to reach $15 billion by the end of 2007, an 11% increase over last year, according to the latest IEG research report. Companies are now allocating on average 17% of their budgets to sponsorships, compared to just 13% last year. However, 76% of companies spend less than 1% of their total sponsorship budgets on research, and in a shaky economy, if you don't measure it, you can't show ROI. And if you can't show ROI, chances are good that the program will be killed.

Another factor in the measurement of experiential marketing is that simply "reaching large numbers of eyeballs" is no more relevant a goal than is "increasing market share in the buggy whip market." Therefore, new alternative metrics -- like relationship measures, engagement indices and cost per touch -- are increasingly demanded.

We suggest that there are six basic steps to developing a solid measurement program for your events and sponsorships.

Step 1: Be clear about what outcomes your communications program is designed to achieve.

As with any effort, you can't start to measure success until you know what success means for you. For any given event the objectives might be:

• Percent of attendees more likely to purchase
• Percent of attendees remembering the brand
• Number of qualified sales leads generated
• Conversion rate of attendees
• Total potential sales = [number of attendees] x [conversion rate] x [average sale]
• For press events: number of key editors and analysts attending, percent of attendees writing or quoted on the issue, total exposure of key messages in resulting press.

Each objective, of course, requires a different type of measurement. Some types isolate the impact of PR from other communications efforts better than others. And the best objectives are specific and measurable.

Step 2: Determine criteria -- quality as well as quantity

Once you've agreed upon your objectives, establish the specific criteria of success that you will measure. If your objective is awareness, the criterion might be the percentage increase of unaided awareness of brand or product. If your objective is to sell product, the criterion might be the incremental sales after a particular PR or promotional program took place. Consider those numbers that best reflect the health of your business, or that best represent characteristics that most affect your business:

• Increase in awareness
• Increase in preference
• Increase in purchase intent
• Increase in customer loyalty
• Percent improvement in customer experience

To a certain extent, your choice of criteria is dependent on the type of event you are evaluating. If the customer experience you are measuring takes place at a trade show or exhibit booth, you might choose as key criteria the percentage of new visitors or the cost per minute spent with a client in the booth. If the experience takes place at an event such as a concert, you need to count how many people were exposed to your brand or the brand experience you were offering. Don't trust the promoter's numbers; do your own counting. Here's why:

I once met with a major auto manufacturer who wanted to measure the effectiveness of different sponsorships. It was spending many millions sponsoring car races, golf tournaments, antique car auctions and a variety of other events. According to the event organizers, attendance and therefore results were always better than the year before. But then I asked them what they were trying to achieve by these efforts. The first response was essentially: "We are always a major sponsor of these types of events." After about two hours of discussion, we agreed that the business objective was to drive potential customers into dealer showrooms.

We began a series of surveys at each event to determine first if the attendees remembered the sponsors, and secondly if they were more or less likely to test drive and/or buy the sponsor's car. We collected names at the event itself and called attendees two weeks after the event. Our results showed that on average 50% of all attendees were more likely to test drive a "sponsor" vehicle after attending the event.

After we had measured three different events, we were able to compare and contrast the cost effectiveness ($$ per person reached) of different events. The results enabled the sponsor to better understand the ROI from each event. By looking at the percent of people more likely to go to a dealership, they could determine the number of potential buyers. By subtracting the cost of the sponsorship from the profit and the projected number of car sales, they determined a projected ROI from the event. These results led the sponsor to dramatically alter their sponsorship strategies.

It's important to look beyond simple quantitative data, especially if your objective is exposure, to assess the quality of your communication. When companies clearly define their objectives it becomes a relatively simple matter to define a set of criteria against which to measure the relationships the company establishes with its constituencies in different events. Then it's a matter of comparing and contrasting results consistently cross a number of different events to ensure that the company has the most effective sponsorship program possible.

Step 3: Decide upon a benchmark

The key point to remember about any evaluation program is that measurement is a comparative tool. You need to compare one set of results to something else. The most meaningful comparisons are between different events, or between you and your peer sponsors at one event. If your fellow sponsors aren't direct competitors you can learn a lot by sharing information and determining what best practices really moved the needle.

4. Select a measurement instrument

Do not rely on data gathered at the event itself. It is rarely useful or actionable. Consider the following instead:

• Post-show awareness survey
  In our opinion, the most reliable way to measure relationships with your customers is to collect names at the event and interview them afterwards to see what they can remember of about the experience. Do they remember being in your booth? Did they even know you were a sponsor? Did they remember your brand, and the brand benefits or brand positioning you were trying to convey? Did they leave more likely to purchase or to recommend?
• Web analytics
  Because customer actions are so easily trackable from website activity, more and more companies are relying on web analytics to measure the impact of sponsorships. Determining not just the increase in traffic but the actual conversion rate is a relatively easily task using Google Analytics, WebTrends or Visual Sciences.
• Cost per touch
  As a way of comparing different marketing opportunities, some organizations are replacing the old "CPM" or cost per contact with Cost per Touch. In other words what does it cost to make contact (by phone, email, in person, or however you define it for your research) with one potential customer?
• Cost per minute spent with prospect
  If the purpose of the event or sponsorship is to collect sales leads you may want to calculate the value of your effort in monetary terms. If you have a product or service that has a long sales cycle and needs time to explain, this is a particularly good way to evaluate marketing efforts. Several years ago, the pharmaceutical company Glaxo figured out that it cost them $300 to get a salesperson into a doctor's office for about five minutes. That's $60 per minute spent in front of a prospect. Now suppose that by giving a speech you get 60 minutes in front of a qualified audience of 100 people. That's $0.83 per minute spent with each potential prospect. Relatively, this appears to be very efficient.
  
  Once you have expressed it in that form, you can compare value to other marketing efforts. So, for example, if a 20-second underwriting spot on NPR costs $5,000, and assuming it communicates your key message and reaches 500,000 listeners, that's $.01 per opportunity to hear a key message and $0.03 per minute spent with a prospect.
  
  You should also monitor the media coverage around the event to see if additional messages are communicated to your audience via the media surrounding the event. You need to look at the reach of the publication and also determine whether the article contained a key message. You can then take the cost of the program and divide it by the total number of opportunities to see or hear a key message to determine the cost per opportunity to see a key message. That way you can decide if the event is more or less efficient than other programs at getting your messages across and achieving your goals

A relationship study

You can also use the Grunig relationship survey instrument (downloadable for free from the IPR web site) to test the health and strength of your customers' relationships to your brand. The Grunig instrument has been thoroughly tested and shown to be an extremely effective measure of how customers perceive their relationships with an organization.

The key thing about whatever measurement tool you use is to make sure that you are tying results back to the desired business outcome. A number of years ago, Country Music Television (CMT) hired tractor trailers to set up country music events in Wal-Mart parking lots. Their first measure was to ask the truck drivers how many people showed up. Realizing that those answers didn't tell them anything about what those potential customers were actually doing when they went home, CMT decided to hire my company to do follow-up studies with the attendees. The stated purpose of the events was to convince attendees to call their local cable company and request CMT or to take some other action on behalf of CMT. CMT used a sweatshirt giveaway at the events to collect names and addresses of attendees, whom we called two weeks after the event. We agreed that the common criteria against which we should measure all events was an ad hoc "level of engagement." Specifically, we asked people to rate how much the event experience affected their relationship with CMT, and then we established the mean level of interest for each event. We further defined the success criterion as the percentage of people willing to take action on behalf of CMT. Our surveys showed that 93 percent of attendees were willing to take some action and 89% were willing to make a phone call to their local cable company. We then compared results between different cities so CMT could determine where to expand the program the following year.

Five final lessons for sponsorship measurement:

1. You become what you measure, so be very clear internally on the objectives of your sponsorship or event that you will measure.

2. Match the event, the speaker, and the metrics to the objectives.

3. Document any immediate or tangible results: media coverage, phone calls or inquiries, messages delivered to key audience, etc.

4. Share and leverage results internally.

5. Repeat your measurement consistently.

Other resources:

www.instituteforpr.org offers several white papers on measuring events, including:

• Measuring the Effectiveness of Speakers Programs
  By Marianne Eisenmann and Katie Paine
• A Guide for Measuring Event Sponsorships
  By Bruce Jeffries-Fox
• Guidelines For Measuring Relationships in Public Relations
  By Linda Childers Hon and James E. Grunig

My just-published book Measuring Public Relationships includes at least a couple chapters that bear on event measurement. You can purchase the book from the KDP&P's office at 603-319-1047 (25% off the list price of $29.95 until 1/31/08), or you can download the pre- press draft at no charge right here.

Hitting on Guys in Bars and Other Tools of the Trade for Obtaining Customer Feedback

Jenny Schade's
Making It Count

 

Research at the Scene of the Dime

In a sports bar in downtown Chicago, my colleague and I approach a twenty-something guy. I'm doing the talking and she's ready with her clipboard.

"Excuse me," I begin. "Can we speak to you for a few minutes?"

"Yeah, just a second," he responds. He leans over to one of his buddies and whispers, "I can't believe this, we just got here and already chicks are hitting on me!"

He returns to us. "Hi!"

"We'd like to ask you some questions about the video games here,"I explain. "It's market research. Our interview will take about 15 minutes and we'll give you $25 for making time for us."

"Oh," he replies. "Well, I guess that sounds okay. What are your questions?"

Our interview begins:

Q: "Why do you come to this bar in particular?"

A: "My friends and I probably come here two or three nights a week. I don't have to plan anything – it's all here for me. All I have to do is show up."

Q: "Do you ever play the video games they offer here? "Why or why not?"

A: "More often than not, we play video games. It's something to do and after playing one a few times, you can actually get better at it.

Q: "What do you look for in a good video game?"

A: "Something happens on the screen and I have to do the right thing. It makes me feel kind of good when I do well. A couple of weeks ago, I was winning and this one game made all sorts of noise. Pretty soon, I had a crowd gathered around me. It was this game right here – let me show you what I mean…"

His responses exemplify why doing these interviews on-site is so valuable for obtaining practical information for our video game manufacturing client. We've quickly covered a lot of ground in this interview and it's all very important to our client:

• The idea that this sports bar patron is a "regular" and looks to the bar he frequents to provide easy entertainment.
• The fact that he develops a sense of competency as he plays a video game.
• The emotional reward he receives when he interacts with a game and it signals to him and to others that he is succeeding.

When we multiply this single interview times the 30 video game players we will talk with this week, we start to recognize some patterns. Product innovation and marketing recommendations emerge almost effortlessly. For example, our research immediately suggests:

• Developing new games that offer increased interaction and lights/sound acknowledgment of winning players.
• Offering bar promotions that provide patrons with added entertainment and rewards such as video game tournaments and play-offs.
• Capitalizing on players' desire for feeling competent by publicly recognizing winners through a bar "Wall of Fame."

Research At the Scene of the Dime

By being on-site for these interviews, we are able to talk directly to our client's customers and observe them in their natural surroundings – their habitat, so to speak. We're catching them off-guard and encouraging them to share how they use our client's products as well as what they value about them.

This case study about interviewing video game players in sports bars is an example of going directly to your customers and conducting research in the field rather than having them come to a research facility to learn about their opinions. Traditional qualitative market research that takes place in a focus group facility can be very informative, however, it is invaluable to observe and talk with customers of certain kinds of products and services right where they make the purchase decision or, as I like to call it, "Right At the Scene of the Dime."

By talking with customers as they are pulling out their wallets to plug dimes – or dollars – into video games, we can capture their top-of-mind thoughts and emotions, gaining insights into their needs and connections with our products.

Jenny Schade's Three Questions for Determining Interview Location

So how do you identify when conducting research in the field would be advantageous over a more traditional setting? Ask yourself Jenny Schade's Three Questions for Determining Interview Location:

1. Is there a benefit to observing the people we want to interview in the setting where they actually use our products?
2. Would it be easier/more cost effective to go to our respondents rather than trying to recruit them to come talk to us?
3. Do others influence how our customers interact with our products?

In the case of our video game machine producing client, our responses to all three questions were affirmative.

1. It was tremendously beneficial to observe the bar patrons on-site and understand the other attractions that competed for their attention.
2. It was actually quite easy to recruit respondents in the bar, rather than paying for recruits that might not actually show up (probably because they were in the bar that night!).
3. Only by observing our respondents were we able to understand the role their friends and other patrons played in their choice to play video games and keep playing throughout the evening.

Following are examples of other research initiatives that we deemed more appropriate to conduct in the field rather than in a professional research facility:

• Interviewing construction workers at their sites regarding the qualities they seek in the truck boxes that house their tools in the back of their pickup trucks. Doing the interviews at the work sites enabled the workers to show us the truck box features they preferred and respond to product ideas under consideration.
• Running "rap groups"of teens in their after-school clubs to identify the program offerings they found most appealing. Holding the sessions at the schools greatly increased participation and allowed us and our client to understand the situational pressures faced by club members.
• Interviewing bakery employees right on the assembly line as they frosted and put together cookies and cakes. This setting helped us see directly the logistics involved in the workers' responsibilities and understand their reactions to corporate initiatives that they felt were insensitive to their needs. (This was also memorable as the only time I ever conducted interviews while wearing a hair net! Just picture the "The Lucy Show.")

When you really need to see your customers in action, there's nothing as effective as conducting research in the field. Whether that means putting on your construction hat or a hair net, you're sure to gain an in-depth perspective on what motivates your customers to invest financially – and emotionally – in your products.

Jenny Schade is president of JRS Consulting, Inc., a firm that helps organizations build leading brands and efficiently attract and motivate employees and customers. Subscribe to the free JRS newsletter on www.jrsconsulting.net/newsletter.html
© JRS Consulting, Inc. 2007

New Book on How to Measure Relationships is Now in Final Proofing and Available for Download at No Charge

The new book "Measuring Public Relationships" by Katie Delahaye Paine is a practical guide to hands-on public relations measurement. As public relations professionals realize, PR measurement is a multi-disciplinary combination of communications, research and statistical methods, social psychology, and office politics. Katie Paine's new book adds the study of human relationships to the mix—a theme that’s always been present, but that is now far more accessible thanks to the recent development of practical survey techniques for measuring relationships. Katie's book is now almost ready to be printed, and interested readers can download it as a pdf file here.