The Measurement Standard: Blog Edition

How Many Completed Questionnaires Does It Take to Measure a Relationship?

While working on last issue's article "Measuring Naked Relationships: Your step-by-step guide to using relationship metrics to evaluate the success of your social media program," a question came up concerning relationship measurement surveys: How many completed questionnaires are sufficient? Suppose you are going to do an email survey of your target audience with the Grunig relationship questions (or see just the questions here on Katie Paine's blog), and you send out 500 surveys. What response rate is enough to be sure your results are significant? Should you keep resurveying your audience until you get a 10% return? A 20% return?

We asked our in-house survey experts, Peter Kowalski and Bruce Aube:

Peter Kowalski
Director of Research Strategy, KDPaine & Partners:  

The typical response rate of an e-mail survey of a respondent set of less than 10,000 is between 24-27% (based on a number of methodology reviews and some data from online survey companies). So at 10%, I'd say that there would be some sort of interfering factor like the length of the survey or the perceived topical relevance of the survey to the respondents.

If anything below a 25% response rate would be observed, I would be wary of some results, as the non-responds may be indicative of those with say, exchange relationships, or those who really don't think much about the organization at all (the class president, for instance, is more likely to answer these questions about her high school than the burnout is). Of course this is an inherent problem in most surveys, and is just something that should be monitored.

Here's a related thought, as long as we are on the topic: The Grunig instrument differs from most "typical" commercial surveys, possibly so much that people will begin to feel confused about the purpose of the survey and perhaps exit early as they begin to feel more like guinea pigs than empowered consumers. (I have absolutely no hard evidence on this as yet, it's just a thought.) To counteract this, I'd recommend that the incentive for a relationship survey should be higher than the incentive for a typical customer satisfaction survey.

Bruce, what do you think?

Bruce Aube
Senior Account Manager, KDPaine & Partners:

In general, I think a 24-27% response rate for an online survey would be deemed above average in most scenarios. (We did experience a 32% response rate with our Coos County survey, but I’ve also seen response rates below 10% for consumer-related studies.) I agree with the rest of your comments and explanation.

A 15-20% response rate is a generally accepted business practice. But, without boring everyone with the details, I recall managing a project for a consumer-protection entity that required a 60% response rate. It really depends on how the data is used – to help direct business decision making vs. something like that consumer protection project that was going to be published in industry journals. 

Measuring Naked Relationships: How to evaluate the impact of a social media program on your public relationships.

Your step-by-step guide to using relationship metrics to evaluate the success of your social media program.

by Katie Delahaye Paine

I've talked a lot here -- also in my speeches, in my blog, and of course in my recently published book Measuring Public Relationships -- about the importance of measuring relationships. I believe that without factoring in the impact that your social media program has on your relationships (with employees, community and constituencies) you are undervaluing your efforts.

So, how does one actually measure relationships? Well, now that you asked, my book explains this with regard to most types of public relations programs. Social media being all the rage right now, I thought it would be appropriate to provide a step-by-step guide as to how to actually do it for social media programs.

Whatever type of program you want to measure, the basic technique is similar: You conduct a survey of your audience using a special set of questions designed to specifically measure the different components of relationships. You do this before and after your social media program is in place, and you do it for your organization and as many competing organizations as you can afford to. Then you compare the data before and after, and between your organization and the others, and then you know where your relationship with your audience stands and where you need to go.

The Grunig/Hon Relationship Research

Before we get to the nuts and bolts, here's a bit of background. A decade or so ago, University of Maryland Professors Jim and Laurie Grunig and Linda Hon synthesized communications and sociology research and theory into a paper published by the Institute for PR called "Guidelines for Measuring Relationships in Public Relations." Their feeling was that, amidst all the brouhaha about quantifying public relations, we were forgetting the essential truth that PR was about relationships. And so you needed to measure the impact that your efforts were having on those relationships.

Their research isolated six fundamental components of relationships -- Trust, Satisfaction, Commitment, Control Mutuality and Exchange/Communal -- and they designed and tested 75 statements to measure those components. These statements are of the sort: "This organization treats people like me fairly and justly," and, "I feel a sense of loyalty to this organization." You can see them all, and copy them for your own use, on my blog or in the Grunig/Hon paper. For an example of these statements successfully used to measure public relationships in a non-social media context, see this research by Forrest W. Anderson and Paul Raab.

Nine Steps to Measure Your Social Media Relationships

Before we start, remember that to isolate the effect of your social media program, you must begin your measurement before you launch that program. Then you'll have a benchmark to compare against later: Before Social Media vs. After Social Media. Without a Before benchmark you won't know how your social media program has changed your relationships.

Of course, you can always begin measuring after your social media program has started, and by doing so you will be able to do ongoing evaluation of your relationships. Which is a good thing. But a better thing is to isolate the effect of social media, and to do that you must compare before and after.

So use Steps 1 through 9 below to survey your audience before your program begins, and thus establish your benchmark. Then begin your social media program, and after an appropriate time, say three or six months (depending on your situation), re-survey to see what has changed.

To be sure that whatever change you find is solely attributable to the social media program, you must hold constant any other PR programs that affect your social media audience's perception of your organization. Yes, it's tricky, and it's not always an ideal world. But if you are trying to measure the effect of your new blog at the same time as the Promotions Department decides to give away A Free Cadillac With Every Purchase, then you can kiss your results goodbye.

Step 1: Define the audience for your social media program.
Social media is about conversation and engagement, so decide with whom you want to converse and engage. If you're starting an internal blog, your audience is your employees. If you're starting an IdeaStorm-type customer community, your audience is anyone likely to buy or recommend your product. If your mission is advocacy, your audience might be voters, politicians or industry influencers.

Step 2: Get a list of your audience's email addresses and/or phone numbers.
To get a representative sample you will need at least 500 names for each organization you are testing (more on that later). If you already have a list of your members, subscribers or customers, then you are ahead of the game.

If you have to purchase your list, then potential vendors vary with the type of sample you're looking for (mail, phone, web). Most lists are sorted based on demographic or title data. There are a lot more resources out there for mail addresses and those resources do not necessarily need to be survey sample companies. For email addresses, some reputable firms include Survey Sampling, e-Rewards and Zoomerang.

Step 3: Pick a survey methodology.
The Grunigs would recommend in-person surveys for the best results, but most researchers find that to be very expensive. Phone surveys are fast and provide very accurate results, but again, depending on the audience, may be cost-prohibitive. Email surveys are an increasingly accepted methodology, and for social media can be appropriate and highly reliable, since, presumably, your audience is all on email.

You may be able to piggyback on an existing survey going out to your community. If marketing, customer satisfaction, business development or anyone else in your organization is doing a survey, see if you can add a few of the Grunig/Hon statements to it.

Step 4: Select which of the Grunig/Hon statements are most appropriate to your organization.
You can probably only impose on someone for 7-10 minutes of their time, so you need to pick which statements you will include. Grunig and Hon suggest that if you want to shorten the survey, you use only the boldfaced items. Not all statements are appropriate for all organizations, so pick and chose the ones that will be most meaningful to your audience.

Step 5: Prepare your survey.
If you are using an electronic survey system like Survey Monkey, you need to create an introductory screen that explains what you are doing and how the scoring works. For instance:

In order to better understand the needs and perceptions of our marketplace, we'd like to ask you some questions. Please tell us whether you agree or disagree with the following statements as they apply to X company/organization.

Explain that 1 means "totally disagree" and 7 means "completely agree," and give them an option for "no opinion." You also need to ask them the same questions about a competing company or organization, so you have comparable data on the competition to compare to.

Step 6: Send out the survey.

Step 7. Resend the survey.
Depending on your audience it may take several tries and an incentive to get sufficient responses (I'll do just about anything for an Amazon or Starbucks card). How many is sufficient? Well, it depends on how you plan to break down your analysis, but in general plan to resurvey until you get at least a 10% return. If in doubt, talk to your local survey expert.

Step 8. Analyze and learn from the results.
Calculate a mean score for each relationship component. There are both positive and negative statements in the survey instrument, so make sure you take that into account. Compare your mean on each score to the competition. (And of course to your earlier survey results, if this survey is not the benchmark.)

Step 9. Implement your program and measure again.
If this is your pre-social media program (benchmark) survey, then implement your program now, and measure again in six months. Or, if your program has been running for a while and your analysis indicates you need to make changes, then make the changes now and let them work on your audience for six months before you measure again.

Good luck, and let me know how things go.

Should We Trust Web-Based Studies? (This research says Yes.)

This comparative study of Web-based survey samples and paper-and-pencil surveys (by Samuel D. Gosling, Simine Vazire, Sanjay Srivastava and Oliver P. John and which appeared in American Psychologist) is now four years old, but its conclusions are probably still valid: Yes, Web-based self-report and self-selected samples can be trusted for surveys. From the study's summary:

"Internet data collection methods, with a focus on self-report questionnaires from self-selected samples, are evaluated and compared with traditional paper-and-pencil methods. Six preconceptions about Internet samples and data quality are evaluated by comparing a new large Internet sample (N
361,703) with a set of 510 published traditional samples. Internet samples are shown to be relatively diverse with respect to gender, socioeconomic status, geographic region, and age. Moreover, Internet findings generalize across presentation formats, are not adversely affected by nonserious or repeat responders, and are consistent with findings from traditional methods."

Get the whole study here.

Hitting on Guys in Bars and Other Tools of the Trade for Obtaining Customer Feedback

Jenny Schade's
Making It Count

 

Research at the Scene of the Dime

In a sports bar in downtown Chicago, my colleague and I approach a twenty-something guy. I'm doing the talking and she's ready with her clipboard.

"Excuse me," I begin. "Can we speak to you for a few minutes?"

"Yeah, just a second," he responds. He leans over to one of his buddies and whispers, "I can't believe this, we just got here and already chicks are hitting on me!"

He returns to us. "Hi!"

"We'd like to ask you some questions about the video games here,"I explain. "It's market research. Our interview will take about 15 minutes and we'll give you $25 for making time for us."

"Oh," he replies. "Well, I guess that sounds okay. What are your questions?"

Our interview begins:

Q: "Why do you come to this bar in particular?"

A: "My friends and I probably come here two or three nights a week. I don't have to plan anything – it's all here for me. All I have to do is show up."

Q: "Do you ever play the video games they offer here? "Why or why not?"

A: "More often than not, we play video games. It's something to do and after playing one a few times, you can actually get better at it.

Q: "What do you look for in a good video game?"

A: "Something happens on the screen and I have to do the right thing. It makes me feel kind of good when I do well. A couple of weeks ago, I was winning and this one game made all sorts of noise. Pretty soon, I had a crowd gathered around me. It was this game right here – let me show you what I mean…"

His responses exemplify why doing these interviews on-site is so valuable for obtaining practical information for our video game manufacturing client. We've quickly covered a lot of ground in this interview and it's all very important to our client:

• The idea that this sports bar patron is a "regular" and looks to the bar he frequents to provide easy entertainment.
• The fact that he develops a sense of competency as he plays a video game.
• The emotional reward he receives when he interacts with a game and it signals to him and to others that he is succeeding.

When we multiply this single interview times the 30 video game players we will talk with this week, we start to recognize some patterns. Product innovation and marketing recommendations emerge almost effortlessly. For example, our research immediately suggests:

• Developing new games that offer increased interaction and lights/sound acknowledgment of winning players.
• Offering bar promotions that provide patrons with added entertainment and rewards such as video game tournaments and play-offs.
• Capitalizing on players' desire for feeling competent by publicly recognizing winners through a bar "Wall of Fame."

Research At the Scene of the Dime

By being on-site for these interviews, we are able to talk directly to our client's customers and observe them in their natural surroundings – their habitat, so to speak. We're catching them off-guard and encouraging them to share how they use our client's products as well as what they value about them.

This case study about interviewing video game players in sports bars is an example of going directly to your customers and conducting research in the field rather than having them come to a research facility to learn about their opinions. Traditional qualitative market research that takes place in a focus group facility can be very informative, however, it is invaluable to observe and talk with customers of certain kinds of products and services right where they make the purchase decision or, as I like to call it, "Right At the Scene of the Dime."

By talking with customers as they are pulling out their wallets to plug dimes – or dollars – into video games, we can capture their top-of-mind thoughts and emotions, gaining insights into their needs and connections with our products.

Jenny Schade's Three Questions for Determining Interview Location

So how do you identify when conducting research in the field would be advantageous over a more traditional setting? Ask yourself Jenny Schade's Three Questions for Determining Interview Location:

1. Is there a benefit to observing the people we want to interview in the setting where they actually use our products?
2. Would it be easier/more cost effective to go to our respondents rather than trying to recruit them to come talk to us?
3. Do others influence how our customers interact with our products?

In the case of our video game machine producing client, our responses to all three questions were affirmative.

1. It was tremendously beneficial to observe the bar patrons on-site and understand the other attractions that competed for their attention.
2. It was actually quite easy to recruit respondents in the bar, rather than paying for recruits that might not actually show up (probably because they were in the bar that night!).
3. Only by observing our respondents were we able to understand the role their friends and other patrons played in their choice to play video games and keep playing throughout the evening.

Following are examples of other research initiatives that we deemed more appropriate to conduct in the field rather than in a professional research facility:

• Interviewing construction workers at their sites regarding the qualities they seek in the truck boxes that house their tools in the back of their pickup trucks. Doing the interviews at the work sites enabled the workers to show us the truck box features they preferred and respond to product ideas under consideration.
• Running "rap groups"of teens in their after-school clubs to identify the program offerings they found most appealing. Holding the sessions at the schools greatly increased participation and allowed us and our client to understand the situational pressures faced by club members.
• Interviewing bakery employees right on the assembly line as they frosted and put together cookies and cakes. This setting helped us see directly the logistics involved in the workers' responsibilities and understand their reactions to corporate initiatives that they felt were insensitive to their needs. (This was also memorable as the only time I ever conducted interviews while wearing a hair net! Just picture the "The Lucy Show.")

When you really need to see your customers in action, there's nothing as effective as conducting research in the field. Whether that means putting on your construction hat or a hair net, you're sure to gain an in-depth perspective on what motivates your customers to invest financially – and emotionally – in your products.

Jenny Schade is president of JRS Consulting, Inc., a firm that helps organizations build leading brands and efficiently attract and motivate employees and customers. Subscribe to the free JRS newsletter on www.jrsconsulting.net/newsletter.html
© JRS Consulting, Inc. 2007

Public relations measurement can now -- hey! -- measure relationships.

Relationship Measurement Case Study

Relationships Audit Reveals Precise Public Relations Weaknesses
"The client even applauded at the end of our presentation!"

by Forrest W. Anderson

(Editor's Note: After reading this article, you can learn more about measuring relationships in Katie Delahaye Paine's new almost-hot-off-the-press book "Measuring Public Relationships" here.)

Until I tried it, I dismissed the idea of measuring relationships. However, I've changed my thinking since using these measures in a communications audit led by Paul Raab at Denver-based Linhart Public Relations. The relationship measures gave us unexpected insights that led to solid business and communications recommendations. The client, the National Governing Board (NGB) for a U.S. Olympic sport, even applauded at the end of our presentation! They tell us they refer to it and use it every day -- the COO actually cuts and pastes our recommendations when assigning communications tasks to staff!

Relationship measurement is a fairly recent development. As you may know, in 1999 Jim Grunig and Linda Childers Hon published a paper on how to evaluate relationships. One of their premises is that the real business of public relations is managing relationships. Using professional and academic information, they derived six factors that collectively measure the strength of a relationship from the stakeholder's point of view:

1. Mutual Control
2. Trust
3. Satisfaction
4. Commitment
5. Exchange
6. Community

Grunig and Hon also developed specific questions to measure these factors. For our NGB audit, we used these questions to survey current and potential team athletes, their parents and coaches, donors and trustees. The data showed relationships were strong across all but two of the six relationship factors: Exchange and Mutual Control.

We disallowed the Exchange factor results, which showed that stakeholders did not believe the organization did anything without expecting something in return. Considering the particular situation, this seemed appropriate.

However, the Mutual Control issue was different. Our client wants commitment from athletes and coaches to training and integrity, and to their organization. And it needs money from donors. The athletes and coaches want financial and logistic support from the organization. Donors want to see performance, integrity and support for the organization from the athletes who reach the podium. The survey results, however, showed that most stakeholder groups believed the client didn't listen or respond to their wants and needs. The data enabled us to demonstrate to the client that to achieve its goals, it needed to listen better and actually respond to stakeholder concerns, rather than ignore them.

I was surprised by how sensitive this technique is. Not only did it tell me the relationship was weak, but it told me how. As a communications professional, the recommendations for addressing that kind of issue were second nature.

Until now, I had not given the approach much attention, mostly because it doesn't get at sales or share price, the issues I've always believed CEOs really care about. Now I think evaluating the strength of relationships would be useful for any organization. It obviously makes sense for public and community affairs. Other applications could include assessing relationships with customers, employees, shareholders, financial analysts, technical analysts and so on. It also offers a benchmark against which to evaluate the success of communications programs.

Forrest W. Anderson is an insight consultant helping organizations communicate better through better understanding their target audiences. He is a founding member of the IPR Commission on PR Measurement and Evaluation.

Paul Raab, APR, is senior vice president and partner at Linhart Public Relations.

Sex, Surveys and Faulty PR Measurement!

Sex, surveys and faulty PR measurement: Can an article about public relations research get any more exciting than that? PR measurement depends an awful lot on surveys for its raw data, and "The Myth, the Math, the Sex" (in yesterday's New York Times) shows we can't trust that data. I'm as wicked keen about about this article as I was about Freakonomics. And it's about sex, too, did I mention that? --WTP