The Measurement Standard: Blog Edition

How Many Completed Questionnaires Does It Take to Measure a Relationship?

While working on last issue's article "Measuring Naked Relationships: Your step-by-step guide to using relationship metrics to evaluate the success of your social media program," a question came up concerning relationship measurement surveys: How many completed questionnaires are sufficient? Suppose you are going to do an email survey of your target audience with the Grunig relationship questions (or see just the questions here on Katie Paine's blog), and you send out 500 surveys. What response rate is enough to be sure your results are significant? Should you keep resurveying your audience until you get a 10% return? A 20% return?

We asked our in-house survey experts, Peter Kowalski and Bruce Aube:

Peter Kowalski
Director of Research Strategy, KDPaine & Partners:  

The typical response rate of an e-mail survey of a respondent set of less than 10,000 is between 24-27% (based on a number of methodology reviews and some data from online survey companies). So at 10%, I'd say that there would be some sort of interfering factor like the length of the survey or the perceived topical relevance of the survey to the respondents.

If anything below a 25% response rate would be observed, I would be wary of some results, as the non-responds may be indicative of those with say, exchange relationships, or those who really don't think much about the organization at all (the class president, for instance, is more likely to answer these questions about her high school than the burnout is). Of course this is an inherent problem in most surveys, and is just something that should be monitored.

Here's a related thought, as long as we are on the topic: The Grunig instrument differs from most "typical" commercial surveys, possibly so much that people will begin to feel confused about the purpose of the survey and perhaps exit early as they begin to feel more like guinea pigs than empowered consumers. (I have absolutely no hard evidence on this as yet, it's just a thought.) To counteract this, I'd recommend that the incentive for a relationship survey should be higher than the incentive for a typical customer satisfaction survey.

Bruce, what do you think?

Bruce Aube
Senior Account Manager, KDPaine & Partners:

In general, I think a 24-27% response rate for an online survey would be deemed above average in most scenarios. (We did experience a 32% response rate with our Coos County survey, but I’ve also seen response rates below 10% for consumer-related studies.) I agree with the rest of your comments and explanation.

A 15-20% response rate is a generally accepted business practice. But, without boring everyone with the details, I recall managing a project for a consumer-protection entity that required a 60% response rate. It really depends on how the data is used – to help direct business decision making vs. something like that consumer protection project that was going to be published in industry journals. 

Al Qaeda and YouTube: Public Relations Measurement Fights the War on Terror

Those of you who enjoyed our articles "How PR Measurement Can Win the War on Terror" and "How To Measure YouTube" will also be interested in "Fight Terror With YouTube" in today's NY Times. Judging by this article, measurement is already fighting the war on terror by evaluating al Qaeda's social media effectiveness, and YouTube is a major front in the action. -- Bill Paarlberg

Pornography, Public Relations Measurement and Anytown, USA

There are few things in life as exciting as solving a measurement problem with the innovative use of some new source of data. Followers of this blog and The Measurement Standard newsletter probably share our enthusiasm about Freakonomics (see our review here, and another example on beauty premiums here), and know how much we enjoy coming up with new ways of gaining insight from data. (Heck, it is in large part the theme of Katie Delahaye Paine's new book Measuring Public Relationships, and is the topic of a couple of her recent blog posts: "Another alternative to the Ad Value nightmare" and her Optimum Content Score proposal for media measurement. [She is the publisher of The Measurement Standard.])

So we encourage you to head right over and read an article in today's NY Times about using Google Trends data to demonstrate community standards in a pornography trial: "What’s Obscene? Google Could Have an Answer." It seems that these legal proceedings often face difficulty in establishing just what is or isn't obscene. And, wouldn't you know, Google has data on that: if Anytown, USA, searches as often for "orgy" as it does for "apple pie," then maybe some conclusions can be made about community standards. Or maybe not. Read the article. --Bill Paarlberg, editor