While working on last issue's article "Measuring Naked Relationships: Your step-by-step guide to using relationship metrics to evaluate the success of your social media program," a question came up concerning relationship measurement surveys: How many completed questionnaires are sufficient? Suppose you are going to do an email survey of your target audience with the Grunig relationship questions (or see just the questions here on Katie Paine's blog), and you send out 500 surveys. What response rate is enough to be sure your results are significant? Should you keep resurveying your audience until you get a 10% return? A 20% return?
We asked our in-house survey experts, Peter Kowalski and Bruce Aube:
Director of Research Strategy, KDPaine & Partners:
The typical response rate of an e-mail survey of a respondent set of less than 10,000 is between 24-27% (based on a number of methodology reviews and some data from online survey companies). So at 10%, I'd say that there would be some sort of interfering factor like the length of the survey or the perceived topical relevance of the survey to the respondents.
If anything below a 25% response rate would be observed, I would be wary of some results, as the non-responds may be indicative of those with say, exchange relationships, or those who really don't think much about the organization at all (the class president, for instance, is more likely to answer these questions about her high school than the burnout is). Of course this is an inherent problem in most surveys, and is just something that should be monitored.
Here's a related thought, as long as we are on the topic: The Grunig instrument differs from most "typical" commercial surveys, possibly so much that people will begin to feel confused about the purpose of the survey and perhaps exit early as they begin to feel more like guinea pigs than empowered consumers. (I have absolutely no hard evidence on this as yet, it's just a thought.) To counteract this, I'd recommend that the incentive for a relationship survey should be higher than the incentive for a typical customer satisfaction survey.
Bruce, what do you think?
Senior Account Manager, KDPaine & Partners:
In general, I think a 24-27% response rate for an online survey would be deemed above average in most scenarios. (We did experience a 32% response rate with our Coos County survey, but I’ve also seen response rates below 10% for consumer-related studies.) I agree with the rest of your comments and explanation.
A 15-20% response rate is a generally accepted business practice. But, without boring everyone with the details, I recall managing a project for a consumer-protection entity that required a 60% response rate. It really depends on how the data is used – to help direct business decision making vs. something like that consumer protection project that was going to be published in industry journals.