The Measurement Standard: Blog Edition

Research Bias: Another example of why you must consider the source of research.

Biased research results can creep into even the most carefully designed and executed studies. That's why medical research is often double-blind, so, hopefully, the experimenter's bias can't affect the results. In public relations research, bias is endemic because most every measurement project may affect someone's job or budget, and every measurement vendor wants their results to justify their fees.

So here's a review of cell-phone health research to remind all of us just how bias can affect results ("Source of Funding and Results of Studies of Health Effects of Mobile Phone Use: Systematic Review of Experimental Studies" by Anke Huss, Matthias Egger, Kerstin Hug, Karin Huwiler-Müntener, and Martin Röösli.) (Or see a summary here.) It analyzed research on the biologic effects of cell phone use, and found that industry-funded studies were far less likely to identify negative consequences than studies funded by governments and non-profits. Researchers analyzed 57 studies that appeared in the academic literature between 1995 and 2005. Only a third of the industry-funded studies identified a biologic effect with possible health consequences from exposure to cell phone radio waves, while 82% of the government- or non-profits-funded studies found such effects, as did 77% of the studies whose funding source was not identified.

How can public relations measurement identify and reduce research bias? --WTP

Why Public Relations Measurement Is Like Phrenology, or... Just because it is easy to measure, doesn't mean it's the best thing to measure.

PR measurement and marketing measurement often find themselves in a situation analogous to the 19th-century practice of phrenology. Way back then, some people used to think that the shape of your skull indicated your personality or intelligence. It was a very easy thing to measure, so they did. And they hung on to it as a serious area of study for much longer than they should have, because it was far more difficult to measure personality or intelligence by other, more effective, means.

Click-throughs aren't quite as useless as bumps on the head, but still... In the news today is a study by Starcom USA, behavioral targeting firm Tacoda, and comScore that suggests that clickthroughs might not be a very effective measure of the usefulness of online ads. See here for the news and here for Sam Harrelson's ReveNews discussion.

The obvious point here is that clickthroughs, while easy to measure, probably are not very effective at measuring what we would like to measure. The more interesting point is: What aren't we measuring what we really want to know?

I have noticed that PR and marketing metrics are often determined by what technology makes easily possible, rather than by what we would really like to know. Thus people tend to measure what technology drops in their laps (the easy things like click-throughs or AVEs) rather than something more relevant but difficult to measure (like, "Did the ad with the red type result in more sales among wealthy women?"). Then at some point they realize that their easy-but-less-relevant metric is not very effective, and then they spend a lot of time asking why the metric doesn't work.

What they should be asking themselves is: "What made us think that that metric would work in the first place?"

Instead of focussing on what technology is available (in this case click-throughs), wouldn't it be more productive to think about human thought and behavior first? (I've written about this before: "Measurement's Empty Head: Measurement ignores the most complex part of PR.") First think about what it is that might make PR work -- why it affects people's thoughts and behaviors -- and then decide what might make a good metric. First determine what we really want to measure (regardless of the technology or data available), then figure out a way to measure it. -- Bill Paarlberg, editor, The Measurement Standard

Sex, Surveys and Faulty PR Measurement!

Sex, surveys and faulty PR measurement: Can an article about public relations research get any more exciting than that? PR measurement depends an awful lot on surveys for its raw data, and "The Myth, the Math, the Sex" (in yesterday's New York Times) shows we can't trust that data. I'm as wicked keen about about this article as I was about Freakonomics. And it's about sex, too, did I mention that? --WTP