by Bill Paarlberg, editor, The Measurement Standard
I was reading an interesting essay by Jonathan Stray entitled, "Metrics, metrics everywhere: How do we measure the impact of journalism?" over at Nieman Journalism Lab. As the subtitle says, it's encouragement to "get beyond counting pageviews and ad impressions and build better ways of judging how our work changes the world around us."
And that got me to thinking about this newsletter and how we measure it.
Continue reading "How We Measure The Measurement Standard" »
The real-life lesson of how a tourism organization can measure the ROI of their communications -- without using Advertising Value Equivalency (AVE).
by Katie Delahaye Paine
Normally we would have named New Hampshire Grand a Measurement Menace of the Month, but we decided that it would be a much more useful exercise to give them a Measurement Makeover. In the future we’ll be combining the menace with a makeover so watch these pages for more examples. I welcome your submissions.
This month's Measurement Makeover features New Hampshire Grand. New Hampshire Grand is an organization that promotes tourism in the North Country of the state of New Hampshire. And, since I was intimately involved with its creation, I was appalled when I read this statement in New Hampshire Grand’s recent progress report:
“In 2010, we... published over 369 stories with an earned media value of $512,000. Earned media is the value of our exposure if we had to pay for each column inch of the article.”
"Earned media value" is another name for Advertising Value Equivalence (also known as AVE), an inaccurate and discredited technique for evaluating media coverage. We immediately contacted Montagne Communications, the PR firm for New Hampshire Grand, for more details, but have not heard from them. Nevertheless, thanks to New Hampshire's tourism recordkeeping (see below) we can help New Hampshire Grand express the value of its efforts in a more accurate and accepted fashion. So let's give it a Measurement Makeover. Here goes...
Step #1: Resources Needed
1. Excel spreadsheet.
2. Data establishing relationship between impressions and actual tourist visits.
Step #2: Initial Data and Assumptions
It turns out all the data necessary is publicly available from the www.nhgrand.com and visitnh.gov websites. We have very useful historical data to work with, because the State of New Hampshire keeps good records on the value of its tourism efforts. They know that for every 100 positive tourism impressions they make (or buy) in the media, 3 people will visit the state as tourists. They also know that the average tourist spends about $82 dollars per visit in a combination of taxes and purchases.
Because New Hampshire Grand deals with the North Country specifically, and not the entire state, those averages need to be adjusted:
We can use these numbers to express the value of New Hampshire Grand's efforts in tourist dollars.
We also know that the New Hampshire Grand effort has cost about $1 million so far. We can use that figure to calculate an approximate Return on Investment for the project so far.
Step #3: Collect Some Data
We reviewed the monthly newsletter reports of Montagne that were posted to the New Hampshire Grand website and counted the total number of positive impressions. Since all the stories posted were favorable, we took the reported impressions as our total number of favorable impressions. That number is 5,099,541 impressions. (Montagne also reported a total Unique Visitor count for their efforts of 342 million. This number seems vastly overstated and unreliable, as it exceeds the total population of the U.S. by about 10%. So we did not use it.)
Step #4: Calculate Potential Value of Media Coverage
Using our estimate that every 100 impressions results in one visitor, that gets us to a potential visitor count of 50,995.
And if each of those 50,995 spend $41, that makes a potential spend of about $2 million. Thus the value of New Hampshire Grand's communications can be estimated as $2 million in revenue from tourist visits.
Step #5: Estimate ROI
The formula for ROI is Revenue minus Investment divided by Investment. Revenue in this case is what the potential tourists spend, or $2 million. Investment is about $1 million (which is what the New Hampshire Grand effort has cost thus far). So the ROI estimate would be $2,000,000 - $1,000.000/$1,000,000 or 100%.
Step #6: The Big Reveal
$2 million in tourist spending and an ROI of 100% sound a whole lot better -- and are far more accurate -- than $512,000 in fake Advertising Value Equivalency (AVE) numbers. Congrats, New Hampshire Grand, on your Measurement Makeover.
(Tip of the Hat to Chris Jensen at New Hampshire Public Radio who’s report on the need for New Hampshire Grand to demonstrate its value alerted us to an organization in need of a measurement makeover.)
(Transparency Alert: NCIC, which provides funding to New Hampshire Grand, is a former client of KDPaine & Partners. We conducted research for NCIC and were part of the discussion prior to the creation of New Hampshire Grand. We love what NCIC and The Tillotson Foundation have done and are on record as supporting New Hampshire Grand.)
###
Katie Delahaye Paine is CEO of KDPaine & Partners, a company that delivers custom research to measure brand image, public relationships, and engagement. Katie Paine is a dynamic and experienced speaker on public relations and social media measurement. Click here for the schedule of Katie’s upcoming speaking engagements. Katie and Beth Kanter are authors of the book “Measuring the Networked Nonprofit,” to be published this year by Wiley.
The Measurement Standard is a publication of KDPaine & Partners, a company that delivers custom research to measure brand image, public relationships, and engagement. Katie Paine, CEO of KDPaine & Partners, will be glad to talk with you about measurement for your organization.
In the field of public relations measurement, it is a given that decisions based on data are better — better informed and thus wiser and more accurate — than decisions based on gut feelings. That’s the point of measurement: You gather data on your success and use that data to be more successful.
And it’s also a given that there are a great many decision makers out there who do not trust the data. Or the process of measurement. Especially when it points to a decision that is different than the one they would like to make.
As Homer Simpson said, “Facts are meaningless. You could use facts to prove anything that's even remotely true!”
And so I was especially pleased to find, in yesterday’s The New York Times, an intriguing article by Daniel Kahneman: “Don’t Blink! The Hazards of Confidence.” It is about how us human beings are often erroneously over-confident in the wisdom of our judgements, to the point of not doubting our decisions when it is obvious that we should. To the point, in fact, of rejecting even the most compelling demonstrations of our incompetence.
As an example, the author tells a story of researching the effectiveness of stock brokers. He and a partner examined the portfolio performance of 25 brokers over the course of eight years, and found that the brokers were no better than chance at making money. The researchers then had to break the news to the directors of the brokerage:
...we told the directors of the firm... that, at least when it came to building portfolios, the firm was rewarding luck as if it were skill. This should have been shocking news to them, but it was not. There was no sign that they disbelieved us... our findings and their implications were quickly swept under the rug... The illusion of skill is not only an individual aberration; it is deeply ingrained in the culture of the industry. Facts that challenge such basic assumptions — and thereby threaten people’s livelihood and self-esteem — are simply not absorbed. The mind does not digest them. This is particularly true of statistical studies of performance, which provide general facts that people will ignore if they conflict with their personal experience...
I added the bold italics to that last sentence. Because it made me realize that it is no wonder that there is often such great resistance to measurement and data-informed decisionmaking. As Kahneman goes on to say,
Overconfidence arises because people are often blind to their own blindness... overconfident professionals sincerely believe they have expertise, act as experts and look like experts. You will have to struggle to remind yourself that they may be in the grip of an illusion.
I encourage you to read the article: “Don’t Blink! The Hazards of Confidence.”
I myself, of course, am extremely confident in the effectiveness of public relations measurement. --WTP
###
--Bill Paarlberg is editor of The Measurement Standard blog and newsletter, and of Katie Paine's new book Measure What Matters. The Measurement Standard is a publication of KDPaine & Partners, a company that delivers custom research to measure brand image, public relationships, and engagement.

Link fraud and a hint of under-the-table quid pro quo at the universe’s largest Web analytics company? Measurement skullduggery doesn’t get much better than this.
See last Saturday’s NYTimes for David Segal’s article “The Dirty Little Secrets of Search.” Therein, department store giant JCPenney, one of Google’s largest advertisers, is shown to have seriously played Google’s organic search rankings during the recent holiday shopping season:
For months, [JCPenney] was consistently at or near the top in searches for “skinny jeans,” “home decor,” “comforter sets,” “furniture” and dozens of other words and phrases, from the blandly generic (“tablecloths”) to the strangely specific (“grommet top curtains”).
This striking performance lasted... most crucially through the holiday season, when there is a huge spike in online shopping.
Investigation by the NYTimes revealed a massive link farming scheme that for months pushed JCPenney to Google’s #1 spot. Somehow Google, which had caught JCPenney doing something like this on three previous occasions, remained clueless to the scam throughout the lucrative online shopping season. Could it have anything to do with the fact that JCPenney was spending $2.5 million per month on Google ads?
Nothing suspicious there.
When confronted with the evidence, both Google and JCPenney were stunned and appalled. “We are just shocked—shocked!—that anything like this could happen,” is definitely not a direct quote from spokespersons for either company.
Coincidence or back-room bamboozle? Read the article and decide for yourself.
For related commentary, see:
—John Battelle’s “JC Penney’s Black Hat Link Farms—This Is News?” at Business Insider: “The never-ending battle between Google and link-baiting sites is as old as search itself. The story told in the Times' piece sheds absolutely no new light on the tale...”
and
—Link Farm Evolution: “High quality one-way links. On thousands of unique domains. Fully automated. Sounds like something you would be interested in? Then read on for a bullshit-free description...”
--Bill Paarlberg, Editor, The Measurement Standard
The Measurement Standard is a publication of KDPaine & Partners, a company that delivers custom research to measure brand image, public relationships, and engagement.
![]() |
Olympic hopeful Caleb Paine rounds a buoy |
A young sailor maps his course for the gold.
Editor's Note: This article is about how to work from your goals to develop KPIs and metrics.
- To learn about how to actually do the measurement, see "KDPaine's Trade Show & Event Measurement Checklist: Your framework for measuring trade shows, events and sponsorships.")
- To learn more about how to measure the ROI of a conference, see "How to Measure the ROI of a Conference," where you can download a sample spreadsheet that Katie Paine uses to measure conferences she speaks at.
by Katie Delahaye Paine
When I heard my cousin Caleb Paine talk about his goal to become an Olympic sailor, I immediately started thinking about metrics that could help him get there. For any public relations measurement program, setting goals and KPIs is a vital beginning step. (In fact, KDPaine & Partners' new product, the KBI Development System, helps companies do just that, among other things.) So I became intrigued about how I could apply some lessons from measurement to Caleb's quest for gold.
I first met Caleb when he was still in diapers. His father Doug, my first cousin, took me along as crew in a sailboat race off San Diego. Caleb spent most of the race wandering cheerfully around the boat, using every piece of hardware as his own private jungle gym. We were sailing very well and approaching the finish line when it became abundantly clear that Caleb’s diaper needed changing. Doug didn't hesitate or waver from his course—he took the tiller in his knees, changed his son’s diaper, and won the race.
No wonder the kid became a sailor!
At the age of seven, when he was already a hot shot Sabot racer, Caleb learned that sailing was an Olympic sport. That’s when his personal goal was established: He would win Olympic Gold in sailing.
Now, 11 years later, he’s a member of the Junior Olympic US Sailing Alphagraphics team, training for London in 2012. I was honored when he asked me to help him with his marketing. That’s when it occurred to me that I could use his campaign as a perfect example of how to measure events and sponsorships.
Defining Strategies, KPIs, and Metrics
Let's leave Caleb to his sailing for a moment and review the public relations measurement basics involved with KPIs and metrics. Your overall goal is pursued by means of one or more strategies. We define each strategy in terms of KPIs (Key Performance Indicators), which indicate progress on the strategy. The metrics are the specific measurements -- the numbers -- that we use to assess the KPIs, and thus assess progress on the strategy. Each metric is provided by a measurement tool.
Consider an example: If, for instance, you were measuring progress on your goal to increase a company's reputation as a thought leader in green technology, then:
Sail On, Sailor
Now let's return to Caleb's Olympic quest, and apply the above framework to analyze his campaign like we would any program. The goal is Olympic gold. I happen to know that he intends to pursue his goal by means of three strategies:
KPIs for the first two strategies are relatively easy and straight forward.
KPIs for Strategy #1: He clearly needs to achieve certain benchmarks in strength and endurance to know how strong he is getting and by what date. So typical KPIs might be:
In the marketing world these might translate into activity metrics. For instance:
KPIs for Strategy Part 2: The second set of KPIs are provided by the races in which he competes and the structure of US Sailing. Caleb needs to finish in the top ten or top twenty in every race and show consistent progress.
In the marketing world, these could be considered basic marketing performance metrics, such as:
KPIs for Strategy Part 3: The KPIs for the third strategy should also sound pretty familiar: Demonstrate marketing clout to attract sponsors. This could be measured by:
All these metrics can easily be tracked in a simple Excel spread sheet. If you’re a major corporation or non-profit, you’ll probably need a more sophisticated dashboard which will enable you to set dates, alarms, and benchmarks more easily.
But the point is that none of these metrics is particularly difficult to track. You just need agreement on what they are before you begin. ![]()

In the PR measurement business we are always talking about measuring this and evaluating that and finding ways to express goals and progress in numbers. But sometimes measuring your progress by merciless web stats can get to be a bit extreme.
Read Jeremy Peter's "In a World of Online News, Burnout Starts Younger" in today's NYTimes to learn how those readership numbers can be a harsh master. (And be sure to come back here to the TMS blog several times to boost our visit stats.) --Bill Paarlberg, Editor
Real-Life Public Relations Measurement Adventures
The
Measurement Standard's Real-Life Measurement Adventures series
includes true tales from the lives of actual public relations
measurement people.
(Well,
mostly true, and mostly actual.) We'd be pleased
to print your real-life measurement adventure, too. Just
click here to email it to us. This particular
Real-Life Adventure was submitted by Katie Delahaye Paine. Here's
a case in which measurement was instrumental in changing the bad
habits of both a company and its C-suite members. It's a true story
about a company we'll call The Smith Company. (For obvious
reasons, they don't want their
name used.) As of ten years ago, The Smith Company had done some
pretty bad things to the environment. In fact, it had earned itself
the
moniker, "One of
the
Worst Polluters on Earth." Since
that time, however, the company had seen the benefits of cleaning
up its act. As of the last few years it could have touted
a very good environmental
record, if it wanted to. But it had been so burned by bad publicity
in the past that it had adopted a policy of staying away from that
sort
of coverage,
doing its best to stay out of the limelight and the conversation.
Particularly
with respect to social
media. The Problem:
Boys Will Be (Bad) Boys The
company was founded by Mr. Smith, and several of his progeny
still have close
ties to it. In fact, they are members of its C-suite. These progeny
are not nearly as shy about publicity as the corporation is. Their
public
behavior
frequently makes headlines, leading
to unwanted publicity, and some confusion and negativity about
the corporate brand. Enter
Measurement So Smith's
PR person decided it was time to quantify the impact that the family
members' antics were having on the corporate brand. She
conducted a social media content analysis and found
that most of the conversations that mentioned The Smith
Company were negative. Every time the family members did something
at all newsworthy, the blogerati invariably brought up
the 10-year-old story of the company's environmental transgressions.
And, since there was no corporate voice to explain that the company
had, in fact, changed their evil ways and now had a very respectable
environmental
record, the net result was a steady drumbeat of criticism and negativity. With
her hands tied by company policy, and few communications alternatives,
there was little the PR person could do but to gather her data every
month and present the results to the C-suite.
Coverage was
at best holding steady, but
mostly getting significantly worse. Measurement's
Motivational Magic But then
something magical happened. Smith family members in the C-suite
saw
the numbers and the charts each month and realized
that their actions were generating coverage that negatively impacted
the corporation. And they changed their behavior. They
stopped doing stupid stuff, and instead, started doing good,
smart, philanthropic stuff. So,
simply by showing
the
charts and the numbers to the C-suite every month,
the PR person had changed the thinking and behavior that was
affecting her company's image. Of
late, discussion about The Smith Company is
predominantly positive. While the negative past is not entirely
left behind, it is subsumed by the more positive conversations
about the company. PR
measurement lesson: Focus on the big picture and let the numbers speak for themselves. 
Bad boys, bad boys: What you gonna do?
![]()
Jenny
Schade's
Making It Count
Note to Salesgenie.com's CEO: A little investment in research generates a huge return.
Vinod Gupta, the InfoUSA CEO who owns Salesgenie.com, learned this lesson the hard way when he had to stop airing his television commercial after its debut on Superbowl XLII because of protests about its culturally-insensitive presentation of animated pandas with Chinese accents. It was an expensive lesson: The cost of airtime alone for a 30-second spot on Superbowl 2008 was $2.7 million. View the ad here at YouTube:
Gupta, who wrote and produced the ad himself, told USA Today (February 11, 2008) that next year, he'll test his ads with consumer focus groups. This year, he said, he only ran the ad by some friends. "None said it was offensive," he said.
A Major Retailer Discovers that Ideas are Beautiful, but Reality Is Something Else
Vinod Gupta isn't the only top executive to become enamored with his own creativity. The Salesgenie.com ad debacle reminded me of another marketing campaign developed by a retail chain leadership team. Fortunately, those executives had the good sense to test the materials before going public. I'll never forget the shock experienced by the campaign's creators when they saw the reactions of their intended audiences.
By the time I was retained, a seven-figure budget had already gone into the creation of advertising and internal communications materials that featured real employees. It was my job to test the communications with focus groups to determine if they were relevant and would produce the desired reactions and results.
The stores in the materials looked wonderful. Inviting. Immaculate. Smiling employees welcomed you. Salespeople exhibited pride in their work. Several mentioned their impressive employee benefits: health insurance, 401K plans and paid vacations. Others spoke of their aspirations to become store managers. These proud, joyful team members encouraged the public to shop at their stores and consider working with them.
I showed the employees in the first focus group the materials and asked the following questions:
"What's the main idea behind what you see?"
"Would you notice this ad if you were looking through a magazine?"
"How does this information make you feel about working there?"
Immediately I knew we had a major problem. The employees looked angry, confused and resentful. Sample responses included:
The employees filed out. As I prepared for the second group, I popped into the adjoining room where a team of marketing and ad agency executives were watching through a one-way mirror. The creative director had a light film of sweat glistening on his forehead. No one looked at me. They were engaged in an intense discussion that absorbed all of their attention.
I showed the materials to the next focus group, which consisted of seven customers. I asked:
"Any immediate thoughts or feelings?"
"Is the ad clear?"
"How does this make you feel about shopping there?"
The customers were equally incredulous:
Again I ducked into the next room to confer with my clients. Now the creative director was openly mopping his forehead and the marketing director had gone ash white. He whispered to his assistant, "Haven't these benefits gone national?"
The assistant whispered back, "Guess not."
Where the Reality Hits the Road
I said to the marketing team, "For now, the only thing we can do is to proceed with these focus groups and instruct the participants to react to the ads as if they are true. Our goal today isn't to poll them on their actual experiences in your stores; it's to get their responses to this campaign."
The team agreed that this was the best use of our time; however, it was clear that there would be some serious accountability checking back at home office.
I proceeded with the same questions to the remaining 15 focus groups. Then my team and I summarized our findings in a written report that we presented to the client two weeks later.
Upon reviewing the report, the client called me and said, "I must admit, this hurts. We spent a lot of money on those materials, but I'd rather know this now than after rolling out a national, multi-million dollar campaign."
As painful as the findings were, they served a valuable purpose.
The overriding benefit of testing your marketing communications is ensuring that your communications are meaningful to key constituents. In other words, the resources that you devote to research now will pay off in spades when you move forward with communications programs and materials that hit the mark. By ensuring that your marketing is compelling, you can stop pursuing – and start attracting -- your intended audiences.
Jenny
Schade is president of JRS Consulting, Inc., a firm that helps organizations
build leading brands and efficiently attract and motivate
employees and customers. Subscribe to the free JRS newsletter on
www.jrsconsulting.net/newsletter.html
© JRS Consulting, Inc. 2008 ![]()
Jenny
Schade's
Making It Count
Strange
But True Tales
"The Personal Massager" and seven
other bizarre adventures from the consulting frontline.
Colleagues often tease me that I have more than my share of Strange But True adventures. I'm not sure if I actually attract more of these situations than other people, but I do acknowledge that I'm more open to finding the humor in unusual workplace events. I think you'll enjoy the following seven bizarrely humorous real-life adventures. (And a big "Thanks!" to my friends who contributed; you know who you are.)
1.
The Personal Massager
...or, How
to Generate Good Vibes in a Focus Group
One of the most memorable research studies that I have ever observed involved a female moderator, an all-male focus group and a "personal massager." The client had requested focus groups to explore perceptions of its line of sports massage products. At the end of the groups, the client wanted the moderator to casually and discretely display a "personal massager" to see how people compared it to its sports massage product.
At the end of the first group, the moderator walked over to the table with this product, tripped, and the "personal massager" shot out of its box, turned on and began loudly buzzing its way across the table toward the men in the group. They exploded in laughter, and one of them said, "Well, you certainly know how to operate one of those!"
This initial focus group project became an inside joke with our client for years to come, but it also served as an "ice-breaker" when we all laughed at the faux pas. It's a good example of how humor can relieve stress, help forge relationships and generally create a more positive atmosphere in which to conduct good business.
“Data will become the new soil in which our ideas will grow, and data whisperers will become the new messiahs.”