by Katie
Delahaye Paine
When measuring the health of political campaigns, as with any measurement program, the type of system you put in place depends on your objectives and your role. Below are examples of three types of political measurement, with a list of tools you need for each type of program.
Brian Solis is a smart guy. Go read "Social media is about social science not technology," posted yesterday in socialmediatoday. He reminds us that relationships aren't automatically the result of using social media:
"The great myth of social media is that it enables your business to build relationships with customers. Perhaps part of the problem is that the definition of relationships in this social economy is too simplified. Relationships are not a function or derivative of technology. Pursuing the 3F’s of Friends, Fans, and Followers does not directly equate to value."
It's easy to blithely announce that "We're using social media to build relationships with our customers!" But chances are that your SM-powered "relationship" is so shallow it's meaningless. Building relationships is a lot more complicated that tossing out a few Tweets:
"Unfortunately in new media, we tend to put technology ahead of people. Think about your current social media, mobile, or web strategy for a moment. Do you even know who you’re trying to reach? Do you know what customers or stakeholders expect or the challenges they face? Are you familiar with how they connect and communicate and why? Lastly, do you understand the journey they take to make decisions?"
And that journey they take to make decisions can be a lot more complex than the old Sales Funnel two-step that we used to talk about. Go read the whole article. --WTP
###
Summary: The author attempted to develop an automated system for rating media mentions using Grunig's relationship concepts. Traditional media was found to include too little expression of sentiment. A second attempt was made with only social media mentions, and some success was achieved. In general, coding with humans was found to be much faster and more accurate than automated coding.
by Katie Delahaye Paine
With the ubiquity of social media today, the notion of using social media as a way to listen to and gauge public opinion has captured the interest of marketers. At the moment, the hot button is sentiment measurement, which is the notion that you can teach a computer what positive and negative tone is by precoding it with key words. So the computer searches for brand proximity to words like “suck” and “hate” as a proxy for negative sentiment.
However, sentiment analysis has been shown to be notoriously inaccurate, unreliable, and therefore not terribly useful. With the addition of human coding to improve the model, most sentiment analysis tools can achieve 75% accuracy at best.
Instead of sentiment, why not relationships?
We postulate that marketers are chasing after the wrong emotion. Rather than attempt to gauge sentiment, it would be far more useful to understand how customers, prospects, employees and other constituents feel about their relationship with your brand. So we decided to try to use the core concepts of the Grunig Relationship model to define an algorithm that would rate how stakeholders feel about an organization based on their conversations in social media.
What we thought we were going to do
We defined a list of key words, phrases and concepts that would be tracked, individually and as a cluster, around each of Grunig’s six components of relationships: Control Mutuality, Trust, Commitment, Satisfaction, Exchange, and Communal. We selected three institutions which we thought would be good candidates for this experiment: Harvard University, Stanford University, and MIT. To ensure comparability, we had to eliminate coverage of sports, since Stanford was headed for the NCAA championships and MIT is not known for its athletic achievement.
We then used human coders to content analyze 2,000 items for the relationship concepts. The idea was to take the terms and concepts that showed up and teach the computer to identify relationship components. The plan was to use an algorithm to score the conversations on a scale of 1 to 5, so that each institution would receive a score on each element of the relationship.
What really happened, and lessons learned
During the human coding phase we learned a tremendous amount about using content analysis to measure relationships.
Lesson #1: You can’t analyze passion if it’s not there.
The first discovery we made was that when you remove sports from any study of the university environment, most of the emotion and passion goes with it. What was left was a great number of discussions about scientific and academic research, very few of which actually contained any sentiment at all. Of the 2,000 total items, only 265 (13%) actually contained any of the relationship concepts we were trying to study.
Lesson #2: You Need Twitter and Facebook to analyze sentiment.
No matter what liberals and conservatives may say, the vast majority of traditional media is neutral or balanced, i.e., it doesn’t contain much opinion at all. It also doesn’t reflect what people are directly saying, just what a reporter or an editor has allowed in the piece. So, traditional media does not offer much to work with in a relationship study. You can count the comments in the online versions of these stories, but you also should know that only a small percentage of these are actually seen, and by a very small percentage of the audience.
In reality, if you want to measure sentiment you need to be analyzing Twitter and Facebook, and that’s the rub. Getting full content from either of those sources is not easy. Most services only get about 15% of Twitter, and even those with a full Firehose licence only get about 85%. Facebook poses an even greater challenge since most content providers only provide what is publicly available, which again is small percentage of the universe.
Personal blogs are another good source of sentiment, and we definitely recommend including them, but you need to exclude the content farms and fake sites to make sure you are getting a true reflection of people’s opinions, not something a bot created.
Lesson #3: Trust and Communal Relationships were the easiest relationship concepts to detect.
Communal Relationships (each party sees mutual benefit in the relationship succeeding) are frequently reflected in an organization’s corporate social responsibility, so community events and good deeds were easily identified as promoting a positive communal relationship. 44% of all item contained some expression of communal relationship and 30% contained references to trust.
Automated coding: Does not compute
Given the relatively low level of sentiment detected and the challenges we had in training humans to code for relationship concepts, our expectations weren’t very high for automated coding. Nonetheless we used what we learned from human coding to create a taxonomy based on the concepts and key words we had discovered.
While it was interesting to discover that the accuracy level for positive coding was better than negative, overall the machine coding data was unusable. There was simply not sufficient data to analyze with any degree of accuracy. Further, we learned that the linquistic modeling tools we were using require an valid/invalid construct rather than the scale model that we wanted to use. Back to the drawing board.
So we tried focussing on social media
We then decided to analyze an entirely different set of data: 12,000 purely social media items about a highly controversial organization that had been frequently in the news during the past year. This time the results were much more interesting. We found more than half (56%) of the 12,000 contained some expression of a relationship concept.
Lesson #4: Some concepts are easier than others to identify and score
Once again Communal and Trust were found most frequently, with 17% of mentions conveying trust and 13% suggesting a communal relationship. However, we found that Satisfaction and Trust were most accurately identified. And we learned that some of the other Grunig concepts are virtually impossible to track, such as Control Mutuality and Exchange Relationship.
Taking it to another level, we learned that it’s relatively easy to describe a set of terms to define trust, but teaching a computer takes a lot more time. So, for example, a human knows the difference between “I fell in love with Stanford” and “I fell in love at Stanford.” But it requires a fairly sophisticated algorithm to teach a computer the difference.
Pulling trust out of a posting gets even more complicated. So, if someone posts to Facebook that “I fell in love at Stanford and felt very comfortable openly expressing my feelings for my gay lover,” that clearly expresses trust in the Stanford campus environment. But teaching a computer that the feelings are for the campus and not the lover takes a whole other level of complexity.
Lesson #5: It takes too long.
The only restriction on getting a computer to do this accurately is time. If we had nothing to do but perfect our automated relationship taxonomies for the next year or two, we’d eventually get it right. For now, for the most part, humans are faster and more accurate.
What’s Next?
Clearly this is an important first step – perhaps just a baby step – in the direction of using content analysis to determine relationship health. We feel pretty confident that, given sufficient emotional content, human coders can accurately glean the concepts of Trust, Communal Relationships, and Satisfaction from social media conversations. As we conduct more such analyses, we will continue to define more words and concepts that will help refine the process and improve accuracy still further. In terms of getting a computer to be as accurate, we clearly have a long way to go.
Further complicating the process is the difficulty in creating a generalized taxonomy. Good taxonomies must be specific to the organization studied. So, to get any generalized taxonomy you will need to model which words or phrases are more important. In order to get there you need a greater quantity of human-coded relationship data to build a statistical model that would validate relationship constructs in social media.
###
Katie Delahaye Paine is CEO of KDPaine & Partners, a company that delivers custom research to measure brand image, public relationships, and engagement. Katie Paine is a dynamic and experienced speaker on public relations and social media measurement. Click here for the schedule of Katie’s upcoming speaking engagements.
So SAI Chart of the Day sent out a fabulous graph of relationship breakups over time, thanks to Facebook and David McCandless:
I have had some harsh things to say about Chart of the Day in the past (see: “Bad Measurement: This Is The Kind of Chart that Really Pisses Me Off”), but this very interesting graphic pretty much makes up for past transgressions (although note the spelling of “Valentine's Day...”)
I can understand why people break up on Mondays, what with probably an eventful weekend preceding. Likewise it's obvious that what happens on spring break doesn’t always stay on spring break. But what's up with April Fool's Day?
And who would have guessed that so many breakups happen just before Xmas? (Although, come to think of it, that‘s when my ex-wife walked out. So there you have it.)
But hey, how come there are so few breakups at the end of the summer? I wonder what the demographics of this Facebook sample are and how that would affect these results?
So I followed the link trail over to David McCandless’ blog Information Is Beautiful, and wow! there are some seriously great info-graphics there. Including:
Snake Oil? Scientific evidence for popular health supplements
The Billion Dollar-O-Gram. Where the big money goes to.
Reduce Your Odds Of Dying In A Plane Crash
--Bill Paarlberg, Editor, The Measurement Standard
The Measurement Standard is a publication of KDPaine & Partners, a company that delivers custom research to measure brand image, public relationships, and engagement.
The Institute for Public Relations, PRSA, and the University of Maryland hosted the PRSA International Conference on 10/17/10 in Washington DC. At this conference the Third Annual Grunig Lecture was given by, yes, Drs. Jim and Lauri Grunig and entitled: “Public Relations Excellence 2010.”
As many of you know, Jim and Lauri Grunig are developers of the Excellence Study and its principles for excellent PR, which they explain and elaborate on in this lecture. They are strong proponents of relationships as the foundation of public relations, and have developed techniques to measure relationships as a way to measure the effectiveness of public relationships. (Their work forms the foundation of much of Katie Paine's book Measuring Public Relationships, and also her new book due out next spring from Wiley Press Measure What Matters.)
Click here to download a transcript of the Third Annual Grunig Lecture by Jim and Lauri Grunig.
While working on last issue's article "Measuring Naked Relationships: Your step-by-step guide to using relationship metrics to evaluate the success of your social media program," a question came up concerning relationship measurement surveys: How many completed questionnaires are sufficient? Suppose you are going to do an email survey of your target audience with the Grunig relationship questions (or see just the questions here on Katie Paine's blog), and you send out 500 surveys. What response rate is enough to be sure your results are significant? Should you keep resurveying your audience until you get a 10% return? A 20% return?
We asked our in-house survey experts, Peter Kowalski and Bruce Aube:
Peter
Kowalski
Director of Research Strategy, KDPaine & Partners:
The typical response rate of an e-mail survey of a respondent set of less than 10,000 is between 24-27% (based on a number of methodology reviews and some data from online survey companies). So at 10%, I'd say that there would be some sort of interfering factor like the length of the survey or the perceived topical relevance of the survey to the respondents.
If anything below a 25% response rate would be observed, I would be wary of some results, as the non-responds may be indicative of those with say, exchange relationships, or those who really don't think much about the organization at all (the class president, for instance, is more likely to answer these questions about her high school than the burnout is). Of course this is an inherent problem in most surveys, and is just something that should be monitored.
Here's a related thought, as long as we are on the topic: The Grunig instrument differs from most "typical" commercial surveys, possibly so much that people will begin to feel confused about the purpose of the survey and perhaps exit early as they begin to feel more like guinea pigs than empowered consumers. (I have absolutely no hard evidence on this as yet, it's just a thought.) To counteract this, I'd recommend that the incentive for a relationship survey should be higher than the incentive for a typical customer satisfaction survey.
Bruce, what do you think?
Bruce
Aube
Senior Account Manager, KDPaine & Partners:
In general, I think a 24-27% response rate for an online survey would be deemed above average in most scenarios. (We did experience a 32% response rate with our Coos County survey, but I’ve also seen response rates below 10% for consumer-related studies.) I agree with the rest of your comments and explanation.
A 15-20%
response rate is a generally accepted business practice. But, without
boring everyone with the details, I recall managing a project for
a consumer-protection entity that required a 60% response rate. It
really depends on how the data is used – to help direct business
decision making vs. something like that consumer protection project
that was going to be published in industry journals. 
Your step-by-step guide to using relationship metrics to evaluate the success of your social media program.
by Katie Delahaye Paine
I've talked a lot here -- also in my speeches, in my blog, and of course in my recently published book Measuring Public Relationships -- about the importance of measuring relationships. I believe that without factoring in the impact that your social media program has on your relationships (with employees, community and constituencies) you are undervaluing your efforts.
So, how does one actually measure relationships? Well, now that you asked, my book explains this with regard to most types of public relations programs. Social media being all the rage right now, I thought it would be appropriate to provide a step-by-step guide as to how to actually do it for social media programs.
Whatever type of program you want to measure, the basic technique is similar: You conduct a survey of your audience using a special set of questions designed to specifically measure the different components of relationships. You do this before and after your social media program is in place, and you do it for your organization and as many competing organizations as you can afford to. Then you compare the data before and after, and between your organization and the others, and then you know where your relationship with your audience stands and where you need to go.
The Grunig/Hon Relationship Research
Before we get to the nuts and bolts, here's a bit of background. A decade or so ago, University of Maryland Professors Jim and Laurie Grunig and Linda Hon synthesized communications and sociology research and theory into a paper published by the Institute for PR called "Guidelines for Measuring Relationships in Public Relations." Their feeling was that, amidst all the brouhaha about quantifying public relations, we were forgetting the essential truth that PR was about relationships. And so you needed to measure the impact that your efforts were having on those relationships.
Their research isolated six fundamental components of relationships -- Trust, Satisfaction, Commitment, Control Mutuality and Exchange/Communal -- and they designed and tested 75 statements to measure those components. These statements are of the sort: "This organization treats people like me fairly and justly," and, "I feel a sense of loyalty to this organization." You can see them all, and copy them for your own use, on my blog or in the Grunig/Hon paper. For an example of these statements successfully used to measure public relationships in a non-social media context, see this research by Forrest W. Anderson and Paul Raab.
Nine Steps to Measure Your Social Media Relationships
Before we start, remember that to isolate the effect of your social media program, you must begin your measurement before you launch that program. Then you'll have a benchmark to compare against later: Before Social Media vs. After Social Media. Without a Before benchmark you won't know how your social media program has changed your relationships.
Of course, you can always begin measuring after your social media program has started, and by doing so you will be able to do ongoing evaluation of your relationships. Which is a good thing. But a better thing is to isolate the effect of social media, and to do that you must compare before and after.
So use Steps 1 through 9 below to survey your audience before your program begins, and thus establish your benchmark. Then begin your social media program, and after an appropriate time, say three or six months (depending on your situation), re-survey to see what has changed.
To be sure that whatever change you find is solely attributable to the social media program, you must hold constant any other PR programs that affect your social media audience's perception of your organization. Yes, it's tricky, and it's not always an ideal world. But if you are trying to measure the effect of your new blog at the same time as the Promotions Department decides to give away A Free Cadillac With Every Purchase, then you can kiss your results goodbye.
Step
1: Define the audience for your social media program.
Social media is about conversation and engagement, so decide with
whom you want to converse and engage. If you're
starting an internal blog, your audience is your employees. If you're
starting an IdeaStorm-type customer community, your
audience is anyone likely to buy or recommend your product. If your
mission is
advocacy, your audience might be voters, politicians or industry influencers.
Step
2: Get a list of your audience's email addresses
and/or phone numbers.
To get a representative
sample you will need at least 500 names for each organization
you
are testing (more on that later).
If you already have a list of your members, subscribers or customers,
then you are ahead of the game.
If you have to purchase your list, then potential vendors vary with the type of sample you're looking for (mail, phone, web). Most lists are sorted based on demographic or title data. There are a lot more resources out there for mail addresses and those resources do not necessarily need to be survey sample companies. For email addresses, some reputable firms include Survey Sampling, e-Rewards and Zoomerang.
Step
3: Pick a survey methodology.
The Grunigs would recommend in-person
surveys for the best results, but most researchers find that to be
very expensive. Phone surveys are fast and provide very accurate results,
but again,
depending on the audience, may be cost-prohibitive. Email surveys
are an increasingly accepted methodology, and for social media can
be appropriate and highly
reliable, since, presumably, your audience is all on email.
You may be able to piggyback on an existing survey going out to your community. If marketing, customer satisfaction, business development or anyone else in your organization is doing a survey, see if you can add a few of the Grunig/Hon statements to it.
Step
4: Select which of the Grunig/Hon statements are most appropriate
to your organization.
You can probably only impose on someone for 7-10
minutes of their time,
so you
need to pick
which statements you will include.
Grunig and Hon suggest that if you want to shorten the survey, you
use only the boldfaced items.
Not all statements are appropriate for all organizations, so pick
and chose the ones that will be most meaningful to your audience.
Step
5: Prepare your survey.
If you are using an electronic survey system like Survey
Monkey,
you need to create an introductory screen that explains what you
are doing and how the scoring works. For instance:
In order to better understand the needs and perceptions of our marketplace, we'd like to ask you some questions. Please tell us whether you agree or disagree with the following statements as they apply to X company/organization.
Explain that 1 means "totally disagree" and 7 means "completely agree," and give them an option for "no opinion." You also need to ask them the same questions about a competing company or organization, so you have comparable data on the competition to compare to.
Step 6: Send out the survey.
Step
7. Resend the survey.
Depending on your audience it may take several
tries and an incentive to get sufficient responses (I'll do just
about anything for an Amazon or Starbucks card). How many is sufficient? Well, it depends on how you plan to break down your analysis, but in general plan to resurvey until you get at least a 10% return. If in doubt, talk to your local survey expert.
Step
8. Analyze and learn from the results.
Calculate a mean score for each relationship component. There are both
positive and negative statements in the survey instrument, so make
sure
you take
that into
account.
Compare your mean on each score to the competition. (And of course
to your earlier survey results, if this survey is not the benchmark.)
Step
9. Implement your program and measure again.
If this is your pre-social media program (benchmark) survey, then implement your program now,
and
measure again
in six
months. Or, if your program has been running for a while and your analysis
indicates you need to make changes, then make the changes now and let
them work on your audience for six months before you measure again.
Good luck,
and let me know how things go.
by Katie Delahaye Paine
Ah Springtime! When a young man's thoughts turn to love. Or basketball. Or if, like myself, you live in New Hampshire, politics. Given that New Hampshire has the third largest legislative body in the English speaking world (after the British Parliament and the US Congress), and that once a year in March a million or so average citizens get together in town meetings to determine the fate of the state, you can understand why the term "March Madness" takes on a whole new meeting up in these parts.
When measuring relationships with voters, as with any measurement program, the type of system you put in place depends on your objectives and your role. Below is a list of tools you need for each type of program. One important point first...
Timing
is more critical than usual
Because the length and success
of voter measurement programs are so strictly defined by the election
cycle,
you can't
afford to
waste
time or opportunities. Probably
the most important element in any political measurement program is
to
have
the data
on hand
when
you
need it.
And you need it whenever there's a decision to be made about how
much money to spend or what tactics to use. You don't need
it after the votes have been counted.
Measuring PR Efforts on Behalf of a Candidate
My company, KDPaine & Partners, measured political PR for one of the country's best political PR pros, Doug Hattaway, former spokesperson for Al Gore and Tom Daschle. Doug was working on Chellie Pingree's effort to unseat Susan Collins as Senator from Maine. What Doug needed was a measurement system that would tell him which tactics were working or not working in a campaign. The challenge was to do it fast enough so the data would be useful, and cheap enough so the campaign could afford it.
The good news about working with political campaigns is that volunteers are generally plentiful, and in this case the volunteers were charged with collecting the clips from the local publications. If you don't have volunteers to do the clipping, you need to hire a local clipping company or someone like Bacons or Burrelle's to collect the clips for you. Small-town newspapers are just too important to miss. Electronic data aggregators like Nexis and Factiva do not work on local or statewide campaigns; they do a terrible job collecting clips from anything smaller than the Boston Globe. Even Cyberalert and Custom Scoop miss far too many of the small town weeklies.
One we had the clips about both Pingree and Collins in hand, we began to analyze them for positioning on key issues, tone of coverage, type of article, subject of the article and who was quoted. We compiled them monthly and issued a report that included charts like this (click the charts to see them bigger):
We could quickly identify the tactics that were most successful. For example, a visit by Hilary Clinton in support of Pingree was far more successful in garnering visibility for Pingree than any other action. However, Pingree's visit to Washington in support of her health care reform legislation, while it didn't get anywhere near the visibility, was more successful in positioning her as "the health care candidate."
The analysis also revealed weaknesses in Pingree's campaign and strengths in Collins' campaign. While Pingree was almost entirely focused on her health care message, Collins was making points with the voters on the environment and campaign finance reform.
Additionally, the research enabled the campaign to track the correlation between visibility and contributions. By tracking Pingree's share of exposure over time against the contributions over time, the campaign could determine the level of additional exposure necessary to generate the requisite contributions.
In the end, the earned media that was generated by Hattaway's PR efforts narrowed incumbent Collins' lead substantially.
Measuring
Lobbying Efforts on Behalf of a Company
We ran into a government relations person for a major organization who wanted to know how to measure the impact of his efforts. He was trying to get a bill passed that would be highly advantageous to his company. We began to design his "dashboard," and asked him what success meant. He got as far as "If the bill passes, I'm successful." We pointed out that if the bill failed it would be too late to do anything about it.
So what he really needed was to monitor the tone of conversation around the bill in question so that he could tweak his PR program according to its progress. We recommended that he establish a system that gives him feedback all along the way.
There are several ways to do that. The easiest method is to track who is saying what to the media or in local speeches about the bill. Are key influentials supporting it or trashing it? Or is no one talking about it at all? More important to track, however, is the bill's progress through the legislative process. Is it being heard, tabled, or moved? Who are the cosigners to the bill and are they actively promoting it? Is the opposition gaining or weakening? What do bloggers say? What's being discussed in news groups? Are there any list serves you need to track?
Some of these data points can be gathered via media analysis, others should be part of your regular legislative tracking process. Still others may require a poll of the constituency to determine the level of support, and the perceptions of the voters and/or legislators.
If at all possible, a once-a-year survey of your key legislators is recommended to test and evaluate the health of the relationship between your organization and the elected or appointed officials you are striving to measure.
Measuring Relationships with Publics on Behalf of Yourself as a Candidate
Your constituencies come in several forms, from students to senior citizens. You should measure relationships with as many different stakeholders or constituencies groups as you can afford. It may be useful to segment the different publics by gender, age, length of time in the area and political leanings so you can identify any pockets of opportunities or threat within the community.
Clearly active stakeholders who are most likely to protest, boycott or otherwise cause trouble are the most important to understand. However, inactive stakeholders that are clueless about your programs and therefore don't care and won't get engaged can be just as dangerous.
We recommend using the Grunig relationship model to test your relationships and in particular the level of trust that each public feels towards you. Trust is a key component of success in any political campaign. If it's not there, or if it is recently lost, you will have a significantly harder time achieving success. Alternatively, if your constituents trust you, as do those of Alan Greenspan or John McCain, you have a lot more leeway with those publics. Therefore it is critical to establish a benchmark level of trust to begin with. Subsequently you should conduct regular trust/relationship measurement studies to gauge the level of trust and engagement over time. Conduct these studies as often as possible so you can tie any changes in the trust and relationship scores to actual actions you may have taken.
The Ultimate Measure: Votes
There's a ton of great data in any set of election results. The challenge is to use it to make more informed decisions. When I ran for town council here in Durham five years ago, I knew that my efforts to go door-to-door in specific communities paid off since the number of votes I received from those communities far exceeded my expectations. I also knew that my signage (bright purple and white with the slogan "No Paine, No Gain") was successful since I had the highest name recognition, and ultimately the highest vote count of all the candidates.
In larger elections,
there's far more granular data to be gathered,
and much of it is readily available online. Using this data you can
frequently identify the specific areas or pockets or demographics
you've targeted and thus measure your success in getting those groups
to vote
and/or vote your program or candidate.
by Katie Delahaye Paine
In this article I suggest that media analysis programs would more accurately reflect articles' influence on readers if the integrity of the media outlet was taken into account. This could be achieved by developing a standard measure of media integrity which would used to rate different media outlets for different shareholder groups. The resulting Integrity Index would be used to weight coverage, thereby achieving a more accurate measure of media coverage's impact on consumers.
To understand why an Integrity Index is needed, consider:
Item
#1
Why measure what no one believes?
Mazen Nawahi, in an impassioned speech at the 2008 Dubai Measurement Summit, raised the issue of journalistic integrity and the degree to which the integrity of a particular media outlet should be accounted for in any measurement program. Put another way, if:
then why would anyone believe anything that was printed?
And if all your media coverage in a particular outlet lacks credibility, why would you include it in your measurement results? For example, in 22 years in business I have yet to have a client want me to include The Weekly World News in its media list. So how do people justify measurement programs that include stories that do nothing to achieve their goals?
Item
#2
Credibility is in the eye of the beholder.
Suppose a highly credible blogger gets into a mud-wrestling contest with the blogging equivalent of a pig. The contest attracts a lot of attention, a lot of arguments fly back and forth, and a lot of dirt gets thrown around. Now for the people that are into farm animals, the arguments of the pig might be very credible. However, people interested in learning something of value professionally will no doubt pay more attention to the words of the credible blogger. But the real question is: Do you weight coverage of them both equally in your media analysis reports?
How to correct for influence?
Taking examples of this sort into consideration, it is easy to see why some sort of authority or influence weighting of coverage would be of value. There has been a fair amount of research done in this area. For several years, Angie Jeffrey at VMS has been studying various weightings of media coverage to determine which factors most directly impact sales. Her research first found that share of discussion was effective, but she postulates that advertising rates are more telling because they reflect the importance of the various media. (See this paper and this paper.) I think it's a step in the right direction, but I generally frown on the use of advertising rates (and of AVEs), so I'm skeptical.
The Media Integrity Index: Why not rate coverage on integrity as rated by stakeholders?
I suggest that it's not the ad rate but the integrity of the publication that most determines consumer credibility and thus drives consumer action. And I'm suggesting that to rank a media outlet in importance, we should ask our stakeholders how they perceive the media outlet in terms of integrity.
I raised the question on Twitter and got some interesting responses:
• "It's all relative... What is good for one is bad for the other. It has to be specified for a purpose, customized."
• "Integrity or perceived integrity? Seems there's a difference there that's highly subjective." --Ryan Anderson
• "Integrity is the measure of perceived relationship (believability) of one-to-many listeners. So measure relationships. I would suggest looking at the way political integrity is measured. It would be a perceived metric, measured by polling audience. --Videodred
• "What's the goal? What's the client want it to say? In the case of journalists, what's the relationship?
• "Integrity has to be measured parallel to influence as audience determines credibility of source." --Mike Maney
So in one way, my proposed Integrity Index has to start with the goal of the coverage. What is it that the company or organization is trying to accomplish? Consider it this way: If you don't care who or what your brand is associated with, and you want exposure pure and simple, then you don't care about the integrity or credibility of whatever media outlet is talking about you. On the other hand, if you're trying to establish a reputation, or build brand loyalty and trust, or trying to reach an audience with certain media preferences, then the credibility of media to your audience should matter a great deal.
How to determine integrity?
So how do we determine which media is trustworthy and credible, and which journalists have or do not have integrity? The wrong way is to look at a list of media and assign weights or values to each one yourself. What you think doesn't matter. All that really matters is what your customers or stakeholders or members or constituents think. So you need to ask them. The best way to do that is to use the Grunig Relationship Instrument.
Now you probably don't need to ask them about all 150 publications on your media list. Remember: Never ask a question about something that can't be changed. If The New York Times is what your boss' boss reads every morning, there's no chance in hell you'll ever take it off you top tier media list. Even if your target audience is nine-year-old girls. So start with a list of suspect publications, blogs or any other type of media outlet.
Remember that different stakeholder groups may rate a given media outlet differently. As I pointed out above, integrity is not a fixed standard. For instance, we would expect Republican viewers to rate Fox News and The New York Times differently than Democratic viewers. It is conceivable that you will want to generate different Integrity Indexes for different stakeholder groups.
Below are some survey questions adapted from the Grunig Relationship Instrument. You may wish to develop your own, based on the specific components of relationships you wish to measure (refer to this paper). Ask your stakeholders whether they agree or disagree with each statement as it pertains to each media outlet, then use the responses to rate the outlets.
1. This media outlet treats people like me and organizations like mine fairly and justly.
2. Whenever this media outlet makes an important decision, I know it will be concerned about people like me.
3. This media outlet can be relied on to keep its promises.
4. I believe that this media outlet takes the opinions of people like me into account when making decisions.
5. I feel very confident about this media outlet's skills.
6. This media outlet has the ability to accomplish what it says it will do.
7. Sound principles seem to guide this media outlet's behavior.
8. This media outlet does not mislead people like me.
9. I think it is important to watch this media outlet closely.
10. This media outlet is known to be successful at the things it tries to do.
Industry-wide integrity standards?
Now, the logical question is, "Why aren't we doing this as an industry?" Shouldn't we be factoring in credibility based on some industry-wide standard? It certainly is a more accurate weighting factor than simple eyeballs or ad cost. But the reality is that your stakeholders aren't going to be identical to my stakeholders, and what matters is how a significantly valid sample of your stakeholders feel. It would be awfully complex and difficult to set industry-wide standards for many different stakeholder groups.
On the
other hand, if the PR industry wanted to take on a project to
try and accurately weight publications based on their integrity,
the
world would most definitely be a better place. And it is not far-fetched
to imagine an industry-standard Integrity Scale or Survey that could
be used to determine the Integrity Index for different stakeholder
groups.
Social Media Measurement
by Katie Delahaye Paine
This article is condensed from a paper submitted to the 11th Annual International Public Relations Research Conference.
"If we can put a man in orbit, why can't we determine the effectiveness of our communications? The reason is simple and perhaps, therefore, a little old-fashioned: people, human beings with a wide range of choice. Unpredictable, cantankerous, capricious, motivated by innumerable conflicting interests, and conflicting desires."
Ralph D. Paine, Publisher, Fortune Magazine. 1960 Speech to St. Louis Ad Club
Modern technology has come up with many good ways to measure what human beings read, watch and see, but comparatively few ways to measure -- as my father said half a century ago -- those "unpredictable, cantankerous, capricious... conflicting interests and conflicting desires." The recent rise in the influence of social media has turned the entire communications paradigm upside down. Counting column inches and eyeballs is irrelevant when a single YouTube video enjoys a larger audience than Monday Night Football, the average consumer is bombarded with 5000 messages a day, and 90% of CEOs say they are dissatisfied with how their CMOs measure results.
The basic problem is that we have years of research that says that if you "expose" 1 million consumers to a message (or buy 20 GRPs) you will sell X number of cases of shampoo, soda or soap. We have no data that says if 1 million people download your YouTube video, you'll sell any shampoo at all. What we now want to know is how social media affects user's behavior.
Engagement: The Relationship Between the User and the Brand
Like most other buzzwords, "engagement" has come a long way from its original meaning of "an agreement to marry." Essentially, it started with the notion that a website or a blog was "engaging" enough to get a reader to begin to develop a relationship with the brand.
As more and more advertisers and media types realized that hits really do stand for "How Idiots Track Success" and that even unique page views were suspect (given the enormous variation in such statistics), people began to speak of measuring engagement--not just how "sticky" the site was, but the extent to which it enhanced the relationship between the user and the brand. Advertisers now want to measure a site's ability to create an experience that earns a visitor's loyalty and, with luck, its business. As a result "engagement" now means everything from the number of times that a visitor returns to the site to the time spent online.
Another way to think of engagement is as the fourth step in a five step process that the individual user goes through:
Engagement According to Scoble
Popular blogger Robert Scoble (2006) has suggested that engagement is a valid measure of user interaction and authority of Internet-based social media channels. That is, engagement is a way to determine whether you are really having a dialog, or you are just yelling ever more loudly. His premise is that by measuring activity on a blog or social media website as a sign of engagement, you can predict users' behavior. In other words, if they come back to a corporate blog over and over again they'll eventually buy. If it's a YouTube video, if they watch and rate it or comment on it, they are more likely to pass it on to their friends and maybe even take some other action as a result.
Brian Haven of Forrester Research picked up on Scoble's premise and proposed measuring engagement based on a variety of tangible and intangible factors including links, track backs, comments and the frequency sentiment and tonality of comments. He defines engagement as the level of involvement, interaction, intimacy, and influence an individual has with a brand over time:
Peterson's Engagement
Web Analytics expert Eric Peterson, author of Web Analytics Demystified, Web Site Measurement Hacks, and The Big Book of Key Performance Indicators, has proposed alternative measures of engagement based on Web metrics. Peterson suggests that if you want to measure engagement you need to measure stats like the following:
The problem with Peterson's metrics is that for most organizations, that data is only available on their own site, not on competing sites, so there is no way to conduct a benchmark to understand how "engaged" visitors are with one's own site vs. the competition.
While both Peterson and Haven contribute important ideas to the engagement discussion, I suggest that measuring engagement necessitates following the actions and desires of the customer.
An Engagement Index? Not Yet.
There is no such thing, yet, as an engagement index, but there has been a lot of talk about the possibility. Both Scoble and Peterson suggest that their metrics could be reduced to a single index, but they don't say how.
(Others have written excellent discussions of engagement, see, for instance, Steve Bridger's nfp 2.0 blog.)
Most of the discussion on the topic is centered on the necessity for advertisers to quantify the impact of their online advertising. Microsoft's new black box "Engagement Mapping" is designed to make advertisers on Microsoft websites more comfortable with their data (see our Measurement Menace Award for this month). Comscore and Nielsen's efforts are designed to give more meaning to the numbers they provide advertisers.
Unfortunately, metrics that make advertisers happy are not necessarily very useful for other communications functions. As internal and external communications functions become more involved in social media, they too need a way to measure engagement, but numbers from Microsoft, Comscore and Nielsen are only available for large consumer sites, not corporate blogs. More problematic is that those numbers do not factor in the newer more popular social networking sites like Facebook, YouTube and Twitter.
Engagement is a Relationship
I suggest that "engagement" is just another way to say "relationship, but a minor one." So, to the metrics suggested by Scoble and Peterson, I suggest we add those from relationship theory. At some point, you just need to come right out and ask your audience:
In short, we suggest incorporating the concepts of the Grunig Relationship Instrument into the engagement measurement.
Unless one incorporates relationship measurement into the mix, you end up with just data rather than insight. Because, while you can track behavior with increasingly accuracy, all the web metrics in the world may not answer the fundamental question of "Why?" "Why did they stop coming to your site?" "Why are they spending less time there?" Or, more critically, "Why are they buying less?" Without the true understanding of the nature of the relationship, you won't be able to do anything to improve once you find out what the problems are.
Which
leads me to some final, unanswered questions: What is the difference
between engagement and relationship? In fact, do we really know that
engagement is something distinct -- distinct from web stats
and distinct from relationship theory? Suppose we do use Grunig's
questions to measure engagement, how do we know we are measuring
"engagement" rather than "relationships?" If
you think you know, please let
me know.
The Paine of Measurement
How online measures of engagement have predicted recent primary results.
Ever wondered what the effectiveness of political lawn signs is? Supposedly, every lawn sign represents six votes for the candidate. Or maybe ten votes, depending on what you read. And there's a theory in political circles that if you can get someone to put out a lawn sign, then that person is committed enough to not just vote for you, but also to encourage his or her friends to vote for you as well. So, each additional lawn sign means more than just one more vote, it means more of something even more valuable and a lot more difficult to pin down: more loyalty or commitment or what we in communications call engagement.
To my knowledge, no one has ever done a scientific study of how lawn sign displays influence voting habits. But my completely unscientific study of New Hampshire lawns this fall more or less predicted the outcome of our First in the Nation primary: Everywhere you went there were lots of Obama and Ron Paul signs, and both did much better than the polls predicted.
Now let's transfer this scenario into the world of social media. Can online measures of engagement predict votes? I argue that they can and have done so recently:
And in the end, Obama did better in the primaries than the early polls suggested. The primary results have proven that Obama has a stronger than expected following, as hinted at by the strong online engagement we found.
The point here about engagement is bigger than just politics. How and why is engagement a stronger or different measure than just impressions? If, by joining a group, rating a video, or following someone on Twitter, you are actually thinking or behaving differently than if you just viewed an ad or a message, then measuring these signs of engagement is critical to every marketer. In order to hang on to advertising dollars, media companies will need to provide this data. And the good news is that the data is there, they just need to release it.
And finally, I can't help but see engagement as a kind of bridge between measuring outputs and measuring relationships. (Most of you are aware of my recently published book Measuring Public Relationships, learn more here.) If you measure an output like impressions, you only know what has happened to an audience. But if you measure engagement, you are measuring what is done by an audience as the result of their relationship with your output. How does measuring engagement fit in with measuring relationships? That's a good question, let me know if you have the answer.
Here's wishing you large measures of success,
Social Media Measurement
Or... What's the ROI of my living room?
by Katie Delahaye Paine
As everyone who has talked to me recently knows, I'm a serious social media evangelista. And as I travel around I'm constantly confronted with business people who say: "Social media? I don't get it! Who has time? Why should I bother?"
My simple answer has always been that blogging (or Twitter, or Facebook) is a way to engage in a conversation with your customers or your employees.
(And if, Mr. CEO, you do not want to engage your customers or your employees, you deserve to be fired. If not shot.)
But I've been told that that attitude isn't particularly helpful when you're talking to people who think Facebook is still "just a college thing," and who think "twittering" must be something dirty.
So here's my new explanation (a bit long-winded, but, please, bear with me here):
When I built my house, it was designed to be the capital of social capital, with a huge living room and kitchen and dining room so that lots and lots of people could gather there. As that's just what happened:
So are you getting my point? Social Media is just a great big version of my living room. Any social network -- Twitter, Facebook or MyRagan -- starts off as one big noisy place. But soon, people of like minds and like interests start to find each other and sometimes they spin off and form their own separate groups.
So What's the ROI of my Living Room?
Which gets us to measurement. The hot topic right now -- and what everyone wants to know -- is: How do you measure the ROI of the effort you put into these groups?
To answer that, let's go back to my living room. The bride and groom's goal was to save money. The Library's was to build their mailing list. The politician wanted votes and the musician wanted to sell CDs. If they wanted to measure the ROI of their events in my living room, they'd compare their investment in effort to the particular return that was important to them.
And What's My ROI for Social Media?
I, on the other hand, use social media, and (often) my living room, to satisfy my thirst for knowledge and intellectual stimulation. Take Twitter: Do I appreciate the fact that traffic on both my blog and my website has picked up since I started Twittering? Absolutely. But mostly I Twitter because it makes me smarter. It allows me to follow interesting people that I wouldn't ever meet in my living room, and who would never read my blog.
I mess around in Facebook because it's a great way to share stuff with my friends and colleagues and it's a huge time saver when you're trying to pull together an event.
I blog because I like having conversations with people – about my business, my services, and the world as a whole.
And that
ends my longwinded explanation of why you or anyone might want to
bother with social media. It depends on you and
your
goals. And as for measurement, it's the same as for any marketing
effort; first be clear about what you're
trying to achieve, then go about measuring it.
by Katie Delahaye Paine
1. Set up Google Analytics on your blog to find out how many repeat visitors you have. How many pages per visit do they check out? How many go back more than 3 times a week? How many go back and spend more than a second or two on the site?
2. Post a vizu poll on your blog and see how many people respond.
3. Go to xinure and enter the URL of your choice to find out how well it is doing in search engines, links, social bookmarks and a whole bunch of other stats.
4. With many of the leading blog providers like TypePad, check your stats to find out how many people have subscribed, and how many visits per day you're receiving.
5. What's the Conversation Index (the ratio of postings to comments)? In the blogosphere any comment is a good comment because it shows that people are engaged enough in what you are saying to take the time to respond.
6. If you have posted a video on YouTube, or a photo on Flickr, check to see how many people have rated it, and/or commented on it.
7. If you have a presence on Facebook, how many people have joined your group?
8. Ask a question on Facebook and see how many people respond.
9. If you're on Twitter, how many followers do you have? More importantly, how many responses do you get when you ask them a question?
10. Set
up a Technorati, Sphere or Icerocket search to find out whether people
are writing or talking about you.
Many public relations professionals find that media analysis or survey research are tools sufficient to measure their programs. The more adventurous bring in external data and more esoteric measures of success like donations or number of new memberships or even lives saved. In politics, KDPaine and Partners has recently used YouTube video counts to predict primary results, and our Ms. Paine has ruminated on the possibility of using the number of political lawn signs as an election predictor. Not until now, however, has anyone used dreams to measure political progress.
In this week's New Yorker, Ben McGrath writes about Sheila Heti and her new blog "I Dream of Hillary... I Dream of Barack." This blog, a repository of reader-submitted dreams about the candidates, can be interpreted as a rough poll of interest in those candidates. As Mr. McGrath says, "...what if the recurrences of Presidential candidates in people's dreams were meaningful in the aggregate?" As of the writing of the article, "..Obama's edge in the over-all dream count... was roughly equivalent to his lead in the latest Gallup poll."
Ms. Heti's blog has been so successful she plans to open an I Dream of McCain section, so in the future we will have bipartisan dream data to go on. --Bill Paarlberg, Editor, The Measurement Standard
Your Measurement Reading List
Measuring Public Relationships Makes Measurement Simple
Measuring Public Relationships, The Data-Driven Communicator's Guide to Success by Katie Delahaye Paine, 228 pages, paperback. ISBN 978-0-9789899-0-3. Order online or by phone: 603-319-1047.
BERLIN, NH, Dec 12, 2007-- The first how-to book on measuring relationships for public relations professionals was released today by KDPaine & Partners, the Berlin, NH-based leader in social media and public relations measurement. Measuring Public Relationships, The Data-Driven Communicator's Guide to Success was written by KDPaine & Partners' CEO, Katie Delahaye Paine.
Paine takes a "cookbook" approach, providing specific steps to measure all forms of public relationships, from social media measurement to tracking relationships with local communities, industry analysts and social networks. It relies heavily on the relationship theories of James and Laurie Grunig and Linda Hon but translates the theory into an easy to read practical guidebook. The book has been well received by academics and professionals alike who have called it "the must-have practical guide to hands-on PR measurement." Emphasizing the role and evaluation of relationships, this book provides every public relations professional with step-by-step research procedures for measuring programs, improving results, and managing relationships.
"It really is an excellent book," said Professor James Grunig, calling it "lively, engaging, accessible, wise, and candid... a remarkable compendium..."
"Katie Paine is the most practical consultant and educator on public relations research that I know. She is brave enough to link PR to return on investment demands and smart enough to provide marketers and PR professionals with a wide range of useful research tools. If there is cost effective way to do research, Katie Paine will tell you about it," said Clark Caywood of the Department of Integrated Marketing Communications at the Medill School at Northwestern University.
"By force of prose and experience, Katie Paine level-sets the practical and now critical disciplines of media and communication measurement. A student of Grunig theory and a champion of the almighty relationship, Paine plops on every communicator's desk the best new reference for measuring the intangible. Read it, and you'll count your successes," said Alan Kelly, CEO & Founder, The Playmaker's Standard, LLC and author of The Elements of Influence. Adding that it's "...the best new reference for measuring the intangible."
Measuring
Public Relationships is available at a 25% discount until January
31st, 2008 when ordered directly
online or by
calling the KDPaine & Partners' office at 603-319-1047. 
Sponsorship Measurement
Six basic steps, five final lessons and four additional resources.
By Katie Delahaye Paine
"Live from the KDPaine & Partners/Verizon Wireless Arena we have the KDPaine & Partners' Battle of the Sponsorship Suppliers, underwritten by Orville Redenbacher's Microwave Popcorn, the official snack food of KDPaine & Partners; by Starbucks, the official fuel of KDPaine & Partners; and with additional support from Staples, the official office supply company; FedEx, the official overnight delivery system; Diet Coke, the official diet drink; and Catalina, the official pet of KDPaine & Partners."
Perhaps not these exact words, but something like them has passed your ears at least a dozen times in the last few years. From Little League teams to the World Cup, from naming arenas to naming road races, sponsorship of events is now one of the marketing methods of choice. As communications is increasingly impersonal and electronic, the chance to actually touch, feel and experience the brand is ever more important. Increasingly, purchasing is neatly divided into two categories -- on-line and experiential. More and more decisions are based on familiarity with the brand and ease of purchase.
Now, I'm an avid Amazon fanatic, but I still go into Borders' brick and mortar store and end up buying something. Why? Because I get to touch and feel and browse and listen to products. If I need something in a hurry, I go online. If I want "retail therapy," I go to Borders. That's experiential marketing. Marketing that transcends the product and the media and allows direct interaction with the brand.
A 2006 study by Jack Morton Worldwide showed that live event marketing -- experiences where consumers interact with products, brands or "brand ambassadors" face-to-face -- are among the most effective ways to influence coveted consumer audiences. The study, an online survey of 2,574 consumers ages 13-65 in the top 25 US markets, confirmed that this increasingly important marketing medium resonates strongly across all demographic and product categories.
Which is why total spending on experiential marketing is expected to reach $15 billion by the end of 2007, an 11% increase over last year, according to the latest IEG research report. Companies are now allocating on average 17% of their budgets to sponsorships, compared to just 13% last year. However, 76% of companies spend less than 1% of their total sponsorship budgets on research, and in a shaky economy, if you don't measure it, you can't show ROI. And if you can't show ROI, chances are good that the program will be killed.
Another factor in the measurement of experiential marketing is that simply "reaching large numbers of eyeballs" is no more relevant a goal than is "increasing market share in the buggy whip market." Therefore, new alternative metrics -- like relationship measures, engagement indices and cost per touch -- are increasingly demanded.
We suggest that there are six basic steps to developing a solid measurement program for your events and sponsorships.
Step 1: Be clear about what outcomes your communications program is designed to achieve.
As with any effort, you can't start to measure success until you know what success means for you. For any given event the objectives might be:
Each objective, of course, requires a different type of measurement. Some types isolate the impact of PR from other communications efforts better than others. And the best objectives are specific and measurable.
Step 2: Determine criteria -- quality as well as quantity
Once you've agreed upon your objectives, establish the specific criteria of success that you will measure. If your objective is awareness, the criterion might be the percentage increase of unaided awareness of brand or product. If your objective is to sell product, the criterion might be the incremental sales after a particular PR or promotional program took place. Consider those numbers that best reflect the health of your business, or that best represent characteristics that most affect your business:
To a certain extent, your choice of criteria is dependent on the type of event you are evaluating. If the customer experience you are measuring takes place at a trade show or exhibit booth, you might choose as key criteria the percentage of new visitors or the cost per minute spent with a client in the booth. If the experience takes place at an event such as a concert, you need to count how many people were exposed to your brand or the brand experience you were offering. Don't trust the promoter's numbers; do your own counting. Here's why:
I once met with a major auto manufacturer who wanted to measure the effectiveness of different sponsorships. It was spending many millions sponsoring car races, golf tournaments, antique car auctions and a variety of other events. According to the event organizers, attendance and therefore results were always better than the year before. But then I asked them what they were trying to achieve by these efforts. The first response was essentially: "We are always a major sponsor of these types of events." After about two hours of discussion, we agreed that the business objective was to drive potential customers into dealer showrooms.
We began a series of surveys at each event to determine first if the attendees remembered the sponsors, and secondly if they were more or less likely to test drive and/or buy the sponsor's car. We collected names at the event itself and called attendees two weeks after the event. Our results showed that on average 50% of all attendees were more likely to test drive a "sponsor" vehicle after attending the event.
After we had measured three different events, we were able to compare and contrast the cost effectiveness ($$ per person reached) of different events. The results enabled the sponsor to better understand the ROI from each event. By looking at the percent of people more likely to go to a dealership, they could determine the number of potential buyers. By subtracting the cost of the sponsorship from the profit and the projected number of car sales, they determined a projected ROI from the event. These results led the sponsor to dramatically alter their sponsorship strategies.
It's important to look beyond simple quantitative data, especially if your objective is exposure, to assess the quality of your communication. When companies clearly define their objectives it becomes a relatively simple matter to define a set of criteria against which to measure the relationships the company establishes with its constituencies in different events. Then it's a matter of comparing and contrasting results consistently cross a number of different events to ensure that the company has the most effective sponsorship program possible.
Step 3: Decide upon a benchmark
The key point to remember about any evaluation program is that measurement is a comparative tool. You need to compare one set of results to something else. The most meaningful comparisons are between different events, or between you and your peer sponsors at one event. If your fellow sponsors aren't direct competitors you can learn a lot by sharing information and determining what best practices really moved the needle.
4. Select a measurement instrument
Do not rely on data gathered at the event itself. It is rarely useful or actionable. Consider the following instead:
A relationship study
You can also use the Grunig relationship survey instrument (downloadable for free from the IPR web site) to test the health and strength of your customers' relationships to your brand. The Grunig instrument has been thoroughly tested and shown to be an extremely effective measure of how customers perceive their relationships with an organization.
The key thing about whatever measurement tool you use is to make sure that you are tying results back to the desired business outcome. A number of years ago, Country Music Television (CMT) hired tractor trailers to set up country music events in Wal-Mart parking lots. Their first measure was to ask the truck drivers how many people showed up. Realizing that those answers didn't tell them anything about what those potential customers were actually doing when they went home, CMT decided to hire my company to do follow-up studies with the attendees. The stated purpose of the events was to convince attendees to call their local cable company and request CMT or to take some other action on behalf of CMT. CMT used a sweatshirt giveaway at the events to collect names and addresses of attendees, whom we called two weeks after the event. We agreed that the common criteria against which we should measure all events was an ad hoc "level of engagement." Specifically, we asked people to rate how much the event experience affected their relationship with CMT, and then we established the mean level of interest for each event. We further defined the success criterion as the percentage of people willing to take action on behalf of CMT. Our surveys showed that 93 percent of attendees were willing to take some action and 89% were willing to make a phone call to their local cable company. We then compared results between different cities so CMT could determine where to expand the program the following year.
Five final lessons for sponsorship measurement:
1. You become what you measure, so be very clear internally on the objectives of your sponsorship or event that you will measure.
2. Match the event, the speaker, and the metrics to the objectives.
3. Document any immediate or tangible results: media coverage, phone calls or inquiries, messages delivered to key audience, etc.
4. Share and leverage results internally.
5. Repeat your measurement consistently.
Other resources:
www.instituteforpr.org offers
several white papers on measuring events, including:
My
just-published
book Measuring
Public Relationships includes at least a couple chapters that
bear on event measurement. You can purchase the book from the KDP&P's office
at 603-319-1047
(25% off the list price of $29.95 until 1/31/08), or you can download
the pre- press draft at no charge right here.
“Data will become the new soil in which our ideas will grow, and data whisperers will become the new messiahs.” 
