The Measurement Standard: Blog Edition

How Many Completed Questionnaires Does It Take to Measure a Relationship?

While working on last issue's article "Measuring Naked Relationships: Your step-by-step guide to using relationship metrics to evaluate the success of your social media program," a question came up concerning relationship measurement surveys: How many completed questionnaires are sufficient? Suppose you are going to do an email survey of your target audience with the Grunig relationship questions (or see just the questions here on Katie Paine's blog), and you send out 500 surveys. What response rate is enough to be sure your results are significant? Should you keep resurveying your audience until you get a 10% return? A 20% return?

We asked our in-house survey experts, Peter Kowalski and Bruce Aube:

Peter Kowalski
Director of Research Strategy, KDPaine & Partners:  

The typical response rate of an e-mail survey of a respondent set of less than 10,000 is between 24-27% (based on a number of methodology reviews and some data from online survey companies). So at 10%, I'd say that there would be some sort of interfering factor like the length of the survey or the perceived topical relevance of the survey to the respondents.

If anything below a 25% response rate would be observed, I would be wary of some results, as the non-responds may be indicative of those with say, exchange relationships, or those who really don't think much about the organization at all (the class president, for instance, is more likely to answer these questions about her high school than the burnout is). Of course this is an inherent problem in most surveys, and is just something that should be monitored.

Here's a related thought, as long as we are on the topic: The Grunig instrument differs from most "typical" commercial surveys, possibly so much that people will begin to feel confused about the purpose of the survey and perhaps exit early as they begin to feel more like guinea pigs than empowered consumers. (I have absolutely no hard evidence on this as yet, it's just a thought.) To counteract this, I'd recommend that the incentive for a relationship survey should be higher than the incentive for a typical customer satisfaction survey.

Bruce, what do you think?

Bruce Aube
Senior Account Manager, KDPaine & Partners:

In general, I think a 24-27% response rate for an online survey would be deemed above average in most scenarios. (We did experience a 32% response rate with our Coos County survey, but I’ve also seen response rates below 10% for consumer-related studies.) I agree with the rest of your comments and explanation.

A 15-20% response rate is a generally accepted business practice. But, without boring everyone with the details, I recall managing a project for a consumer-protection entity that required a 60% response rate. It really depends on how the data is used – to help direct business decision making vs. something like that consumer protection project that was going to be published in industry journals. 

Measuring Naked Relationships: How to evaluate the impact of a social media program on your public relationships.

Your step-by-step guide to using relationship metrics to evaluate the success of your social media program.

by Katie Delahaye Paine

I've talked a lot here -- also in my speeches, in my blog, and of course in my recently published book Measuring Public Relationships -- about the importance of measuring relationships. I believe that without factoring in the impact that your social media program has on your relationships (with employees, community and constituencies) you are undervaluing your efforts.

So, how does one actually measure relationships? Well, now that you asked, my book explains this with regard to most types of public relations programs. Social media being all the rage right now, I thought it would be appropriate to provide a step-by-step guide as to how to actually do it for social media programs.

Whatever type of program you want to measure, the basic technique is similar: You conduct a survey of your audience using a special set of questions designed to specifically measure the different components of relationships. You do this before and after your social media program is in place, and you do it for your organization and as many competing organizations as you can afford to. Then you compare the data before and after, and between your organization and the others, and then you know where your relationship with your audience stands and where you need to go.

The Grunig/Hon Relationship Research

Before we get to the nuts and bolts, here's a bit of background. A decade or so ago, University of Maryland Professors Jim and Laurie Grunig and Linda Hon synthesized communications and sociology research and theory into a paper published by the Institute for PR called "Guidelines for Measuring Relationships in Public Relations." Their feeling was that, amidst all the brouhaha about quantifying public relations, we were forgetting the essential truth that PR was about relationships. And so you needed to measure the impact that your efforts were having on those relationships.

Their research isolated six fundamental components of relationships -- Trust, Satisfaction, Commitment, Control Mutuality and Exchange/Communal -- and they designed and tested 75 statements to measure those components. These statements are of the sort: "This organization treats people like me fairly and justly," and, "I feel a sense of loyalty to this organization." You can see them all, and copy them for your own use, on my blog or in the Grunig/Hon paper. For an example of these statements successfully used to measure public relationships in a non-social media context, see this research by Forrest W. Anderson and Paul Raab.

Nine Steps to Measure Your Social Media Relationships

Before we start, remember that to isolate the effect of your social media program, you must begin your measurement before you launch that program. Then you'll have a benchmark to compare against later: Before Social Media vs. After Social Media. Without a Before benchmark you won't know how your social media program has changed your relationships.

Of course, you can always begin measuring after your social media program has started, and by doing so you will be able to do ongoing evaluation of your relationships. Which is a good thing. But a better thing is to isolate the effect of social media, and to do that you must compare before and after.

So use Steps 1 through 9 below to survey your audience before your program begins, and thus establish your benchmark. Then begin your social media program, and after an appropriate time, say three or six months (depending on your situation), re-survey to see what has changed.

To be sure that whatever change you find is solely attributable to the social media program, you must hold constant any other PR programs that affect your social media audience's perception of your organization. Yes, it's tricky, and it's not always an ideal world. But if you are trying to measure the effect of your new blog at the same time as the Promotions Department decides to give away A Free Cadillac With Every Purchase, then you can kiss your results goodbye.

Step 1: Define the audience for your social media program.
Social media is about conversation and engagement, so decide with whom you want to converse and engage. If you're starting an internal blog, your audience is your employees. If you're starting an IdeaStorm-type customer community, your audience is anyone likely to buy or recommend your product. If your mission is advocacy, your audience might be voters, politicians or industry influencers.

Step 2: Get a list of your audience's email addresses and/or phone numbers.
To get a representative sample you will need at least 500 names for each organization you are testing (more on that later). If you already have a list of your members, subscribers or customers, then you are ahead of the game.

If you have to purchase your list, then potential vendors vary with the type of sample you're looking for (mail, phone, web). Most lists are sorted based on demographic or title data. There are a lot more resources out there for mail addresses and those resources do not necessarily need to be survey sample companies. For email addresses, some reputable firms include Survey Sampling, e-Rewards and Zoomerang.

Step 3: Pick a survey methodology.
The Grunigs would recommend in-person surveys for the best results, but most researchers find that to be very expensive. Phone surveys are fast and provide very accurate results, but again, depending on the audience, may be cost-prohibitive. Email surveys are an increasingly accepted methodology, and for social media can be appropriate and highly reliable, since, presumably, your audience is all on email.

You may be able to piggyback on an existing survey going out to your community. If marketing, customer satisfaction, business development or anyone else in your organization is doing a survey, see if you can add a few of the Grunig/Hon statements to it.

Step 4: Select which of the Grunig/Hon statements are most appropriate to your organization.
You can probably only impose on someone for 7-10 minutes of their time, so you need to pick which statements you will include. Grunig and Hon suggest that if you want to shorten the survey, you use only the boldfaced items. Not all statements are appropriate for all organizations, so pick and chose the ones that will be most meaningful to your audience.

Step 5: Prepare your survey.
If you are using an electronic survey system like Survey Monkey, you need to create an introductory screen that explains what you are doing and how the scoring works. For instance:

In order to better understand the needs and perceptions of our marketplace, we'd like to ask you some questions. Please tell us whether you agree or disagree with the following statements as they apply to X company/organization.

Explain that 1 means "totally disagree" and 7 means "completely agree," and give them an option for "no opinion." You also need to ask them the same questions about a competing company or organization, so you have comparable data on the competition to compare to.

Step 6: Send out the survey.

Step 7. Resend the survey.
Depending on your audience it may take several tries and an incentive to get sufficient responses (I'll do just about anything for an Amazon or Starbucks card). How many is sufficient? Well, it depends on how you plan to break down your analysis, but in general plan to resurvey until you get at least a 10% return. If in doubt, talk to your local survey expert.

Step 8. Analyze and learn from the results.
Calculate a mean score for each relationship component. There are both positive and negative statements in the survey instrument, so make sure you take that into account. Compare your mean on each score to the competition. (And of course to your earlier survey results, if this survey is not the benchmark.)

Step 9. Implement your program and measure again.
If this is your pre-social media program (benchmark) survey, then implement your program now, and measure again in six months. Or, if your program has been running for a while and your analysis indicates you need to make changes, then make the changes now and let them work on your audience for six months before you measure again.

Good luck, and let me know how things go.

How to Measure Relationships with Voters, Legislators and Other Political Constituencies

by Katie Delahaye Paine

Ah Springtime! When a young man's thoughts turn to love, or basketball, or, if you live in New Hampshire, politics. Given that New Hampshire has the third largest legislative body in the English speaking world (after the British Parliament and the US Congress), and that once a year in March a million or so average citizens get together in town meetings to determine the fate of the state, you can understand why the term "March Madness" takes on a whole new meeting up in these parts.

When measuring relationships with voters, as with any measurement program, the type of system you put in place depends on your objectives and your role. Below is a list of tools you need for each type of program. One important point first...

Timing is more critical than usual
Because the length and success of voter measurement programs are so strictly defined by the election cycle, you can't afford to waste time or opportunities. Probably the most important element in any political measurement program is to have the data on hand when you need it. And you need it whenever there's a decision to be made about how much money to spend or what tactics to use. You don't need it after the votes have been counted.

Measuring PR Efforts on Behalf of a Candidate

My company, KDPaine & Partners, measured political PR for one of the country's best political PR pros, Doug Hattaway, former spokesperson for Al Gore and Tom Daschle. Doug was working on Chellie Pingree's effort to unseat Susan Collins as Senator from Maine. What Doug needed was a measurement system that would tell him which tactics were working or not working in a campaign. The challenge was to do it fast enough so the data would be useful, and cheap enough so the campaign could afford it.

The good news about working with political campaigns is that volunteers are generally plentiful, and in this case the volunteers were charged with collecting the clips from the local publications. If you don't have volunteers to do the clipping, you need to hire a local clipping company or someone like Bacons or Burrelle's to collect the clips for you. Small-town newspapers are just too important to miss. Electronic data aggregators like Nexis and Factiva do not work on local or statewide campaigns; they do a terrible job collecting clips from anything smaller than the Boston Globe. Even Cyberalert and Custom Scoop miss far too many of the small town weeklies.

One we had the clips about both Pingree and Collins in hand, we began to analyze them for positioning on key issues, tone of coverage, type of article, subject of the article and who was quoted. We compiled them monthly and issued a report that included charts like this (click the charts to see them bigger):

 

 

 

 

We could quickly identify the tactics that were most successful. For example, a visit by Hilary Clinton in support of Pingree was far more successful in garnering visibility for Pingree than any other action. However, Pingree's visit to Washington in support of her health care reform legislation, while it didn't get anywhere near the visibility, was more successful in positioning her as "the health care candidate."

The analysis also revealed weaknesses in Pingree's campaign and strengths in Collins' campaign. While Pingree was almost entirely focused on her health care message, Collins was making points with the voters on the environment and campaign finance reform.

Additionally, the research enabled the campaign to track the correlation between visibility and contributions. By tracking Pingree's share of exposure over time against the contributions over time, the campaign could determine the level of additional exposure necessary to generate the requisite contributions.

In the end, the earned media that was generated by Hattaway's PR efforts narrowed incumbent Collins' lead substantially.

Measuring Lobbying Efforts

We ran into a government relations person for a major organization who wanted to know how to measure the impact of his efforts. He was trying to get a bill passed that would be highly advantageous to his company. We began to design his "dashboard," and asked him what success meant. He got as far as "If the bill passes, I'm successful." We pointed out that if the bill failed it would be too late to do anything about it.

So what he really needed was to monitor the tone of conversation around the bill in question so that he could tweak his PR program according to its progress. We recommended that he establish a system that gives him feedback all along the way.

There are several ways to do that. The easiest method is to track who is saying what to the media or in local speeches about the bill. Are key influentials supporting it or trashing it? Or is no one talking about it at all? More important to track, however, is the bill's progress through the legislative process. Is it being heard, tabled, or moved? Who are the cosigners to the bill and are they actively promoting it? Is the opposition gaining or weakening? What do bloggers say? What's being discussed in news groups? Are there any list serves you need to track?

Some of these data points can be gathered via media analysis, others should be part of your regular legislative tracking process. Still others may require a poll of the constituency to determine the level of support, and the perceptions of the voters and/or legislators.

If at all possible, a once-a-year survey of your key legislators is recommended to test and evaluate the health of the relationship between your organization and the elected or appointed officials you are striving to measure.

 

Measuring your Relationship with Publics, Both Active and In-Active

Your constituencies come in several forms, from students to senior citizens. You should measure relationships with as many different stakeholders or constituencies groups as you can afford. It may be useful to segment the different publics by gender, age, length of time in the area and political leanings so you can identify any pockets of opportunities or threat within the community.

Clearly active stakeholders who are most likely to protest, boycott or otherwise cause trouble are the most important to understand. However, inactive stakeholders that are clueless about your programs and therefore don't care and won't get engaged can be just as dangerous.

We recommend using the Grunig relationship model to test your relationships and in particular the level of trust that each public feels towards you. Trust is a key component of success in any political campaign. If it's not there, or if it is recently lost, you will have a significantly harder time achieving success. Alternatively, if your constituents trust you, as do those of Alan Greenspan or John McCain, you have a lot more leeway with those publics. Therefore it is critical to establish a benchmark level of trust to begin with. Subsequently you should conduct regular trust/relationship measurement studies to gauge the level of trust and engagement over time. Conduct these studies as often as possible so you can tie any changes in the trust and relationship scores to actual actions you may have taken.

The Ultimate Measure: Votes

There's a ton of great data in any set of election results. The challenge is to use it to make more informed decisions. When I ran for town council here in Durham five years ago, I knew that my efforts to go door-to-door in specific communities paid off since the number of votes I received from those communities far exceeded my expectations. I also knew that my signage (bright purple and white with the slogan "No Paine, No Gain") was successful since I had the highest name recognition, and ultimately the highest vote count of all the candidates.

In larger elections, there's far more granular data to be gathered, and much of it is readily available online. Using this data you can frequently identify the specific areas or pockets or demographics you've targeted and thus measure your success in getting those groups to vote and/or vote your program or candidate.

The Media Integrity Index

A call for weighting media coverage based on trust and credibility.

by Katie Delahaye Paine

In this article I suggest that media analysis programs would more accurately reflect articles' influence on readers if the integrity of the media outlet was taken into account. This could be achieved by developing a standard measure of media integrity which would used to rate different media outlets for different shareholder groups. The resulting Integrity Index would be used to weight coverage, thereby achieving a more accurate measure of media coverage's impact on consumers.

To understand why an Integrity Index is needed, consider:

Item #1
Why measure what no one believes?

Mazen Nawahi, in an impassioned speech at the 2008 Dubai Measurement Summit, raised the issue of journalistic integrity and the degree to which the integrity of a particular media outlet should be accounted for in any measurement program. Put another way, if:

1. Everyone knows that a given media outlet is going to print whatever a company sends it because they always do or because they're a major advertiser, and
2. Everyone also knows that they will never print stories that don't agree with the governing dogma,

then why would anyone believe anything that was printed?

And if all your media coverage in a particular outlet lacks credibility, why would you include it in your measurement results? For example, in 22 years in business I have yet to have a client want me to include The Weekly World News in its media list. So how do people justify measurement programs that include stories that do nothing to achieve their goals?

Item #2
Credibility is in the eye of the beholder.

Suppose a highly credible blogger gets into a mud-wrestling contest with the blogging equivalent of a pig. The contest attracts a lot of attention, a lot of arguments fly back and forth, and a lot of dirt gets thrown around. Now for the people that are into farm animals, the arguments of the pig might be very credible. However, people interested in learning something of value professionally will no doubt pay more attention to the words of the credible blogger. But the real question is: Do you weight coverage of them both equally in your media analysis reports?

How to correct for influence?

Taking examples of this sort into consideration, it is easy to see why some sort of authority or influence weighting of coverage would be of value. There has been a fair amount of research done in this area. For several years, Angie Jeffrey at VMS has been studying various weightings of media coverage to determine which factors most directly impact sales. Her research first found that share of discussion was effective, but she postulates that advertising rates are more telling because they reflect the importance of the various media. (See this paper and this paper.) I think it's a step in the right direction, but I generally frown on the use of advertising rates (and of AVEs), so I'm skeptical.

The Media Integrity Index: Why not rate coverage on integrity as rated by stakeholders?

I suggest that it's not the ad rate but the integrity of the publication that most determines consumer credibility and thus drives consumer action. And I'm suggesting that to rank a media outlet in importance, we should ask our stakeholders how they perceive the media outlet in terms of integrity.

I raised the question on Twitter and got some interesting responses:

• "It's all relative... What is good for one is bad for the other. It has to be specified for a purpose, customized."

• "Integrity or perceived integrity? Seems there's a difference there that's highly subjective." --Ryan Anderson

• "Integrity is the measure of perceived relationship (believability) of one-to-many listeners. So measure relationships. I would suggest looking at the way political integrity is measured. It would be a perceived metric, measured by polling audience. --Videodred

• "What's the goal? What's the client want it to say? In the case of journalists, what's the relationship?

• "Integrity has to be measured parallel to influence as audience determines credibility of source." --Mike Maney

So in one way, my proposed Integrity Index has to start with the goal of the coverage. What is it that the company or organization is trying to accomplish? Consider it this way: If you don't care who or what your brand is associated with, and you want exposure pure and simple, then you don't care about the integrity or credibility of whatever media outlet is talking about you. On the other hand, if you're trying to establish a reputation, or build brand loyalty and trust, or trying to reach an audience with certain media preferences, then the credibility of media to your audience should matter a great deal.

How to determine integrity?

So how do we determine which media is trustworthy and credible, and which journalists have or do not have integrity? The wrong way is to look at a list of media and assign weights or values to each one yourself. What you think doesn't matter. All that really matters is what your customers or stakeholders or members or constituents think. So you need to ask them. The best way to do that is to use the Grunig Relationship Instrument.

Now you probably don't need to ask them about all 150 publications on your media list. Remember: Never ask a question about something that can't be changed. If The New York Times is what your boss' boss reads every morning, there's no chance in hell you'll ever take it off you top tier media list. Even if your target audience is nine-year-old girls. So start with a list of suspect publications, blogs or any other type of media outlet.

Remember that different stakeholder groups may rate a given media outlet differently. As I pointed out above, integrity is not a fixed standard. For instance, we would expect Republican viewers to rate Fox News and The New York Times differently than Democratic viewers. It is conceivable that you will want to generate different Integrity Indexes for different stakeholder groups.

Below are some survey questions adapted from the Grunig Relationship Instrument. You may wish to develop your own, based on the specific components of relationships you wish to measure (refer to this paper). Ask your stakeholders whether they agree or disagree with each statement as it pertains to each media outlet, then use the responses to rate the outlets.

1. This media outlet treats people like me and organizations like mine fairly and justly.

2. Whenever this media outlet makes an important decision, I know it will be concerned about people like me.

3. This media outlet can be relied on to keep its promises.

4. I believe that this media outlet takes the opinions of people like me into account when making decisions.

5. I feel very confident about this media outlet's skills.

6. This media outlet has the ability to accomplish what it says it will do.

7. Sound principles seem to guide this media outlet's behavior.

8. This media outlet does not mislead people like me.

9. I think it is important to watch this media outlet closely.

10. This media outlet is known to be successful at the things it tries to do.

Industry-wide integrity standards?

Now, the logical question is, "Why aren't we doing this as an industry?" Shouldn't we be factoring in credibility based on some industry-wide standard? It certainly is a more accurate weighting factor than simple eyeballs or ad cost. But the reality is that your stakeholders aren't going to be identical to my stakeholders, and what matters is how a significantly valid sample of your stakeholders feel. It would be awfully complex and difficult to set industry-wide standards for many different stakeholder groups.

On the other hand, if the PR industry wanted to take on a project to try and accurately weight publications based on their integrity, the world would most definitely be a better place. And it is not far-fetched to imagine an industry-standard Integrity Scale or Survey that could be used to determine the Integrity Index for different stakeholder groups.

Are We Engaged Yet?

Social Media Measurement

 

Engagement in social media: Web stats, visitor behavior, and relationship theory.

by Katie Delahaye Paine

This article is condensed from a paper submitted to the 11th Annual International Public Relations Research Conference.

"If we can put a man in orbit, why can't we determine the effectiveness of our communications? The reason is simple and perhaps, therefore, a little old-fashioned: people, human beings with a wide range of choice. Unpredictable, cantankerous, capricious, motivated by innumerable conflicting interests, and conflicting desires."

Ralph D. Paine, Publisher, Fortune Magazine. 1960 Speech to St. Louis Ad Club

Modern technology has come up with many good ways to measure what human beings read, watch and see, but comparatively few ways to measure -- as my father said half a century ago -- those "unpredictable, cantankerous, capricious... conflicting interests and conflicting desires." The recent rise in the influence of social media has turned the entire communications paradigm upside down. Counting column inches and eyeballs is irrelevant when a single YouTube video enjoys a larger audience than Monday Night Football, the average consumer is bombarded with 5000 messages a day, and 90% of CEOs say they are dissatisfied with how their CMOs measure results.

The basic problem is that we have years of research that says that if you "expose" 1 million consumers to a message (or buy 20 GRPs) you will sell X number of cases of shampoo, soda or soap. We have no data that says if 1 million people download your YouTube video, you'll sell any shampoo at all. What we now want to know is how social media affects user's behavior.

Engagement: The Relationship Between the User and the Brand

Like most other buzzwords, "engagement" has come a long way from its original meaning of "an agreement to marry." Essentially, it started with the notion that a website or a blog was "engaging" enough to get a reader to begin to develop a relationship with the brand.

As more and more advertisers and media types realized that hits really do stand for "How Idiots Track Success" and that even unique page views were suspect (given the enormous variation in such statistics), people began to speak of measuring engagement--not just how "sticky" the site was, but the extent to which it enhanced the relationship between the user and the brand. Advertisers now want to measure a site's ability to create an experience that earns a visitor's loyalty and, with luck, its business. As a result "engagement" now means everything from the number of times that a visitor returns to the site to the time spent online.

Another way to think of engagement is as the fourth step in a five step process that the individual user goes through:

1. Finding, usually search
2. Lurking
3. Participation
4. Engagement
5. Relationship or outcome.

Engagement According to Scoble

Popular blogger Robert Scoble (2006) has suggested that engagement is a valid measure of user interaction and authority of Internet-based social media channels. That is, engagement is a way to determine whether you are really having a dialog, or you are just yelling ever more loudly. His premise is that by measuring activity on a blog or social media website as a sign of engagement, you can predict users' behavior. In other words, if they come back to a corporate blog over and over again they'll eventually buy. If it's a YouTube video, if they watch and rate it or comment on it, they are more likely to pass it on to their friends and maybe even take some other action as a result.

Brian Haven of Forrester Research picked up on Scoble's premise and proposed measuring engagement based on a variety of tangible and intangible factors including links, track backs, comments and the frequency sentiment and tonality of comments. He defines engagement as the level of involvement, interaction, intimacy, and influence an individual has with a brand over time:

• Involvement—Includes web analytics like site traffic, page views, time spent, etc. This essentially is the component that measures if a person is present.
• Interaction—This component addresses the more robust actions people take, such as buying a product, requesting a catalog, signing up for an email, posting a comment on a blog, uploading a photo or video, etc. These metrics come from e-commerce or social media platforms.
• Intimacy—The sentiment or affinity that a person exhibits in the things they say or the actions they take, such as the meaning behind a blog post or comment, a product review, etc. Services such as brand monitoring help track these types of conversations.
• Influence—Addresses the likelihood that a person will recommend your product or service to someone else. It can manifest itself through brand loyalty or through recommendations to friends, family, or acquaintances. These metrics mostly come from surveys (both qualitative and quantitative).

Peterson's Engagement

Web Analytics expert Eric Peterson, author of Web Analytics Demystified, Web Site Measurement Hacks, and The Big Book of Key Performance Indicators, has proposed alternative measures of engagement based on Web metrics. Peterson suggests that if you want to measure engagement you need to measure stats like the following:

1. Percent increase or decrease in unique visits
2. Change in page rank - e.g., a list of the top ten most popular areas and how it has changed in the last week
3. How many sessions on the blog or website  represent more than five page views?
4. In the past month, what percent of all sessions represent more than five page views?
5. Percent of sessions that are greater than five minutes in duration
6. Percent of visitors that come back for more than five sessions
7. Percent of sessions that arrive at your site from a Google search, or a direct link from your website or other site that is related to your brand
8. Percent of visitors that become a subscriber
9. Percent of visitors that download something from the site
10. Percent of visitors that provide an email address

The problem with Peterson's metrics is that for most organizations, that data is only available on their own site, not on competing sites, so there is no way to conduct a benchmark to understand how "engaged" visitors are with one's own site vs. the competition.

While both Peterson and Haven contribute important ideas to the engagement discussion, I suggest that measuring engagement necessitates following the actions and desires of the customer.

An Engagement Index? Not Yet.

There is no such thing, yet, as an engagement index, but there has been a lot of talk about the possibility. Both Scoble and Peterson suggest that their metrics could be reduced to a single index, but they don't say how.

(Others have written excellent discussions of engagement, see, for instance, Steve Bridger's nfp 2.0 blog.)

Most of the discussion on the topic is centered on the necessity for advertisers to quantify the impact of their online advertising. Microsoft's new black box "Engagement Mapping" is designed to make advertisers on Microsoft websites more comfortable with their data (see our Measurement Menace Award for this month). Comscore and Nielsen's efforts are designed to give more meaning to the numbers they provide advertisers.

Unfortunately, metrics that make advertisers happy are not necessarily very useful for other communications functions. As internal and external communications functions become more involved in social media, they too need a way to measure engagement, but numbers from Microsoft, Comscore and Nielsen are only available for large consumer sites, not corporate blogs. More problematic is that those numbers do not factor in the newer more popular social networking sites like Facebook, YouTube and Twitter.

Engagement is a Relationship

I suggest that "engagement" is just another way to say "relationship, but a minor one." So, to the metrics suggested by Scoble and Peterson, I suggest we add those from relationship theory. At some point, you just need to come right out and ask your audience:

• "Do you trust us?"
• "Are you committed?"
• "Do you believe that we are committed to you?"
• "Do we interact with you only out of necessity or a sense of reciprocity? Or are we working together to see each other succeed?"

In short, we suggest incorporating the concepts of the Grunig Relationship Instrument into the engagement measurement.

Unless one incorporates relationship measurement into the mix, you end up with just data rather than insight. Because, while you can track behavior with increasingly accuracy, all the web metrics in the world may not answer the fundamental question of "Why?" "Why did they stop coming to your site?" "Why are they spending less time there?" Or, more critically, "Why are they buying less?" Without the true understanding of the nature of the relationship, you won't be able to do anything to improve once you find out what the problems are.

Which leads me to some final, unanswered questions: What is the difference between engagement and relationship? In fact, do we really know that engagement is something distinct -- distinct from web stats and distinct from relationship theory? Suppose we do use Grunig's questions to measure engagement, how do we know we are measuring "engagement" rather than "relationships?" If you think you know, please let me know.

Why Are Political Lawn Signs Like YouTube Downloads?

The Paine of Measurement

How online measures of engagement have predicted recent primary results.

Ever wondered what the effectiveness of political lawn signs is? Supposedly, every lawn sign represents six votes for the candidate. Or maybe ten votes, depending on what you read. And there's a theory in political circles that if you can get someone to put out a lawn sign, then that person is committed enough to not just vote for you, but also to encourage his or her friends to vote for you as well. So, each additional lawn sign means more than just one more vote, it means more of something even more valuable and a lot more difficult to pin down: more loyalty or commitment or what we in communications call engagement.

To my knowledge, no one has ever done a scientific study of how lawn sign displays influence voting habits. But my completely unscientific study of New Hampshire lawns this fall more or less predicted the outcome of our First in the Nation primary: Everywhere you went there were lots of Obama and Ron Paul signs, and both did much better than the polls predicted.

Now let's transfer this scenario into the world of social media. Can online measures of engagement predict votes? I argue that they can and have done so recently:

• Our YouTube study of candidates showed Obama having a significant lead over Hilary Clinton both in terms of viewership, and in terms of the number of videos that were rated by viewers.
• On Facebook we noted that there were some 500,000 plus groups supporting Obama, compared to Hillary's 100 or so. (In fact there are far more groups opposed to Hillary than there are those in favor.)
• In terms of Facebook's US Politics application that has Facebook voters register their opinions on a variety of topics as well as on the candidates themselves, Obama has consistently maintained a 50- to 60-point lead over Clinton.

And in the end, Obama did better in the primaries than the early polls suggested. The primary results have proven that Obama has a stronger than expected following, as hinted at by the strong online engagement we found.

The point here about engagement is bigger than just politics. How and why is engagement a stronger or different measure than just impressions? If, by joining a group, rating a video, or following someone on Twitter, you are actually thinking or behaving differently than if you just viewed an ad or a message, then measuring these signs of engagement is critical to every marketer. In order to hang on to advertising dollars, media companies will need to provide this data. And the good news is that the data is there, they just need to release it.

And finally, I can't help but see engagement as a kind of bridge between measuring outputs and measuring relationships. (Most of you are aware of my recently published book Measuring Public Relationships, learn more here.) If you measure an output like impressions, you only know what has happened to an audience. But if you measure engagement, you are measuring what is done by an audience as the result of their relationship with your output. How does measuring engagement fit in with measuring relationships? That's a good question, let me know if you have the answer.

Here's wishing you large measures of success,

 

What Is a Social Network and Why Does it Matter?

Social Media Measurement

 

Or... What's the ROI of my living room?

by Katie Delahaye Paine

As everyone who has talked to me recently knows, I'm a serious social media evangelista. And as I travel around I'm constantly confronted with business people who say: "Social media? I don't get it! Who has time? Why should I bother?"

My simple answer has always been that blogging (or Twitter, or Facebook) is a way to engage in a conversation with your customers or your employees.

(And if, Mr. CEO, you do not want to engage your customers or your employees, you deserve to be fired. If not shot.)

But I've been told that that attitude isn't particularly helpful when you're talking to people who think Facebook is still "just a college thing," and who think "twittering" must be something dirty.

So here's my new explanation (a bit long-winded, but, please, bear with me here):

When I built my house, it was designed to be the capital of social capital, with a huge living room and kitchen and dining room so that lots and lots of people could gather there. As that's just what happened:

• When I moved in, I threw a party and lots of people showed up. As always happens at parties, different groups of people gathered in different nooks and crannies of the house to talk about what was interesting to them.
• A few months later, my best friend had her wedding at the house, saving her the expense and aggravation of renting a hall. Lots of people showed up to wish the newlyweds well and to share their stories and experiences.
• A little later, we had a benefit for the Durham Public Library. The author Joyce Maynard spoke, the place was packed, and the library more than tripled its mailing list.
• A lovely lady named Carol Shea-Porter decided to run for Congress and we did a fundraiser at my house. She told the crowd, "I'm not asking for your money, I'm asking for your votes. If I have your votes, I don't need the money." She was outspent 5 to 1, but we now call her "Congresswoman Shea-Porter."
• Later on, we had a house concert with a hundred or so fans of a local musician. He sold lots of his new CDs and added to his mailing list.

So are you getting my point? Social Media is just a great big version of my living room. Any social network -- Twitter, Facebook or MyRagan -- starts off as one big noisy place. But soon, people of like minds and like interests start to find each other and sometimes they spin off and form their own separate groups.

So What's the ROI of my Living Room?

Which gets us to measurement. The hot topic right now -- and what everyone wants to know -- is: How do you measure the ROI of the effort you put into these groups?

To answer that, let's go back to my living room. The bride and groom's goal was to save money. The Library's was to build their mailing list. The politician wanted votes and the musician wanted to sell CDs. If they wanted to measure the ROI of their events in my living room, they'd compare their investment in effort to the particular return that was important to them.

And What's My ROI for Social Media?

I, on the other hand, use social media, and (often) my living room, to satisfy my thirst for knowledge and intellectual stimulation. Take Twitter: Do I appreciate the fact that traffic on both my blog and my website has picked up since I started Twittering? Absolutely. But mostly I Twitter because it makes me smarter. It allows me to follow interesting people that I wouldn't ever meet in my living room, and who would never read my blog.

I mess around in Facebook because it's a great way to share stuff with my friends and colleagues and it's a huge time saver when you're trying to pull together an event.

I blog because I like having conversations with people – about my business, my services, and the world as a whole.

And that ends my longwinded explanation of why you or anyone might want to bother with social media. It depends on you and your goals. And as for measurement, it's the same as for any marketing effort; first be clear about what you're trying to achieve, then go about measuring it.

Ten No-Cost Ways to Measure Online Engagement

by Katie Delahaye Paine

1. Set up Google Analytics on your blog to find out how many repeat visitors you have. How many pages per visit do they check out? How many go back more than 3 times a week? How many go back and spend more than a second or two on the site?

2. Post a vizu poll on your blog and see how many people respond.

3. Go to xinure and enter the URL of your choice to find out how well it is doing in search engines, links, social bookmarks and a whole bunch of other stats.

4. With many of the leading blog providers like TypePad, check your stats to find out how many people have subscribed, and how many visits per day you're receiving.

5. What's the Conversation Index (the ratio of postings to comments)? In the blogosphere any comment is a good comment because it shows that people are engaged enough in what you are saying to take the time to respond.

6. If you have posted a video on YouTube, or a photo on Flickr, check to see how many people have rated it, and/or commented on it.

7. If you have a presence on Facebook, how many people have joined your group?

8. Ask a question on Facebook and see how many people respond.

9. If you're on Twitter, how many followers do you have? More importantly, how many responses do you get when you ask them a question?

10. Set up a Technorati, Sphere or Icerocket search to find out whether people are writing or talking about you.

The Interpretation of Dreams: Dreams as Public Relations Measurement -- I Dream of Hillary, I Dream of Barack

Many public relations professionals find that media analysis or survey research are tools sufficient to measure their programs. The more adventurous bring in external data and more esoteric measures of success like donations or number of new memberships or even lives saved. In politics, KDPaine and Partners has recently used YouTube video counts to predict primary results, and our Ms. Paine has ruminated on the possibility of using the number of political lawn signs as an election predictor. Not until now, however, has anyone used dreams to measure political progress.

In this week's New Yorker, Ben McGrath writes about Sheila Heti and her new blog "I Dream of Hillary... I Dream of Barack." This blog, a repository of reader-submitted dreams about the candidates, can be interpreted as a rough poll of interest in those candidates. As Mr. McGrath says, "...what if the recurrences of Presidential candidates in people's dreams were meaningful in the aggregate?" As of the writing of the article, "..Obama's edge in the over-all dream count... was roughly equivalent to his lead in the latest Gallup poll."

Ms. Heti's blog has been so successful she plans to open an I Dream of McCain section, so in the future we will have bipartisan dream data to go on. --Bill Paarlberg, Editor, The Measurement Standard

KDPaine & Partners Releases New Book on Measuring PR and Corporate Communications

Your Measurement Reading List

Measuring Public Relationships Makes Measurement Simple

Measuring Public Relationships, The Data-Driven Communicator's Guide to Success by Katie Delahaye Paine, 228 pages, paperback. ISBN 978-0-9789899-0-3. Order online or by phone: 603-319-1047.

BERLIN, NH, Dec 12, 2007-- The first how-to book on measuring relationships for public relations professionals was released today by KDPaine & Partners, the Berlin, NH-based leader in social media and public relations measurement. Measuring Public Relationships, The Data-Driven Communicator's Guide to Success was written by KDPaine & Partners' CEO, Katie Delahaye Paine.

Paine takes a "cookbook" approach, providing specific steps to measure all forms of public relationships, from social media measurement to tracking relationships with local communities, industry analysts and social networks. It relies heavily on the relationship theories of James and Laurie Grunig and Linda Hon but translates the theory into an easy to read practical guidebook. The book has been well received by academics and professionals alike who have called it "the must-have practical guide to hands-on PR measurement." Emphasizing the role and evaluation of relationships, this book provides every public relations professional with step-by-step research procedures for measuring programs, improving results, and managing relationships.

"It really is an excellent book," said Professor James Grunig, calling it "lively, engaging, accessible, wise, and candid... a remarkable compendium..."

"Katie Paine is the most practical consultant and educator on public relations research that I know. She is brave enough to link PR to return on investment demands and smart enough to provide marketers and PR professionals with a wide range of useful research tools. If there is cost effective way to do research, Katie Paine will tell you about it," said Clark Caywood of the Department of Integrated Marketing Communications at the Medill School at Northwestern University.

"By force of prose and experience, Katie Paine level-sets the practical and now critical disciplines of media and communication measurement. A student of Grunig theory and a champion of the almighty relationship, Paine plops on every communicator's desk the best new reference for measuring the intangible. Read it, and you'll count your successes," said Alan Kelly, CEO & Founder, The Playmaker's Standard, LLC and author of The Elements of Influence. Adding that it's "...the best new reference for measuring the intangible."

Measuring Public Relationships is available at a 25% discount until January 31st, 2008 when ordered directly online or by calling the KDPaine & Partners' office at 603-319-1047.

How To Evaluate Events and Sponsorships

Sponsorship Measurement

Six basic steps, five final lessons and four additional resources.

By Katie Delahaye Paine

"Live from the KDPaine & Partners/Verizon Wireless Arena we have the KDPaine & Partners' Battle of the Sponsorship Suppliers, underwritten by Orville Redenbacher's Microwave Popcorn, the official snack food of KDPaine & Partners; by Starbucks, the official fuel of KDPaine & Partners; and with additional support from Staples, the official office supply company; FedEx, the official overnight delivery system; Diet Coke, the official diet drink; and Catalina, the official pet of KDPaine & Partners."

Perhaps not these exact words, but something like them has passed your ears at least a dozen times in the last few years. From Little League teams to the World Cup, from naming arenas to naming road races, sponsorship of events is now one of the marketing methods of choice. As communications is increasingly impersonal and electronic, the chance to actually touch, feel and experience the brand is ever more important. Increasingly, purchasing is neatly divided into two categories -- on-line and experiential. More and more decisions are based on familiarity with the brand and ease of purchase.

Now, I'm an avid Amazon fanatic, but I still go into Borders' brick and mortar store and end up buying something. Why? Because I get to touch and feel and browse and listen to products. If I need something in a hurry, I go online. If I want "retail therapy," I go to Borders. That's experiential marketing. Marketing that transcends the product and the media and allows direct interaction with the brand.

A 2006 study by Jack Morton Worldwide showed that live event marketing -- experiences where consumers interact with products, brands or "brand ambassadors" face-to-face -- are among the most effective ways to influence coveted consumer audiences. The study, an online survey of 2,574 consumers ages 13-65 in the top 25 US markets, confirmed that this increasingly important marketing medium resonates strongly across all demographic and product categories.

Which is why total spending on experiential marketing is expected to reach $15 billion by the end of 2007, an 11% increase over last year, according to the latest IEG research report. Companies are now allocating on average 17% of their budgets to sponsorships, compared to just 13% last year. However, 76% of companies spend less than 1% of their total sponsorship budgets on research, and in a shaky economy, if you don't measure it, you can't show ROI. And if you can't show ROI, chances are good that the program will be killed.

Another factor in the measurement of experiential marketing is that simply "reaching large numbers of eyeballs" is no more relevant a goal than is "increasing market share in the buggy whip market." Therefore, new alternative metrics -- like relationship measures, engagement indices and cost per touch -- are increasingly demanded.

We suggest that there are six basic steps to developing a solid measurement program for your events and sponsorships.

Step 1: Be clear about what outcomes your communications program is designed to achieve.

As with any effort, you can't start to measure success until you know what success means for you. For any given event the objectives might be:

• Percent of attendees more likely to purchase
• Percent of attendees remembering the brand
• Number of qualified sales leads generated
• Conversion rate of attendees
• Total potential sales = [number of attendees] x [conversion rate] x [average sale]
• For press events: number of key editors and analysts attending, percent of attendees writing or quoted on the issue, total exposure of key messages in resulting press.

Each objective, of course, requires a different type of measurement. Some types isolate the impact of PR from other communications efforts better than others. And the best objectives are specific and measurable.

Step 2: Determine criteria -- quality as well as quantity

Once you've agreed upon your objectives, establish the specific criteria of success that you will measure. If your objective is awareness, the criterion might be the percentage increase of unaided awareness of brand or product. If your objective is to sell product, the criterion might be the incremental sales after a particular PR or promotional program took place. Consider those numbers that best reflect the health of your business, or that best represent characteristics that most affect your business:

• Increase in awareness
• Increase in preference
• Increase in purchase intent
• Increase in customer loyalty
• Percent improvement in customer experience<