Katie Delahaye Paine's Measurement Blog

Another nail in the "counting eyeball" coffin

First let me say that I actually trust comscore data far more than most numbers i see. but I was reading their most recent update in their efforts to more accurately count online viewership and discovered this astounding statement:

• One of the earliest – and perhaps most obvious – findings along the way is that we’ve seen ample evidence of the foibles of server-side analytics for measuring the number of unique visitors (i.e. people) who visit a site. Due to many inflationary factors, including cookie deletion and rejection, bot and spider traffic, and site visitation from multiple locations, we’ve found clear and direct evidence that web site servers routinely overstate actual people counts by a factor of two and higher. In particular, the inflationary impact of cookie deletion is consistent with independent research from a variety of other research companies, including Forrester, Belden, Jupiter and Nielsen. The inflation in server data has also now become apparent to the IAB (see their Audience Reach Measurement Guidelines) and academics such as Max Fomitchev, an assistant professor of Computer Science & Engineering at Pennsylvania State University who conducted an exhaustive study and concluded:

  “Cookies are about just as inaccurate in estimating unique visitors as unique network addresses. This is the new and unrealized fact in the industry that has a direct impact on Internet advertising as currently reported unique visitor / core audience size numbers tend to overestimate the true audience size by a large factor (7-30, depending on the visitation frequency and the sampling period).”

As we have reached critical mass among websites participating in 360 programs, we have found some astounding statistics. In the month of December alone, we have seen over 1.5 Billion unique cookies just in the US. For instance, we have found 259 million cookies for AOL. That is a reach in excess of 125% compared to a true reach of 54%. These statistics mean that it is imperative to develop sophisticated methods to remove inflation caused by cookie deletion, rejection, multiple browsers, multiple devices and multiple locations of access such as home and work.

I totally applaud their honesty but privately wonder how many many too many decisions have been based on just the sort of inflated numbers they describe.

Social Media is Making Measurement More Vertical

Amber Naslund's (@Ambercadabra's) brilliant post about getting started in Social Media Measurement got me thinking. Her premise is dead on, that measurement of social media isn't the issue, and it's measurement in general. I can't tell you how many companies either don't measure any of their marketing programs or don't KNOW how they measure their marketing programs. And her advice is terrific – if you are in the business of selling stuff, in other words, if you are either a B to B or B to C Company. But for the other 50% of PR people out there that work for non-profits, educational institutions, government entities, associations etc. most of this good advice is irrelevant.

This brings us to another reality of social media measurement. How and what you measure, and how you define ROI is very much dependent on what vertical market you play in. If you're in higher ed, you're "selling" ideas – the brilliance of your faculty as industry experts, and/or you are in the business of recruiting – be it faculty or students or parents of students. You have lots of different audiences to deal with and the measurement playing field looks very different.

If you're a non-profit, you may be selling the cause, but your metrics are, again, different. You want to measure not just donations, but engagement in the cause, and the enthusiasm of your fans, followers and donors.

If you're a government entity, your mission may be to educate, to lower cost, to "sell" a concept – and thus you are measuring not "leads" or "conversions" but rather the degree to which people buy into whatever it is you are trying to convince them of.

Last summer, I gave two workshops back to back. The first had the full spectrum of industries represented. Government, media, agency, consumer packaged goods, services, education etc. The other just happened to be entirely populated with people in charge of communications at educational institutions. The content of both workshops was the same, but the satisfaction levels were dramatically different. After the second one, every attendee went away happy, with tools they could use, and full of ideas of how to better measure his/her success. The feedback on the first one was all over the map, with some people very disappointed, and others thrilled. What I learned was that whether you're teaching measurement or implementing it, you have to tailor your thinking to the industry you're dealing with.

This may seem to be an obvious conclusion, but one only needs to read a few posts to hear a dozen people express the need for "One good social media metric we can all use." There is no such thing.

Pat Robertson, zombie metrics and other things that just don’t die

 

Until he popped up on cable news this week, suggesting that Haiti's recent disaster was all their fault for "making a pact with the devil," I had assumed that tele-evangelist Pat Robertson was sufficiently irrelevant (and crazy) that he had passed quietly from our lives. But there he was, generating publicity for a suggestion that was so heinous that even conservative talk show hosts repudiated it.

That's kind of how I feel about AVEs. Back in November, I wrote with unbridled exuberance about the fact that IPR's measurement had essentially killed off the AVE, by voting overwhelmingly to reject the theory, concept and practice of calculating PR based on what the equivalent ad space would cost. And now, just when I thought we were safe from this insidious notion, its back, this time disguised as "Weighted Media Costs," and the very IPR that rejected the notion last fall, is now promoting it.

And while media cost weightings may not be as heinous as Pat Robertson, there are at least half a dozen things wrong with this notion.

1. Renaming a bad idea seldom make it better. Whether you call it AVEs or WMC, it is still valuing PR for its outputs (i.e. clips and publicity) rather than for the relationships it builds, education or reputation it builds or cost savings it generates.
2. It is a metric promoted by and only available from VMS, a commercial supplier of clippings. Our neighbors to the north at CPRS ran into a similar controversy when they launched their MRP system several years ago, but they at least put out an industry-wide RFP for solutions before they crowned News Canada's system as "the standard."
3. The paper uses correlations to sales as a way to compare three different ways to "evaluate" PR. Specifically, it factored in the tonality of each article. It then looked at actual story counts, impressions and WMC - refined Story Counts. But for the WMC calculation the researchers determined how much of each story was "owned" by the brand under study. This introduces another factor to the analysis, so I'm not clear as to whether the better results are due to this "owned" factor or to the actual WMC. If I did a similar analysis using what we call dominance/prominence – in other words was the story exclusively about the brand, or was the brand mentioned in passing, or in an equal way with other brands -- as well as where in the story the brand was mentioned – the headline, top 20% of the story or bottom 80% -- wouldn't I get the same results?
4. Because a great deal of influence in today's society is generated via Twitter and other on-line media properties not covered by SQUAD, Nielsen or VMS, it gives greater to weight to mediums that are in decline and insufficient weight to media that is on the rise.
5. This takes PR in the wrong direction. PRSA recently published a great document entitled 'The Business Case for PR" that shows people how to measure their contribution to the bottom line, Increasingly, smart communicators who want to be accountable are measuring greater efficiency, cost avoidance and actual sales revenue or leads from their efforts. This is essentially says to PR people – hey, keep measuring your activity, that's what people really care about.
6. It's a distraction. Do we really need a better way to measure our activity? Shouldn't we be focusing our time effort and attention on better ways of measuring outcomes?

 

 

 

Measuring Engagement is just another term for Measuring Relationships

I was asked over the weekend to comment on Jason Fall's post on measuring engagement and it got me thinking. The reason that engagement is so important to today's marketers, and why everyone, is trying to figure out how to measure it, is that engagement is essentially the first step in building a relationship between your customers and your brand. And in this era of drive by flaming and overwhelming inundation of data and messages, an organization's relationship are what will differentiate it from everyone else. Engagement is a way to determine whether you are having a dialog, or are you just yelling ever more loudly.

If I'm managing communications for the USO and my ultimate goal is to increase donations, I know that somehow I need to form a relationship with a potential donor if I'm ever going to have a prayer of getting him or her to give me money. So let's call our potential donor/stakeholder Veronica. The USO's relationship with Veronica starts when she sees a volunteer at the airport, or when a friend suggests she become a fan on Facebook, or when she watched Stephen Colbert's show in Bagdad. All those points of contact are potential points of engagement. However, as we know, just reaching Veronica's "eyeballs" is an opportunity not engagement. Engagement begins when Veronica takes some sort of action beyond lurking that indicates a desire to have further contact. So it may be subscribing to your blog, following you on Twitter, downloading your YouTube video or becoming a fan on Facebook. I call these actions "Level 1 engagement. "

For those of you familiar with Grunig's relationship theory, at this point the relationship is pretty much pure exchange. In an exchange relationship, one party gives benefits to the other only because the other has provided benefits in the past or is expected to do so in the future. In other words, it's not much of a relationship at all. Veronica registers, and the USO sends her content that she wants. Veronica become a fan, and as a result is entered into a contest. Both the USO and Veronica benefit, but there's not much loyalty involved.

Nonetheless, that action is a step up from lurking and therefore, to the USO or to Procter & Gamble or any organization participating in social media, it may move people closer to that donation or sale. Unfortunately, first level engagement is where most relationships end. The vast majority of fans, friends and followers never go past that initial point. So Engagement Metric #1 should be the ratio of lurkers to people taking any action at all.

But let's assume that Veronica has won the prize, or is interested in the content the USO is sending her. Maybe she's found a friend with similar interest on the Facebook page. Something happens that moves her to Level 2 engagement. That's where Veronica is now engaged enough to comment on the blog, respond to a thread on Facebook, and maybe rate a YouTube video. Now the relationship is more of a communal one. -- In a communal relationship, both parties provide benefits to the other because they are concerned for the welfare of the other -- even when they get nothing in return. For most public relations activities, developing communal relationships with key constituencies is much more important to achieve than would be developing exchange relationships. Because, when the relationship is communal, you will be forgiven for mistakes, you will get past a crisis faster, people will pay more for the product and they'll recommend it to their friends.

So your second key metric is the increase in level 2 engagement over time, which you can determine by the analytics that most sites provide. What you really need to know (and can find out by analyzing your web analytics data) is: What are you doing/writing/posting that has convinced all those lurkers and Level 1 folks to go to the next step? What is convincing them to care more about you/your organization/your cause. You need to examine your activities, post by post, tweet by tweet to see what is increasing the engagement level and bringing that stakeholder into a communal relationship.

At some point, some percentage of these partially engaged fan/follower/friend will either get bored or become just passive observers and the relationship stagnates. At this point your key metric is to determine if your relationships are progressing, therefore, you need to examine the ratio between new and repeat visitors, and between those that come once and those that return more than 4 times a month.

The good news is that Veronica hasn't gotten bored, and in fact, thanks to a combination of Colbert shows, compelling videos, and good word of mouth, is ready to move to the third level of engagement -- taking action on your behalf. Somehow from all that Veronica has read, or learned about the USO from all the PSAs, stories, videos, word of mouth and updates, she is now ready to take action, i.e. volunteer to stuff care packages, or share a message with the troops or better yet donate. What this means from a relationship standpoint is that you've been able to move her to a level 3 -- at which you have added the components of trust and commitment to the relationship. In Grunig's definition:

• Trust -- One party's level of confidence in and willingness to open oneself to the other party. There are three dimensions to trust: integrity: the belief that an organization is fair and just … dependability: the belief that an organization will do what it says it will do … and, competence: the belief that an organization has the ability to do what it says it will do.
• Commitment -- The extent to which each party believes and feels that the relationship is worth spending energy to maintain and promote. Two dimensions of commitment are continuance commitment, which refers to a certain line of action, and affective commitment, which is an emotional orientation.

   

So your key metric for Level 3 engagement is the number of new volunteers, number of new contributors and the number of visitors to those pages on www.uso.org that encourage a visitor to take action. Again, what you should be measuring is the increase over time, not just the raw numbers.

By now Veronica is fully engaged, actively participating in Facebook threads, retweeting news updates from the USO and sending out YouTube videos to all of her friends. – and presumably convincing her friends to do the same. But measurement can't end there. You still need to make sure that she is satisfied with the relationship. According to Grunig Satisfaction is the extent to which each party feels favorably toward the other because positive expectations about the relationship are reinforced. A satisfying relationship is one in which the benefits outweigh the costs. So to measure this the fourth and final level of engagement you need to look at Veronica's loyalty to the cause. How often does she contribute or volunteer? Is she bringing other volunteers with her, and is she expressing satisfaction with the relationship to her friends? You can track her comments thru Social Mention, Google News or any number of monitoring tools like Sysomos and Radian 6. But ultimately at some point you will probably want to survey all the Veronica's (and Archie's) out there and really find out how they're feeling about your organization. This is where the Grunig relationship instrument comes in.

For each of the components of relationships there are a series of statements (listed below) that you can pose to your audience and ask them to agree or disagree with each one. You can do this via phone (very expensive) or mail or email, but ideally you would administer the survey prior to starting a social media campaign and then six months into it to see how you're doing on each score.

So back to Jason's post – it there one simple way to measure engagement? No, there are lots, but if you follow this path, you'll know a lot more than "are they engaged" you'll know what is increasing engagement, what is turning people off, and how likely are they to act on your behalf.

 

And if you're wondering, here are Grunig's relationship statements.

Questions to measure trust, including the dimensions of integrity, competence, dependability

 

This organization treats people like me fairly and justly. (Integrity)

Whenever this organization makes an important decision, I know it will be concerned about people like me. (Integrity; original dimension: faith).

This organization can be relied on to keep its promises. (Dependability)

I believe that this organization takes the opinions of people like me into account when making decisions. (Dependability)

I feel very confident about this organization's skills. (Competence)

This organization has the ability to accomplish what it says it will do. (Competence)

Sound principles seem to guide this organization's behavior. (Integrity)

This organization does not mislead people like me. (Integrity)

I am very willing to let this organization make decisions for people like me. (Dependability)

I think it is important to watch this organization closely so that it does not take advantage of people like me. (Dependability) (Reversed)

This organization is known to be successful at the things it tries to do. (Competence)

 

Commitment

 

I feel that this organization is trying to maintain a long-term commitment to people like me.

I can see that this organization wants to maintain a relationship with people like me.

There is a long-lasting bond between this organization and people like me.

Compared to other organizations, I value my relationship with this organization more.

I would rather work together with this organization than not.

I have no desire to have a relationship with this organization. (Reversed)

I feel a sense of loyalty to this organization.

I could not care less about this organization. (Reversed)

 

Satisfaction:

 

I am happy with this organization.

Both the organization and people like me benefit from the relationship.

Most people like me are happy in their interactions with this organization.

Generally speaking, I am pleased with the relationship this organization has established with people like me.

Most people enjoy dealing with this organization.

The organization fails to satisfy the needs of people like me. (Reversed)

I feel people like me are important to this organization.

In general, I believe that nothing of value has been accomplished between this organization and people like me. (Reversed)

Communal Relationships

 

This organization does not especially enjoy giving others aid. (Reversed)

This organization is very concerned about the welfare of people like me.

I feel that this organization takes advantage of people who are vulnerable. (Reversed)

I think that this organization succeeds by stepping on other people. (Reversed)

This organization helps people like me without expecting anything in return.

I don't consider this to be a particularly helpful organization. (Reversed)

I feel that this organization tries to get the upper hand. (Reversed)

 

Exchange Relationships

 

Whenever this organization gives or offers something to people like me, it generally expects something in return.

Even though people like me have had a relationship with this organization for a long time, it still expects something in return whenever it offers us a favor.

This organization will compromise with people like me when it knows that it will gain something.

This organization takes care of people who are likely to reward the organization.

A walk in the snow with not-so-strangers

After sitting inside for three days glued to my laptop, finishing up the last of the rewrites on version 2.0 of Measuring Public Relationships, I finally emerged to a foot of snow (well almost) and no let up in sight. I'd been watching the storms come over the mountains from my perch on School Street  
since Friday – fire in the fireplace, polar fleece aplenty, enjoying the winter as only someone raised in New England can do.

But shortly after I hit "send," shipping the last chapter off to my editor, I donned my boots, grabbed a shovel and dug my self out. I grabbed my camera and decided to see what I could see. The answer was, not much, since it was still snowing and the normally glorious views were still shrouded in gray. But I did get a glimpse of a the North Country that one doesn't often see.

I made my way up School to Spring Street,
and then took a right on Hillside a steep upward climb that almost always yields a perfect shot or two. I pass a woman, a considerably older than myself coming down the street who greets me with a smile that would light up a room. "Isn't it great out here?," she asked. "I couldn't stay inside a minute longer." I agreed with her, at which point she looked at me quizzically and said "Don't I know you? Lori, right ?" and I responded, "No, Katie." She reached out her hand warmly and said, "It's so good to see you, I haven't seen you in years, how have you been?"

Truth be told, as far as I know, I've never set eyes on her before, but the warmth of her eyes and strength of her handshake was all that mattered. I said fine, she wished me a happy New Year and we went on our way.

Next I ran into a man who was clearing his driveway and brushing snow off his car. "How far do you have to go? Can I give you a ride?" he asked. I explained that "no, thank you, I'm really out for the exercise." He explained that he thought my car couldn't make it up the hill, and wanted to make sure I could get where I was going. I thanked him profusely and continued up the hill.

Another woman, easily in her 60s was finishing up the shoveling of her very long driveway and stopped, leaned on her shovel and said, "isn't it beautiful out?" We chatted a bit, agreed that it was indeed beautiful. She wished me a happy New Year, and went on with her shoveling and I continued up the hill. After a few more good wishes along the way, I made it up to my favorite view, 
only to find that it was all pretty gray.  
So I headed down again, taking the long way home down Norway Street to the river.  

I heard a shout and take NHPR out of my ears, and there 's a man shoveling at least 18 inches of snow off his roof. "Happy New Year" he shouted at me. I wished him the same and once again, we agreed that it was a great day.

Now I'm sure it was just as great a day, and perhaps even better in Florida and California and Arizona, where my friends have been sending me tempting invitations and weather reports. But for now, the kindness of strangers is keeping me plenty warm, thank you.