Are We Engaged Yet?

Social Media Measurement

 

Engagement in social media: Web stats, visitor behavior, and relationship theory.

by Katie Delahaye Paine

This article is condensed from a paper submitted to the 11th Annual International Public Relations Research Conference.

"If we can put a man in orbit, why can't we determine the effectiveness of our communications? The reason is simple and perhaps, therefore, a little old-fashioned: people, human beings with a wide range of choice. Unpredictable, cantankerous, capricious, motivated by innumerable conflicting interests, and conflicting desires."

Ralph D. Paine, Publisher, Fortune Magazine. 1960 Speech to St. Louis Ad Club

Modern technology has come up with many good ways to measure what human beings read, watch and see, but comparatively few ways to measure -- as my father said half a century ago -- those "unpredictable, cantankerous, capricious... conflicting interests and conflicting desires." The recent rise in the influence of social media has turned the entire communications paradigm upside down. Counting column inches and eyeballs is irrelevant when a single YouTube video enjoys a larger audience than Monday Night Football, the average consumer is bombarded with 5000 messages a day, and 90% of CEOs say they are dissatisfied with how their CMOs measure results.

The basic problem is that we have years of research that says that if you "expose" 1 million consumers to a message (or buy 20 GRPs) you will sell X number of cases of shampoo, soda or soap. We have no data that says if 1 million people download your YouTube video, you'll sell any shampoo at all. What we now want to know is how social media affects user's behavior.

Engagement: The Relationship Between the User and the Brand

Like most other buzzwords, "engagement" has come a long way from its original meaning of "an agreement to marry." Essentially, it started with the notion that a website or a blog was "engaging" enough to get a reader to begin to develop a relationship with the brand.

As more and more advertisers and media types realized that hits really do stand for "How Idiots Track Success" and that even unique page views were suspect (given the enormous variation in such statistics), people began to speak of measuring engagement--not just how "sticky" the site was, but the extent to which it enhanced the relationship between the user and the brand. Advertisers now want to measure a site's ability to create an experience that earns a visitor's loyalty and, with luck, its business. As a result "engagement" now means everything from the number of times that a visitor returns to the site to the time spent online.

Another way to think of engagement is as the fourth step in a five step process that the individual user goes through:

1. Finding, usually search
2. Lurking
3. Participation
4. Engagement
5. Relationship or outcome.

Engagement According to Scoble

Popular blogger Robert Scoble (2006) has suggested that engagement is a valid measure of user interaction and authority of Internet-based social media channels. That is, engagement is a way to determine whether you are really having a dialog, or you are just yelling ever more loudly. His premise is that by measuring activity on a blog or social media website as a sign of engagement, you can predict users' behavior. In other words, if they come back to a corporate blog over and over again they'll eventually buy. If it's a YouTube video, if they watch and rate it or comment on it, they are more likely to pass it on to their friends and maybe even take some other action as a result.

Brian Haven of Forrester Research picked up on Scoble's premise and proposed measuring engagement based on a variety of tangible and intangible factors including links, track backs, comments and the frequency sentiment and tonality of comments. He defines engagement as the level of involvement, interaction, intimacy, and influence an individual has with a brand over time:

• Involvement—Includes web analytics like site traffic, page views, time spent, etc. This essentially is the component that measures if a person is present.
• Interaction—This component addresses the more robust actions people take, such as buying a product, requesting a catalog, signing up for an email, posting a comment on a blog, uploading a photo or video, etc. These metrics come from e-commerce or social media platforms.
• Intimacy—The sentiment or affinity that a person exhibits in the things they say or the actions they take, such as the meaning behind a blog post or comment, a product review, etc. Services such as brand monitoring help track these types of conversations.
• Influence—Addresses the likelihood that a person will recommend your product or service to someone else. It can manifest itself through brand loyalty or through recommendations to friends, family, or acquaintances. These metrics mostly come from surveys (both qualitative and quantitative).

Peterson's Engagement

Web Analytics expert Eric Peterson, author of Web Analytics Demystified, Web Site Measurement Hacks, and The Big Book of Key Performance Indicators, has proposed alternative measures of engagement based on Web metrics. Peterson suggests that if you want to measure engagement you need to measure stats like the following:

1. Percent increase or decrease in unique visits
2. Change in page rank - e.g., a list of the top ten most popular areas and how it has changed in the last week
3. How many sessions on the blog or website  represent more than five page views?
4. In the past month, what percent of all sessions represent more than five page views?
5. Percent of sessions that are greater than five minutes in duration
6. Percent of visitors that come back for more than five sessions
7. Percent of sessions that arrive at your site from a Google search, or a direct link from your website or other site that is related to your brand
8. Percent of visitors that become a subscriber
9. Percent of visitors that download something from the site
10. Percent of visitors that provide an email address

The problem with Peterson's metrics is that for most organizations, that data is only available on their own site, not on competing sites, so there is no way to conduct a benchmark to understand how "engaged" visitors are with one's own site vs. the competition.

While both Peterson and Haven contribute important ideas to the engagement discussion, I suggest that measuring engagement necessitates following the actions and desires of the customer.

An Engagement Index? Not Yet.

There is no such thing, yet, as an engagement index, but there has been a lot of talk about the possibility. Both Scoble and Peterson suggest that their metrics could be reduced to a single index, but they don't say how.

(Others have written excellent discussions of engagement, see, for instance, Steve Bridger's nfp 2.0 blog.)

Most of the discussion on the topic is centered on the necessity for advertisers to quantify the impact of their online advertising. Microsoft's new black box "Engagement Mapping" is designed to make advertisers on Microsoft websites more comfortable with their data (see our Measurement Menace Award for this month). Comscore and Nielsen's efforts are designed to give more meaning to the numbers they provide advertisers.

Unfortunately, metrics that make advertisers happy are not necessarily very useful for other communications functions. As internal and external communications functions become more involved in social media, they too need a way to measure engagement, but numbers from Microsoft, Comscore and Nielsen are only available for large consumer sites, not corporate blogs. More problematic is that those numbers do not factor in the newer more popular social networking sites like Facebook, YouTube and Twitter.

Engagement is a Relationship

I suggest that "engagement" is just another way to say "relationship, but a minor one." So, to the metrics suggested by Scoble and Peterson, I suggest we add those from relationship theory. At some point, you just need to come right out and ask your audience:

• "Do you trust us?"
• "Are you committed?"
• "Do you believe that we are committed to you?"
• "Do we interact with you only out of necessity or a sense of reciprocity? Or are we working together to see each other succeed?"

In short, we suggest incorporating the concepts of the Grunig Relationship Instrument into the engagement measurement.

Unless one incorporates relationship measurement into the mix, you end up with just data rather than insight. Because, while you can track behavior with increasingly accuracy, all the web metrics in the world may not answer the fundamental question of "Why?" "Why did they stop coming to your site?" "Why are they spending less time there?" Or, more critically, "Why are they buying less?" Without the true understanding of the nature of the relationship, you won't be able to do anything to improve once you find out what the problems are.

Which leads me to some final, unanswered questions: What is the difference between engagement and relationship? In fact, do we really know that engagement is something distinct -- distinct from web stats and distinct from relationship theory? Suppose we do use Grunig's questions to measure engagement, how do we know we are measuring "engagement" rather than "relationships?" If you think you know, please let me know.

Comments

[Eric T. Peterson]

Hi Katie,

I'm a little confused by your statement "Both Scoble and Peterson suggest that their metrics could be reduced to a single index, but they don't say how." I've published in my blog quite extensively on the "how" and there are a number of companies actively using my framework and variations on the calculation today.

You may want to read this post for an update on the calculation, although my work on engagement as advanced since the post was written:

http://blog.webanalyticsdemystified.com/weblog/2007/10/how-to-measure-visitor-engagement-redux.html

You may also want to read the work of Joseph Carrabis from NextStage Evolution on engagement, attention, trust and influence. Joseph has recently offered to help me take my framework to the next level by working out some of the mathematics and validating/correcting some of my assumptions:

http://thefutureof.webanalyticsdemystified.com/2008/03/21/continuing-the-discussion-attention-engagement-authority-influence/

Anyway, good to see you're still engaged in the engagement debate. Like web analytics back in the early part of this decade, things are just now ramping up and it will be exciting to see where all of this goes!

All the best,

Eric T. Peterson
Web Analytics Demystified
http://www.webanalyticsdemystified.com

[Joseph Thornley]

Great post Katie. (It's good to see that you are being consistent in your papers and your book! ;-) )

These are good ideas and I hope that you'll bring them forward to the roundtable on social media measurement and metrics in Toronto this May.