Last Wednesday I was a part of the annual Conclave, at which a score of very smart social media measurement experts deliberated, debated, and decided on definitions of some of the fundamental terms of social media measurement. It was measurement geek heaven.
Mentions. Reach. Engagement. Influence. You might think that you know what these terms mean. And certainly the social media experts must know, right? Wrong.
The problem facing the measurement industry is that a group of entirely competent social media measurement experts can start with the same raw data, yet their research may reach very different results, depending on who they are doing the research for, the type of industry they are studing, and how they define the fundamental terms involved. As a result the research can be flawed, and it probably can’t be compared to other research.
Without consistent and replicable research, the credibility of the whole industry suffers. And if everyone in the marketplace doesn’t agree on just what they are buying and selling, then commerce is slow and difficult. We often take for granted the many standards that allow us to engage in quick and convenient communication and commerce. Everything from the time of day and monetary units, to USB and HTML. It's time social media measurement had some standards of its own.
So, really, how exciting can it be to watch a room full of very smart people wrangle and wrestle over the meaning of research terms? Pretty darn exciting, as it turns out. Ideas and concepts bounce back and forth. Terms are introduced and the noise level increases as possible definitions, nuances, objections, and qualifications are debated. Eventually some conclusions are proposed and things calm down while everyone sits and ponders. Then someone says, "But what about ...?" New points and objections come up, the conclusions get thrown out, and the messy-yet-brilliant process starts again.
And when you consider that these definitions are the groundwork for progress in the field over the next few years, this undertaking is, well, kind of exhilarating.
Here's an example. "Influence" is one of the tricky terms The Conclave dealt with. It has already been defined in a useful way (thanks to Philip Sheldrake), and a not so useful way (thanks to Klout). Our discussion of it led to one of the most interesting moments of the day. Katie Paine, who was leading the session, said, “OK, suppose you had all the money and resources in the world and you wanted to design an experiment to measure influence, how would you do it?” And the room went absolutely quiet. Twenty very intelligent people were rendered speechless as they puzzled over this difficult problem. Now that’s some measurement geek excitement right there.
Here is how you can get involved. The results of The Conclave will be posted at #smmstandards.org, and your comments are important. The standards-setting process is that the first pass work is posted there and circulated for comment, and then the revised progress is published as an interim standard. What you can do right now is to go over to smmeasure.org and take a look at the proposed Transparency Table. It is on page 16 of the March to Standards report. Post your comments. ###
“Do not believe in anything simply because you have heard it. Do not believe in anything simply because it is spoken and rumored by many… Do not believe in anything merely on the authority of your teachers and elders... But after observation and analysis, when you find that anything agrees with reason and is conducive to the good and benefit of one and all, then accept it and live up to it.” 
Too bad all you are talking about is social media. The problem is much more basic and extends to every form of research for communication.
Posted by: Joe Smith | October 08, 2012 at 08:32 AM
Good point, Joe. Communication research is slow to progress because researchers define terms in many different ways.
Posted by: Bill Paarlberg, editor | October 09, 2012 at 06:23 AM