Katie Paine's Seven Basic Steps of Measurement, Remixed for Nonprofit Social Media

Here are Katie Paine's famous Seven Steps of Measurement, in the form of an excerpt from Chapter 4 of Measuring the Networked Nonprofit, the new book by Katie and Beth Kanter. (Order the book here.) The cartoon, also from the book, is the work of Rob Cottingham. Click on it to see it larger.

While this excerpt discusses the seven steps in the context of nonprofit social media, the logic behind them applies to any measurement program undertaken by any organization. The steps are the same, and are carried out in the same order, you just have to adapt them to your own program and goals.

Katie Paine’s Seven Basic Measurement Steps, Remixed for Networked Nonprofits

Measurement is a formal discipline, governed by rules and processes established by academics and researchers. Certainly you do not need a Ph.D. from MIT to measure your networked nonprofit accurately. Far from it. But fuzzy, unproven, or sloppy approaches to the formal discipline of measurement are a waste of time. Moreover, they are not necessary. To do proper measurement, you just have to follow the basic steps.

No matter what your program or campaign—be it an event, a Facebook page, a messaging campaign, or a donor outreach program—there are seven basic steps for doing good measurement and getting valid and actionable results: 

Step 1: Define your goals. What do you want to change? What outcomes is this strategy or tactic going to achieve? What are your measurable objectives?

Step 2: Define your audiences. Who are you trying to reach? How do your efforts connect with those audiences to achieve your goals?

Step 3: Define your benchmarks. Who or what are you going to compare your results to?

Step 4: Define your metrics. What are the key performance indicators that you will use to judge your progress toward your goals?

Step 5: Define your time and costs. What is your investment?

Step 6: Select your data collection tools. What applications or systems are you going to use to collect your data?

Step 7: Collect and analyze your data, turn what you have learned into action, and start to measure again.

Let’s discuss these steps one at a time.

Step 1: Define Your Goals

The biggest mistake that many nonprofits make is taking a “fire, ready, aim” approach to social media. In fact, the first thing you need to do is ask, “To what end?” What is the desired return? What’s the goal? What problem will this program solve?

Without clearly defined outcomes, you won’t know what success looks like and whether social media are bringing you value. Defining goals is not necessarily an easy task. Measurement requires measurable goals, and measurable goals require clearly defined time frames, audiences, and outcomes. It may take a while to achieve consensus on these objectives in your organization. Later in this chapter, you will learn about this process for a social media pilot, and Chapter Five is devoted to a discussion of measures of success.

While your specific results will vary depending on your organization’s mission, programs, intent for social media, and maturity of practice, the outcomes that you will be measuring will probably include one or more of the following:

Brand or mission:

• Changes in perceptions or awareness of your brand or mission
• Changes in positioning of your brand or mission

Relationships:

• Improvement in or establishment of relationships
• Changes in the health and strength of your relationships with stakeholders

Behaviors:

• Increases in specific actions, such as attendance, donations, volunteerism, or memberships

Once you’ve identified your outcomes or intent, the next step is to translate these into SMART objectives. A SMART objective is Specific, Measurable, Attainable, Realistic, and Timely. SMART objectives include the answers to the questions: By when? and How many? For instance, a good SMART objective would be: “Increase by 10 percent the number of volunteers we convert into donors by the end of our fiscal year.”

Many times organizations have difficulty making the link from results and outcomes to social media use. This is where using a theory of change methodology might come in handy. With such a framework you define your organization’s progress as a series of step-by-step efforts and results. (This technique is discussed in depth in Chapter Five.)

Step 2: Define Your Audiences

Now it is time to understand the audiences you are trying to reach and how your efforts connect with those audiences to achieve your goals. There’s never been a nonprofit with enough resources to measure everything it wants to measure. This is why this second step involves setting priorities about what you want to measure. You do this by listing all the stakeholders that influence the success or failure of your mission or your organization. There are probably at least a dozen. Then write down how having a good relationship with each of those groups contributes to the success of the organization. For example, a good relationship with local elected officials helps get bills that affect the policies you want to change passed.

Here’s a fun and easy way to prioritize those stakeholder groups. First, raid your old board games for a pile of play money, or you can use colored dots. Then get all your decision makers in a room and give them money or dots representing $1 million to spend on communicating however they like on your stakeholders. They can spend it all on one group, evenly divide it, or allocate it in some other way, but they have to spend it all. Then add up the totals. Whichever stakeholder group ends up with the biggest budget is the first one you should measure. The second biggest budget is the second group to measure, and so on.

Step 3: Define Your Benchmarks

One of the most common questions that nonprofits ask when crafting SMART objectives is, “How do we know that we’ve identified the right number?” If you determine that 20 percent of the social media conversations contain your key messages for an advocacy campaign, you have no way of knowing if that 20 percent is a good number or a bad one. If the opposition is at 80 percent, then it’s bad. If the opposition is at 5 percent, then it’s great.

This is why measurement is a comparative tool; you always want to know not just the number for the month or the quarter, but whether that number is bigger or smaller than something else. Examples are the number of likes compared to last month or last quarter or the number of comments on your blog compared to the number on a peer’s blog.

So the next step in measurement is to decide who or what you are going to compare yourself to. The most effective comparisons are to peer organizations or to your organization’s past performance over time. Sometimes the latter is difficult because social media is relatively new for many nonprofits. Most organizations begin by measuring over time, and then add in peer institutions or organizations with which they might compete for share of volunteer hours or share of wallet. Again, what is important is to benchmark against what matters to your organization, and often that is whatever keeps your board and the executive director up at night.

Doing a benchmark study of similar organizations doesn’t have to be an elaborate or time-consuming burden. Pang Houa Moua is the first communications director hired for SEARAC, a national organization that advances the interests of Cambodian, Laotian, and Vietnamese Americans by empowering communities to create a just and equitable society. Their communications strategy included working with twenty organizational partners, and they wanted to improve their Twitter results. Says Houa Moua, “I didn’t want to pick a number of out of the air: one that was too high and created unrealistic expectations, or one that was so low that we easily made it.” So to help plan strategy, staff did a quick benchmark study by interviewing their partners to learn the size of the audiences and what Twitter techniques were working for them.

Sometimes, for whatever reason, your organization may not have history to identify the right number. Sometimes you need to acknowledge this, take an educated guess as your baseline, and dive in. By measuring and learning from your first phase or first effort, you will be able to set a more realistic number for the next phase.

The Silicon Valley Arts Council did just that in its first study of the effectiveness of social media as part of the outreach for its Artist Laureate Awards. Says Anna Weldon, director of communications, “We were tracking downloads of the application, as well as applications received. Because it was a new program, we didn’t have a benchmark. We looked at other grant programs and established our numbers based on an educated guess. Now, having measured the results, we have established a baseline moving forward.”

Step 4: Define Your Metrics

The fourth step in the process is to define and get broad agreement on what metrics you will use. Key performance indicators (KPIs) are the metrics that are most important for charting progress toward your SMART objectives. There are hundreds, if not thousands, of metrics you could collect, so you have to decide on a small handful that will be most informative.

Although the formal term is Key Performance Indicator, a more colorfully accurate term is Kick Butt Index. In other words, if the boss comes in and says, “Damn it, we’re getting our butt kicked out there!” what does that mean? And if he or she says, “Congratulations, you’re really kicking butt!” what does that mean?

Vickie McMurchie, the community manager for Surfrider Foundation, an environmental advocacy organization for oceans and beaches, is responsible for social media at her organization and has seen the value in connecting goals to KPIs. According to McMurchie, “We devoted a staff meeting, which included senior staff and department heads, to discussing and identifying our key performance indicators. Getting everyone on the same page for the goals helped us avoid collecting useless data.”

The important thing to remember about KPIs is that you become what you measure, so they have to be meaningful, actionable, and relevant. If you’ve completed steps 1 through 3 already, your KPIs should fall naturally out of your earlier conversations about goals and stakeholder groups. It is just a matter of translating your priorities and goals into a number you can calculate—for instance:

• Percentage increase in donations
• Percentage increase in new donors or members
• Percentage reduction in cost per member acquired

For social media, typical KPIs are:

• Percentage increase in share of desirable coverage (“desirable” can mean a positive tone or favorable positioning)
• Percentage increase in share of conversation
• Percentage increase in number of conversations expressing support for the cause
• Percentage increase in conversations that contain your key messages

Step 5: Define Your Time and Costs

Nothing is free—in life or in social media—so it is important to identify the true costs of your social media programs. Since most of the cost is going to be in staff time, you’ll need to find out how much time social media require and determine how much time you’re going to invest.

Most important, given the amount of time you plan to invest, are your expected results reasonable? You may need to revise your time investment upward or reduce your expectations. Think about the opportunity cost of not investing time and whether you’re shifting resources.

Also, you will want to be able to compare the costs of alternative ways of achieving your goals. For instance, if your plan is to use social media to get your key messages out, and possible alternative media are billboards and print ads, then you will need to find the cost per message communicated for all three media so you can compare them. If the goal is raising dollars, you may need to be able to compare social media costs to the costs of a direct mail campaign. Be honest and transparent about what your real costs are.

Step 6: Select Your Data Collection Tools

Your measurement tools are the techniques you use to collect data or KPIs. These include Google or other Web analytics, surveys, and content analysis. (Measurement tools are covered in more depth in Chapter Eight.) For now, remember that any tool is useless unless it measures what you have defined as a goal. The tools you use have to be able to connect your activities, their impact on the audiences, and your goals.

For example, if your nonprofit’s goal is to improve children’s health, then let’s say you decide that lawmakers and policy advocates are your most important stakeholders, because a good relationship with them will mean that better child health policies are adopted. Your activity would be deploying a Twitter strategy to reach out directly to lawmakers and advocacy groups. You would measure the engagement in your Twitter stream via links and retweets, and then you would tie that activity to the impact of your Twitter strategy as measured by returned phone calls and meetings scheduled. If you have enough time, you’d do a preprogram relationship study before you started the Twitter campaign to see how those lawmakers and advocacy groups perceived your organization. And then you’d follow it up with a postprogram relationship survey six months or a year later. If you don’t have that much time, you might want to track the retweets, responses, use of a hashtag, and message contents of tweets around the campaign.

The tools you choose need to deliver the sort of data that will help you evaluate progress toward your objectives. Don’t allow yourself to get sidetracked. It is very easy to become seduced by the latest social media analytics tool or suffer from what blogger Alexandra Samuel calls “analytophilia”—obsessing over raw data without a clear idea of what you’re looking for.
There are three general types of measurement tools:

• Content analysis of social or traditional media
• Primary audience surveys via online, mail, or phone
• Web and social media analytics

Which one you use depends on the goals you established in step 1 and the KPIs you’ve established in step 4. You start with the object, then the KPI, then figure out what tool you need, as we’ve outlined in Table 4.1.

Step 7: Collect and analyze your data

For any measurement to work, you need to assess results, make changes, see if those changes had impact, make more changes, and so on. That’s how you continuously improve your program.
Do a postmortem and ask what went right and what went wrong. Remember that there are no failures, just unrealized opportunities. And if you don’t use your results to learn, you’ll never be able to take advantage of those opportunities.

Focusing on only the most exciting results is tempting, and typical, but it gets you nowhere. If you are going to improve, you need to stop doing what doesn’t work, so look for the weakest links. Move resources from what’s not working to what is working.

Report regularly. Your data should be used for resource planning, so about a month before budgets are due, make sure you have all the data in hand. Then establish a regular reporting schedule so you don’t just do a data dump at the end of a program.

###

Learn more about Measuring the Networked Nonprofit by Beth Kanter and Katie Paine, and order the book.