Benchmarking requires setting objectives and measuring performance to see if one has surpassed it.

Here we propose five best practice steps to help you develop useful Key Performance Indicators (KPIs) that will result in actionable metrics for your social media activities.

Not a subscriber yet? Want the scoop first? Just leave your email address in the field below.

1. Ensure management understands the difference between operating and financial metrics.

As Anthony (Tony) Tjan put it in one of his recent posts:

“The financial ‘numbers’ ultimately represent the scorecard we care about, but they do not help us understand how to score.“

It is critical to understand what the most important drivers or operating metrics are when it comes success or failure. Accordingly, what contribution for achieving the company’s operating metrics, such as higher customer retention, will come from social media efforts, must be addressed.

Tip 1: Define, write down and agree on the operating metrics you want to use and stick with them.

2. Know your customers and derive everything, including social media activities, from that knowledge.

Companies produce their corporate magazine or place brand advertising in a daily to reach and communicate with the targeted audience. Improving a magazine’s or corporate blog’s reading attractiveness works even better if it:

a) provides information that helps customers do their work smarter, and
b) empowers readers to get more out of services they purchase from the company.

Tip 2: Choose a social media tool that at least 20 percent of your current and potential clients already use themselves (e.g., would they read blog content delivered by email and/or RSS feed) AND provide content that is perceived as useful and adding value.

3. Establish how a particular social media channel can drive cost reductions and/or new product/service uptake.

Every office building has air conditioning and nobody asks for ROI on this. Plus, ROI (Return on Investment) or ROE (Return on Engagement) are the wrong measures for social media.

ROI is the outcome, but we need to understand how social media efforts contribute to operating metrics. Hence, we must establish how the input – social media or corporate blogging – drives the desirable output, such as lowering marketing costs or improving customer satisfaction.

Tip 3: Make sure your operators are clearly tied to your social media efforts.

4. Focus on the 3 to 5 KPIs that represent the most important drivers of value creation for your social media activities.

We suggest that a few clearly defined KPIs are more likely to assure people paying attention, without getting too distracted from the real job at hand.

Most important, KPIs must fit your company’s needs, like custom-made loafers or black heels. Otherwise, who would want to wear or make use of them?

Tip 4: Assure KPI buy-ins from your stakeholders, including your team, superior(s) and possibly key clients and/or investors.

5. Establish a baseline and timeline.

A baseline is a measurement used as a point of comparison. In turn, it can be used to asses progress against the budget or set of other objectives, including KPIs.

So it is advisable to first collect data whenever one begins social media monitoring to see where you stand.
Benchmarking against your own past performance is the only completely relevant measure. Therefore, improvement over the course of several months is what really matters.

For this reason, we developed the ComMetrics Footprint, ComMetrics Health Check, and ComMetrics Resonance, which are each made up of several measures. These allow tracking from the baseline onwards, as well as benchmarking against the ComMetrics Blog Impact index.

Tip 5: Establish the baseline numbers and compare such as Q1 2008 with Q1 2009 in order to improve.

For example, if you focus on backlinks, have they increased over the last quarter? And what about Quarter 3 2008 compared to Quarter 3 2009, what kind of improvements are seen there?

Bottom line
Surely nobody tries to quantify the ROI of air conditioning. Having it when outside temperatures reach 35 degrees Celsius helps office productivity, making it a mediating factor for, though NOT a cause of, ROI. Similar reasoning applies for social media measurement.

Above we outlined five tips that should help in developing KPIs that make sense to your company’s decision-makers and, most importantly, provide social media monitoring that helps your company improve.

More resources on KPI and benchmarking:

- Beth Kanter: Riffing on David Armano’s Listen, Learn, and Adapt: Need Your Organization’s Adaption Stories!
- Vijay Rayapati: Why social media ROI is the most over debated & misunderstood topic?
- Olivier Blanchard: Basics of social media ROI slide presentation.
- Mikey Ames The Hermit: Alumni Association Twitter Use – Baseline Comparisons.
- Beth Kanter: What’s Your Social Media Baseline?
- ComMetrics: Social media analytics and KPIs: From simple to refined.

Looking for telework as a techie? Join our virtual team NOW!

Now, it’s your turn.  What are some of your organization’s metrics, analytics and measurement stories for blogs?

Due to weakening financial markets and decreasing macroeconomic stability, the US has been ousted from the top slot in the World Economic Forum’s latest Global Competitiveness Report by Switzerland.

Effective benchmarking requires key performance indicators (KPIs) that help explain whether educational expenditures are a key driver or operating metric we should look at to help achieve higher rates of youth employment. We address this in more detail.

Not a subscriber yet? Want the scoop first? Just leave your email address in the field below.

Rankings – 4.11 Education expenditure: Public education expenditure as a percentage of GNI, adjusted savings.
Rankings – 5.01 Secondary enrollment: Gross secondary education enrollment rate.
Rankings – 5.02 Tertiary enrollment: Gross tertiary education enrollment rate.
Rankings – 5.03 Quality of the educational system: How well does the educational system in your country meet the needs of a competitive economy? (1 = not well at all; 7 = very well)
Rankings – 5.04 Quality of math and science education: How would you assess the quality of math and science education in your country’s schools? (1 = poor; 7 = excellent, among the best in the world)
Rankings – 5.06 Internet access in schools: How would you rate the level of access to the Internet in schools in your country? (1 = very limited; 7 = extensive)

Hard data: Are they accurate?
Switzerland is doing quite badly when it comes to primary education, as well as secondary education rates (e.g., Rankings 4.11). One explanation is that instead of using absolute numbers, the report uses the percentage of Gross National Income (GNI) spent on primary schooling. This could explain why countries that spend more on primary education as a percentage of GNI do better, like Lesotho or Namibia.

However, whether as a percentage of GNI or in absolute dollars per pupil, mere spending cannot assess the effectiveness of a school system. In fact, countries that spend less in absolute dollars or as part of GNI often do better than others by nevertheless achieving higher literacy rates.

Opinion data: Reliable and valid?
The opinion data is based on a survey of 13,000 executives in the ranked nations, but given Rankings 5.06, how much does an executive really know about the reality of a teacher trying to teach a classroom of fifth graders about math or finding information on the Internet using computers?

Put differently, what does access mean: PCs connected to the Internet for each classroom or access for each child to a computer with nothing more than an operating system and a word processor?

And if we trust that executives can provide an evaluation regarding math and science education, why don’t we ask them about reading and writing as well? Graduates who are unable to write a job application without typos or grammatical errors have a tough time getting a job, don’t they?

Secondary and tertiary education: What about youth employment?
During this financial crisis, education should actually help improve one’s skill-set, thereby making teenagers (15 – 19-year-olds) and young adults (20 – 24-year-olds) more employable.

Unfortunately, the youth unemployment rate in the US was 18.5 percent in July 2009. According to the government this is the highest July rate since recording began in 1948 (download pdf report and tables).

Canada’s employment figures for June 2009 show a loss of 33,000 jobs among those 15 – 24 years old, rocketing the unemployment rate among young people to 15.9 percent, the highest in 11 years (download pdf and report summary).

A recent report for Australia indicates a similar trend whereby the labor force underutilisation rate (LFUR = sum of unemployment and underemployment) for 15 – 24-year-olds rose from 24.2 percent in February 2009 to 26.2 percent in May 2009 (see chart at right, see five-page report).

For Switzerland, the number of unemployed youth (15 – 24-year-olds, including school leavers and students) increased from 4.6 percent in July to 5.3 percent in August 2009. Their numbers reached 29,298 which represents a three-quarter increase over August 2008 (download pdf and more details).

Bottom line
The above indicates that educational expenditures may not be comparable across countries and, as importantly, asking executives to judge an educational system may result in data that could hardly be called reliable and valid.

While Australia, Canada and the US rank highly for their participation rates in tertiary education, Switzerland gets marked down for having a strong dual system (meaning apprentices spend as little as 30 percent of their time in formal education, while the rest is spent acquiring practical skills in the workplace).

Besides, completing an apprenticeship leaves the door open for post-secondary education. Worse, not having enough qualified tradespeople on construction sites does not help improve the country’s competitiveness with regard to its infrastructure. Youth employment means all this human capital is wasted, which definitely does not improve a country’s competitiveness.

Surely, youth unemployment is an important KPI for benchmarking education policy, whereas expenditure levels tell us little about how effectively a country uses its newly trained human capital. The World Competitiveness Report 2009-2010 uses educational benchmarks whose usefulness are questionable, if not outright invalid.

More resources on KPI and benchmarking education:

- Gary Martin: A portrait of the youth labor market in 13 countries, 1980–2007 – Monthly Labor Review July 2009, pp. 3 – 21.
- The Bologna Club: What U.S. Higher Education Can Learn from a Decade of European Reconstruction, May 2008.
- Tuning educational structures in Europe.
- OECD data: Economic returns for an individual obtaining upper secondary or post-secondary non-tertiary education, obtaining tertiary education, part of initial education (2005 data Excel spreadsheet – September 8, 2009).
- more about the World Economic Forum Davos

Looking for telework as a techie? Join our virtual team NOW!

Now, it’s your turn.  What are some of your KPI and benchmarking stories regarding education and competitiveness in your neighbourhood?

Short link blogging coming your way this week – our top stories include Amazon’s Elastic Computer Cloud or EC2 vulnerability to hacking, 34 free social media tools, Microsoft and Bing’s imminent release of a new, joint feature and how the media shift shows why BBC, AOL and the Guardian are becoming increasingly dependent on social networks like Facebook, Xing and Bebo to get traffic (view chart) and how Twitter may be named in a lawsuit after a driver’s tweeting caused freeway carnage.

In case you missed the last six weeks’ best links:

Very useful links: White House to Adobe for week 30.
Very useful links: Nikon to Wii via Yahoo! for week 31.
Very useful links: Friendfeed to Berlusconi for week 32.
Very useful links: Trabant to Obama via Copenhagen for week 33.
Very useful links: UBS to Levi’s via CBS for week 34.
Very useful links: Microsoft vs. Facebook vs. SAP for week 35.

Not a subscriber yet? Want the scoop first? Just leave your email address in the field below.

Social Media TOOLS and GUIDES that make a DIFFERENCE
VERY USEFUL #tools2watch – Social media monitoring wiki – lists 34 free and 60 commercial tools for social media analytics – great help for monitoring your brand.

VERY USEFUL Look who follows NoFollow links - test finds that both Yahoo & Google follow the NoFollow links #metrics2watch

#tools2watch – Bing will soon release the new feature “Bing and Ping”, a means to share your search results with friends – you can preview by becoming a fan of Bing on Facebook – that nets you an invite to the pre-party.

Sentiment analysis for online content: Honest? What the benchmarks show: failure. #trends2watch

Sentiment analysis – human scaling #tools2watch

Economic CRISIS AND POLICY Research

Best quality tea prices rose by 36 percent since January, almost twice the 2005 price (see graphic at left) #trends2watch

Discussing Doha – Europe provides $2 a day per cow in farming subsidies – in India 70 percent of population live on less than $2 a day #bettergovernance

VERY USEFUL #things2read OECD’s new report on the well-being of children worldwide

Social Media BEST PRACTICE
Best business model: Free to use, pay to play – when will venture capitalists pull the plug on Twitter? #bettergovernance #trends2watch

VERY USEFUL – Why following the product development diagram could result in failure for your start-up #things2read

Study raises DOUBTS: opt-in Internet surveys’ validity may not hold up against paper-and-pencil and telephone surveys – find the summary and the full paper here.

Social Media MARKETING BUZZ
VERY USEFUL => Popular on commetrics.com via Twitter => re-tweets happen more often, nice #metrics2watch #trends2watch

Social Media FIRST to Move – TRENDWATCH
#first2move – North Carolina Department of Transportation providing traffic reports via Twitter: what about risk management - micro-blogging causing accidents by drivers reading tweets while driving?. Traffic reports for motorists and ferry passengers by region or interstate highway. #trends2watch

InfoSec
#threats2watch  RISK MGMT: Stealing data from Amazon’s Elastic Computer Cloud (EC2). Research reports way to attack the cloud using side-channel. (300.5 KiB, pdf)
Amazon not ready to talk about this threat this week, but this is a systemic risk that cannot be ignored by Google, Microsoft or Amazon, since all three offer these cloud services, nor by clients depending on these providers – what if your application is hacked this way? #trends2watch

Actionable Metrics: Standards, Testing and Trends
Best business model: Free to use, pay to play – when will venture capitalists pull the plug on Twitter?

VERY USEFUL social media ROI 67 slides #things2read http://bit.ly/8VDWI BUT operating metrics are more important than ROI….

#trends2watch social network sites bringing much  traffic to media sites such as BBC, the Guardian, AOL News (see graphic below).

2009 Social Media Calendar – Upcoming Conferences of Interest

• September 14 – 17 ==> Chicago, USA - Social Media For Government: How to engage your employees and citizens by using the latest Web 2.0 technologies to drive communication results.

• September 29 – October 2 ==> Ottawa, CANADA - Social Media For Government: How to engage your employees and citizens by using the latest Web 2.0 technologies to drive communication results.
• September 29 – October 2 ==> Washington, USA – Social Media for Healthcare: How to use the latest web 2.0 technologies to engage your community, improve quality of service and build your reputation.
• September 30 – October 1 ==> Washington, USA - Rethinking Corporate Communications 2009 Conference.
• October 28 – 29 ==> Atlanta, USA – The eMarketing Association (eMA) – eM9-ATLANTA eMarketing Conference: Forum for the very latest trends, best practices, processes and ideas that are transforming the future of marketing.
• November 7 – 10 ==> San Diego, USA - Public Relations Society of America (PRSA) 2009 International Conference: Delivering Value.

What is happening in your neck of the woods? Please share your conference dates by leaving a comment below or sending us an email.

That’s the way we saw things this week; now it’s your turn. Did we miss anything important? Please add it below as a comment, along with a link. Looking forward to hearing from you!

2010-06-01 – Update – new tools – heated discussion – see bottom of this post.

Recently a customer asked why we have not yet included sentiment analysis in My.ComMetrics.com. Yes, the customer was fully aware that sentiment analysis (also called opinion mining and sometimes semantic analysis) involves classifying text using natural language processing, computational linguistics and text analysis to reveal the sentiment (e.g., positive, neutral or negative) of a particular text.

Recently I came across this statement by Nathan Gilliatt:

“This NY Times article doesn’t include “brand monitoring,” “listening,” or “social media analysis” as it focuses on “sentiment analysis,” but they’re all more or less the same thing.“

Gilliatt puts the finger on a sore issue that indicates how critical it is that we DEFINE sentiment analysis (see above) and then test not only the reliability, but the validity of the results.

Is it reliable?
Reliability means getting the same results for the same text analysis regardless of how often the test is run. According to this criteria, the Financial Times’ Newssift service is strong. It lists the two articles in the Financial Times mentioning ComMetrics (see image below). Repeating the string of keywords with other terms such as social media measurement or social media tracking resulted in the same articles appearing on a larger list.

Sentiment analysis is it reliable – probably is … but being reliable while measuring the wrong stuff cannot be the answer as we explain below.

Is it valid?
That all depends how you define the term validity and which sentiment analysis software one uses.

Using the keyword ‘ComMetrics’, the data served by the Newssift tool revealed:

- one neutral article
- one positive article, and
- one article that has nothing to do with ComMetrics, but is still positive.

But can these findings be interpreted as valid? The article classified as neutral raised some important questions and explained how certain blogging challenges could be successfully resolved. How this sentiment would suggest neutrality while brand monitoring is a bit difficult to understand. Further, the last article listed in the image above has absolutely no relationship with ComMetrics. Accordingly, its appearance in the user’s results list is an error.

In conclusion, 33 percent of search results were correctly identified and classified, regardless of what keywords were used.

What about Twitter?
Various sentiment analysis software is available for Twitter. For instance, the image to the right is from Twendz, which provides new visitors with a hyped-up description of itself. Unfortunately, its analysis is neither well-described nor are the findings coherent.

A similar software called Tweetfeel fails to work for small brands and does not even bring up all tweets for the biggest brands out there. Hence, we could not really test it.

Twitrratr is another one of these tools, but it stopped tracking some of our brands in late 2008. Maybe the system was put into hibernation?

Worse, it is nearly impossible to figure out what criteria are used by any of the applications to decide whether a tweet is positive, neutral or negative.

How such tools and their findings should “…take the pulse of Twitter users about particular topics…”, as the NY Times article claims, suggests that the journalists did not take the time to take these tools for a test-drive. This kind of ’investigative’ reporting will not help reestablish the value of content.

In short, the sentiment analysis tools we were able to quickly test and review for Twitter seem reliable, but they do not pass the validity hurdle (i.e. they do not measure what they are supposed to measure).

More posts about Twitter from this blog

Bottom line
Sentiment analysis is an important, but very hard to master, science and it is still in its infancy. While it can be quite accurate (reliability => maybe high => 80 percent or higher), it does not necessarily make the data valid or useful for making strategic decisions grounded in effective brand monitoring.

Finally, when it comes to languages, things get way more complex than simple tweet or text analysis, making success an ever more elusive concept for sentiment analysis, as illustrated below:

- cultural differences:
When Americans say ‘quite’, they mean ‘very’, and ‘interesting’ implies they want to hear more. In England, ‘quite’ means ‘not at all’ ‘interesting’ equals ‘I can’t think of anything nice to say’, and calling something ‘quite interesting’ indicates the speaker is desperately searching for the nearest exit.
-  positive versus negative meaning: sinful isn’t always sinfully good chocolate

The above is nicely illustrated by a comment I recently emailed regarding a story about buskers working in the London Underground.

The ’busker’ emailed back saying something like: “… you make some interesting and very true observations…” Depending on where the busker makes her home (e.g., United States versus Great Britain), this sentiment can be interpreted in many different ways.

Takeaway: All of the sentiment analysis programs I have tested – not all were mentioned in this blog post – are seriously challenged when it comes to coping with the intricacies of language. Reliability is an important factor but if a tool does not meet validity requirements, I don’t want to use it for brand monitoring or social media tracking.

Your turn: Did we miss something? If you know of any tool that works better magic than those we looked at, please give us a shout. What is your experience with, and opinion of, sentiment analysis?

More resources about social media marketing and the c-suite:

2010-06-01 – Update – discussions about new tools and validity concerns on Xing SM Monitoring group

Usage varies enormously across Europe.

- bloggers may want to boycott Alexa.com rankings, and

- Silicon Valley may not always get it right.

Things around the world are getting more complicated for sure. Hence, firms that focus on understanding and respecting differences across cultures and countries will do better.

In that context, we came across a great map showing social network penetration across the world. Information from the Alexa toolbar was used to draw this map.

Get the original map including more explanations here:

Valleywag.com – data junkie the world map of social networks

Where do these data come from?

Here is how Alexa.com explains how it collects data visualized above:

Alexa could not exist without the participation of the Alexa Toolbar community. Each member of the community, in addition to getting a useful tool, is giving back. Simply by using the Firefox and IE toolbars each member contributes valuable information about the web, how it is used, what is important and what is not. This information is returned to the community as Related Links, Traffic Rankings and more.

The above makes it obvious, install toolbar and Alexa keeps track about which web sites you visit, time spent by leaving your browser tab bar open with that URL and so on. All needed to generate usage statistics a la Alexa.

Why are some important networks not included in the above map?

Therefore, why is such a site as Xing not being included in the above statistics?

It is surely larger than StudiVZ (Studiverzeichnis), at least in Germany.

ANSWER: Xing is a social network site for business folks. Moreover, as we all know, most corporate users do not have admin rights required to be able to install the Alexa toolbar on their PC at the office. The toolbar is, however, the way Alexa collects user data for its statistics. Hence, Alexa’s usage statistics exclude any corporate visitors without an Alexa toolbar checking their Xing account daily or more often.

Same goes for B Kontakte, a Russian network site. It is the forth largest site in the country, according to traffic data and has just about 700′000 registered users.

B Kontakte Russia – network

As well, people who do not speak English will not have the Alexa toolbar installed. I just wonder, how many users you think will have an Alexa toolbar installed in Mongolia. Here hi5 is the dominant social network (see map above) but how many users provided these statistics (= sample size)?

Do your customers and blog readers participate in these social networks?

Social networks serve different clientele with different needs. Xing.com, LinkedIn.com or Plaxo.com serve business people. Each has a slightly different focus, of course. In contrast, Facebook or StudiVZ (Studiverzeichnis) serve students. However, students are more likely to have the Alexa toolbar installed on their personal notebook than the corporate types on their employer’s machine.

It could be that your current and potential clients may participate in Xing.com in the hopes that it will help their work or just help them waste much time – see here for an analysis:

- 5 Netzwerkmarketing fuer KMUs – Soziale Netzwerke und Effektivitaet

Second, what are the chances that they use the Alexa bar? Third, how likely will they spend much time in those social networks?

CyTRAP Labs’ take on this issue

If you do not sell to retail customers and, most importantly, your product is not a branded household name (e.g., Mr Proper or Kit Kat chocolate, Dell computers) the question is – are social networks useful to you?

Do they help reach clients or at least potential customers if you are a chartered accountant or CPA?

Alexa data are not representative of the usage of social networks.

So do you have the Alexa browser plug-in? No? Then you viewing this page will have absolutely no impact on the Alexa ranking for this site. You simply do not count.

- Bloggers Boycotting Alexa.com Rankings

Besides this limit, however, the map shows different folks around the world use different social networks. That in itself is important information. In addition, do not always head for Facebook and MySpace to tap the social networking market. Things are different in Europe, Asia or Africa as the map illustrates.

Tidbit – yes Virginia, you can plan but things may work out very differently

Google set up Orkut.com to target the U.S. market but, interestingly, 55% of its audience is now Brazilian and probably another 10% comes from India.

Tip

Whenever you are shown usage statistics from the internet, ask yourself:

How were these data collected and, in turn, how were they used to arrive at the conclusions presented to me?

Alexa is a perfect case in point on how data may not represent the true situation for you. If you have no Alexa toolbar installed, you are not part of their usage numbers. Having no right to install programs on their corporate PC, most business people are not likely to be part of their sample… However, it may be that your site is targeting those users in particular. If so, Alexa numbers tell you little if anything about usage of the web by corporate types.

Bottom line

Whenever you come across some data and/or statistics just ask:

Can they be meaningful to my business?

============>

If this post was helpful to you, please consider stumbling it or subscribing to feeds from CyTRAP Labs. Cheers.

==========>

Also of interest:

- Why women buy black high heels – the girl’s guide

- the mission of ComMetrics

- 1 sensible metrics – how to measure success of a blog – the basics

==========>

We will bring you another interesting tidbit about browser usage across Europe and what it means soon, so stay tuned.