Warren Buffett: 3 steps to pass the test

by Urs E. Gattiker on 2010/12/29 · 15 comments 8,382 views

in a analytics smarter & actionable KPIs,a analytics: rating and ranking - who's #1,b why benchmark analytics,d business failures

The results of a Financial Times/Harris poll resulted in a great headline: Europeans favour charity over tax for the super-rich. But can this attention-grabbing statement be trusted?

Another example that made me wonder was a Gallup poll containing at least one statement that confused me: You or another household member was the victim of computer/Internet crime, such as fraud or computer hacking, while using your home computer.

Image - picture - student getting annoyed with doing research - Download eBook - why I hate research methodologyWhether respondents inferred that getting spyware, viruses or other malware on their PCs means being hacked is unclear (read more at Stay safe and share the moment).

We focus on 3 steps to take before acting on research findings.

    1. Is the information in context?

To illustrate, the Financial Times/Harris poll suggests that the public in the UK, France, Italy, Spain and Germany believes that charity is “…the right way to go about the super-rich giving back.” However, this does not necessarily mean the public wants to see

    a) tax breaks for the super-rich, or
    b) abolition of the inheritance tax.

In Germany or France, the context might be a direct income tax rate of 60 percent or higher, as well as a hefty inheritance tax. Therefore, respondents in those areas may be feeling charitable and suggest that the super-rich should not be taxed higher than they already, and instead give more to charity, as reported by the pollster.

Tip 1: Try to understand the context in which data are collected (e.g., data taken from Fortune 500 companies or micro-firms?), and check whether the study’s funding source(s) and possible conflicts of interest are disclosed – (see FTC regulations), since that might help explain certain interpretations.

    2. Is your source trustworthy?

Image - tweet - @Albawaba - http://ow.ly/3tzk1 جولة في المدونات: شجرة كريسماس بـ 11 مليون دولار في الإمارات وخواطر كحولية #albawaba #JO #UAE #BlogsAlways remember that ignorance is NOT bliss. Another recent study stated,

    “… [we] analyzed more than a billion tweets generated by over 20 million Twitter users in 2010. This included bio, web site address and location information (both textual and lat / long) disclosed in their profile.”

Turns out, only English tweets were analyzed and which Twitter accounts were selected, and how, was not explained. Nor was I able to find information about the coding system used to categorize them.

Pollsters like Harris Interactive often point out procedural or findings weaknesses in the methodology section:

    “All sample surveys and polls, whether or not they use probability sampling, are subject to multiple sources of error which are most often not possible to quantify or estimate, including sampling error, coverage error, error associated with non-response, error associated with question wording and response options, and post-survey weighting and adjustments.”

Unfortunately, making attention-grabbing headlines and sweeping inferences based on such data is dangerous.

Tip 2: Check the methodology section on how data was collected (e.g., how were subjects selected) and what analysis was done (e.g., sentiment analysis is still looking for a way to obtain valid data).

Why not indulge for 2011 and enter your email in the field below, so you can get the next post about social media trends and metrics first. You’ll be glad you did!

    3. What do the results mean for you?

Image - Hervé Humler - Ritz-Carlton hotel group's COO - Give the customers want they want - 'What they’re saying, and we know it too, is a simple thing. Why do you see Coca-Cola advertise every day? Because if you don’t see Coca-Cola advertise every day, they’re going to use Pepsi. It’s the same thing: if I don’t have enough hotels in some markets, what they’re going to use is the competition. And after that, they won’t come back to me.' If we look at context and how data are collected, it may no longer be so obvious whether we can trust the findings. At the very least, we may question how applicable it really is for our needs.

For instance, US internet users receive more than 6,000 display ads in a quarter. The Gallup press release states,

    “Internet users send a mixed message about how much these ads are influencing them. On the one hand, more than 6 in 10 say they have noticed that some ads are targeted specifically to them based on websites they have previously visited. On the other, 9 in 10 say they pay little or no attention to online ads. These assessments vary little by age and income.”
    (See U.S. Internet Users Ready to Limit Online Tracking for Ads).

Since I cannot afford billboards or TV ads, will spending money on Google Ads make a difference to my bottom line?

More information about research and social media: Customer relationship management and fair disclosure

Tip 3: Completing a quick cost-benefit analysis while reading a study’s findings will help you decide whether choosing the strategy recommended by the report is within your budget.

    Bottom line

Image - tweet - @ComMetrics - #Facebook user #statistics @thomashutter: CH, DE, & AT BUT what does this mean for my biz? http://ad.vu/5bc8 via @mcschindler.Studies and surveys are done all the time, but it is up to you to ask the tough questions before you report the results or – heaven forbid – act on them.

If things do not make sense to you after carefully reading a report, do not trust it. And taking the word of a journalist without getting your hands on the original data is outright irresponsible, if not stupid.

Take a page from Warren Buffett’s book and use a short checklist to evaluate research:

More from Warren Buffett:

If you like this post, please share it with your friends. How about asking them to comment after they have read this post? I love to hear what people think!

Are you with me on these research methodology matters and trust issues, or have I lost my marbles? Will people continue to base important decisions on research reports that may not be worthy of their trust?

And since this is our last post before the new year, we would like to wish you and your family a safe and happy holiday!

Previous post:

Next post: