Sentiment analysis for online content: Honest?

by Urs E. Gattiker on 2009/09/03 · 56 comments 23,300 views

in a analytics taking action,b why benchmark analytics,c blogging - case studies,c micro-blogging Twitter

2010-06-01 – Updatenew tools – heated discussion – see bottom of this post.

Recently a customer asked why we have not yet included sentiment analysis in My.ComMetrics.com. Yes, the customer was fully aware that sentiment analysis (also called opinion mining and sometimes semantic analysis) involves classifying text using natural language processing, computational linguistics and text analysis to reveal the sentiment (e.g., positive, neutral or negative) of a particular text.

Recently I came across this statement by Nathan Gilliatt:

    This NY Times article doesn’t include “brand monitoring,” “listening,” or “social media analysis” as it focuses on “sentiment analysis,” but they’re all more or less the same thing.

Gilliatt puts the finger on a sore issue that indicates how critical it is that we DEFINE sentiment analysis (see above) and then test not only the reliability, but the validity of the results.

Is it reliable?
Reliability means getting the same results for the same text analysis regardless of how often the test is run. According to this criteria, the Financial Times’ Newssift service is strong. It lists the two articles in the Financial Times mentioning ComMetrics (see image below). Repeating the string of keywords with other terms such as social media measurement or social media tracking resulted in the same articles appearing on a larger list.

Sentiment analysis is it reliable – probably is … but being reliable while measuring the wrong stuff cannot be the answer as we explain below.

Is it valid?
That all depends how you define the term validity and which sentiment analysis software one uses.

Using the keyword ‘ComMetrics’, the data served by the Newssift tool revealed:

    - one neutral article
    – one positive article, and
    – one article that has nothing to do with ComMetrics, but is still positive.

But can these findings be interpreted as valid? The article classified as neutral raised some important questions and explained how certain blogging challenges could be successfully resolved. How this sentiment would suggest neutrality while brand monitoring is a bit difficult to understand. Further, the last article listed in the image above has absolutely no relationship with ComMetrics. Accordingly, its appearance in the user’s results list is an error.

In conclusion, 33 percent of search results were correctly identified and classified, regardless of what keywords were used.

What about Twitter?
Various sentiment analysis software is available for Twitter. For instance, the image to the right is from Twendz, which provides new visitors with a hyped-up description of itself. Unfortunately, its analysis is neither well-described nor are the findings coherent.

A similar software called Tweetfeel fails to work for small brands and does not even bring up all tweets for the biggest brands out there. Hence, we could not really test it.

Twitrratr is another one of these tools, but it stopped tracking some of our brands in late 2008. Maybe the system was put into hibernation?

Worse, it is nearly impossible to figure out what criteria are used by any of the applications to decide whether a tweet is positive, neutral or negative.

How such tools and their findings should “…take the pulse of Twitter users about particular topics…”, as the NY Times article claims, suggests that the journalists did not take the time to take these tools for a test-drive. This kind of ‘investigative’ reporting will not help reestablish the value of content.

In short, the sentiment analysis tools we were able to quickly test and review for Twitter seem reliable, but they do not pass the validity hurdle (i.e. they do not measure what they are supposed to measure).

More posts about Twitter from this blog

Bottom line
Sentiment analysis is an important, but very hard to master, science and it is still in its infancy. While it can be quite accurate (reliability => maybe high => 80 percent or higher), it does not necessarily make the data valid or useful for making strategic decisions grounded in effective brand monitoring.

Finally, when it comes to languages, things get way more complex than simple tweet or text analysis, making success an ever more elusive concept for sentiment analysis, as illustrated below:

The above is nicely illustrated by a comment I recently emailed regarding a story about buskers working in the London Underground.

The ‘busker’ emailed back saying something like: “… you make some interesting and very true observations…” Depending on where the busker makes her home (e.g., United States versus Great Britain), this sentiment can be interpreted in many different ways.

Takeaway: All of the sentiment analysis programs I have tested – not all were mentioned in this blog post – are seriously challenged when it comes to coping with the intricacies of language. Reliability is an important factor but if a tool does not meet validity requirements, I don’t want to use it for brand monitoring or social media tracking.

Your turn: Did we miss something? If you know of any tool that works better magic than those we looked at, please give us a shout. What is your experience with, and opinion of, sentiment analysis?

More resources about social media marketing and the c-suite:

2010-06-01 – Updatediscussions about new tools and validity concerns on Xing SM Monitoring group

  • Pingback: web 2.0 marketing

  • Pingback: Anthony Hamelle

  • Pingback: Anthony Hamelle

  • Pingback: david barrowcliff

  • Pingback: World Economic Forum

  • Pingback: Sergio Bogazzi

  • Pingback: Sergio Bogazzi

  • Pingback: Palpitt

  • Pingback: MyComMetrics

  • http://blog.vovici.com Jeffrey Henning

    Thank you so much for doing this research. For a while now I have felt like “sentiment analysis” technology was the Emperor’s New Clothes. Glad to know that I’m not the only one.

  • Pingback: Urs E. Gattiker

  • Pingback: World Economic Forum

  • Pingback: Tweets that mention Sentiment analysis for online content: Honest? -- Topsy.com

  • Pingback: Christophe ASSELIN

  • Pingback: Henri-Paul Roy

  • Pingback: Urs E. Gattiker

  • Pingback: Frédéric Laurent

  • Pingback: ImpactWatch

  • http://blog.ecairn.com dominic

    Thanks,

    This is just pure marketing and a good story for newspapers (i.e it help selling newspaper).
    The list of issues for being anywhere near a generic automatic sentiment algorithm is huge and growing.

    What makes it worse is that the web is becoming more and more social i.e organized in communities that all develop their own context, code and make it very difficult to analyze.

    Let’s say I tell you:

    “This new- technology is as great as automatic sentiment”:
    – in marketing communities, it’s probably positive
    – in the scientific community, it’s probably negative

    :-)

    Best
    .-= ´s last blog ..Cats, Dogs and Social Media =-.

  • Pingback: Sélection de la semaine (weekly) | Demain la veille

  • Pingback: Very useful links: Data-theft to tweet-deaths via free tools

  • Pingback: Coding the Sentiment of Web 2.0 « bill | petti

  • Pingback: Cake-Roll.com » Blog Archive » Market Research 3.0 Is Here: Attitudes Meet Algorithms in Sentiment Analysis

  • Pingback: Rachel Prosser

  • Pingback: Synthesio

  • computerkeepsfreezing

    je vous remercie pour ce poste très intéressant

  • Pingback: CyTRAP

  • Pingback: Urs E. Gattiker

  • keziban coşkun

    you should have some researches about Microsoft Fast

    • http://My.ComMetrics.com Urs E. Gattiker

      Dear Keziban

      Thanks for the tip, will do next time… but at the time we did this Microsoft did not have a comparable offering.

      Look forward to your next comment. Thanks for sharing.

  • http://www.technical-analysis-course.net technical analysis course

    This is really interesting. Sentiment analysis is crossing over to many areas including the stock market.

  • http://My.ComMetrics.com Urs E. Gattiker

    Hi Tech Admin: nnYes, of course, sentiment analysis could be used for the stock market as well. nnIncidentally, do you know of any great examples regarding the stock market and how sentiment analysis was used successfully? Appreciate a comment with some links of you have any to share.nnThanks for sharing.nnLooking forward to your next comment on this blog.

  • http://My.ComMetrics.com Urs E. Gattiker

    Hi Tech Admin: nnYes, of course, sentiment analysis could be used for the stock market as well. nnIncidentally, do you know of any great examples regarding the stock market and how sentiment analysis was used successfully? Appreciate a comment with some links of you have any to share.nnThanks for sharing.nnLooking forward to your next comment on this blog.

  • http://My.ComMetrics.com Urs E. Gattiker

    Hi Tech Admin: nnYes, of course, sentiment analysis could be used for the stock market as well. nnIncidentally, do you know of any great examples regarding the stock market and how sentiment analysis was used successfully? Appreciate a comment with some links of you have any to share.nnThanks for sharing.nnLooking forward to your next comment on this blog.

  • http://My.ComMetrics.com Urs E. Gattiker

    Hi Tech Admin: nnYes, of course, sentiment analysis could be used for the stock market as well. nnIncidentally, do you know of any great examples regarding the stock market and how sentiment analysis was used successfully? Appreciate a comment with some links of you have any to share.nnThanks for sharing.nnLooking forward to your next comment on this blog.

  • http://My.ComMetrics.com Urs E. Gattiker

    Hi Tech Admin: nnYes, of course, sentiment analysis could be used for the stock market as well. nnIncidentally, do you know of any great examples regarding the stock market and how sentiment analysis was used successfully? Appreciate a comment with some links of you have any to share.nnThanks for sharing.nnLooking forward to your next comment on this blog.

  • http://My.ComMetrics.com Urs E. Gattiker

    Hi Tech Admin: nnYes, of course, sentiment analysis could be used for the stock market as well. nnIncidentally, do you know of any great examples regarding the stock market and how sentiment analysis was used successfully? Appreciate a comment with some links of you have any to share.nnThanks for sharing.nnLooking forward to your next comment on this blog.

  • http://My.ComMetrics.com Urs E. Gattiker

    Hi Tech Admin: nnYes, of course, sentiment analysis could be used for the stock market as well. nnIncidentally, do you know of any great examples regarding the stock market and how sentiment analysis was used successfully? Appreciate a comment with some links of you have any to share.nnThanks for sharing.nnLooking forward to your next comment on this blog.

  • http://My.ComMetrics.com Urs E. Gattiker

    Hi Tech Admin: nnYes, of course, sentiment analysis could be used for the stock market as well. nnIncidentally, do you know of any great examples regarding the stock market and how sentiment analysis was used successfully? Appreciate a comment with some links of you have any to share.nnThanks for sharing.nnLooking forward to your next comment on this blog.

  • http://My.ComMetrics.com Urs E. Gattiker

    Hi Tech Admin: nnYes, of course, sentiment analysis could be used for the stock market as well. nnIncidentally, do you know of any great examples regarding the stock market and how sentiment analysis was used successfully? Appreciate a comment with some links of you have any to share.nnThanks for sharing.nnLooking forward to your next comment on this blog.

  • http://My.ComMetrics.com Urs E. Gattiker

    Hi Tech Admin: nnYes, of course, sentiment analysis could be used for the stock market as well. nnIncidentally, do you know of any great examples regarding the stock market and how sentiment analysis was used successfully? Appreciate a comment with some links of you have any to share.nnThanks for sharing.nnLooking forward to your next comment on this blog.

  • http://My.ComMetrics.com Urs E. Gattiker

    Hi Tech Admin: nnYes, of course, sentiment analysis could be used for the stock market as well. nnIncidentally, do you know of any great examples regarding the stock market and how sentiment analysis was used successfully? Appreciate a comment with some links of you have any to share.nnThanks for sharing.nnLooking forward to your next comment on this blog.

  • http://My.ComMetrics.com Urs E. Gattiker

    Hi Tech Admin: nnYes, of course, sentiment analysis could be used for the stock market as well. nnIncidentally, do you know of any great examples regarding the stock market and how sentiment analysis was used successfully? Appreciate a comment with some links of you have any to share.nnThanks for sharing.nnLooking forward to your next comment on this blog.

  • http://My.ComMetrics.com Urs E. Gattiker

    Hi Tech Admin:

    Yes, of course, sentiment analysis could be used for the stock market as well.

    Incidentally, do you know of any great examples regarding the stock market and how sentiment analysis was used successfully? Appreciate a comment with some links of you have any to share.

    Thanks for sharing.

    Looking forward to your next comment on this blog.

  • Pingback: alysha myers

  • Bernard Ferrer

    100% accurate automatic sentiment analysis is certainly today a mirage. However, if you accept it as an excellent productivity tool, it certainly makes sense. This implies that the tool presents verbatims with a confidence rating on their tonality and allows tonality correction. We use such a tool and we observed on our 3 brands about 1200 verbatims extracted monthly. We focus on veratims with less than 50% confidence rating…about 450 verbatims to review. Our marketing expert reviews an average of 600 verbatims/day. so, globally, with less than one man day, we obtain excellent results from our system.

    • http://My.ComMetrics.com Urs E. Gattiker

      Dear BernardThanks for the comment about fully automated sentiment analysis not being a mirage.You provide some very interesting and helpful information for sure. I just wonder about some things like:A – how do you define a quote or part of a text that you take verbatim that has a less than 50% confidence rating For instance, what does verbatim mean exactly, 100 words of text or there abouts?As well, do you use the term confidence interval defined the way statisticians do?B – is your expert doing this work in one language for 3 brands or several such as English, French or Chinese?What about several languages or does it suffice to do this important work for 1 language?nC – Finally, if you take the 1200 verbatims extracted monthly for your 3 brands, what do you do with the information? How do you analyse these data?nWhat actions are taken based on these data?If your expert reviews 600 quotes a day about your 3 brands, what method does she use to mark the quotes according to which criteria (e.g., positive versus negative)?Bernard, thanks so much for sharing and I hope you can share some more by replying my questions above.

  • http://My.ComMetrics.com Urs E. Gattiker

    BernardnnThanks for the reply regarding sentiment analysis and doing it for French and English with your pgoram for 3 brands.nnFor it is interesting that you crawl about “280 sites or blogs, RSS, and tweets” and then dump them into a database providing users with email alerts on specific triggers…nnThis sounds very neat indeed hence for you it is probably just a question of time until you have more languages included as needed.nnAlso interesting is that if I understand correctly, the system pre-assigns the positive, netural or negative value by scoring using a color scheme that the expert then changes if need be.nnBut my curiosity is still if you produce the 25 page report from these data each month, what kind of decisions can be made based on the information and what concrete steps do you take to make use (actionable metrics = once the data is in depending on them you take actions).nnThanks so much for sharing.nn

  • Pingback: H&K Sydney

  • Pingback: Guillem Garcia

  • seema

    nice 

  • http://www.articlebillboard.com/articledetail/42017/23/Investing/Finance/3/Optiontips.in Optiontips.in

    I really appreciate your post.

    It gives a good overview and points out that we have to distinguish between external and internal things we need to assess when doing the audit.

    Thanks for sharing this information.

  • http://twitter.com/cyhex cyhex

    Take a look at Twitter sentiment analysis tool http://smm.streamcrab.com, its written in python and uses Naive Bayes classifier with semi-supervised machine learning

    • http://commetrics.com/articles/2011-trends-get-better-roi-with-facebook-twitter-and-youtube/ Urs E. Gattiker

      Dear Cyhex

      Thanks for sharing this information.  I went over immediately to see how this works and I can share one example here:

      ==>  Twitter – sentiment analysis that makes little sense

      As you can see for yourself, it is very difficult to determine how this works and make sense out of the findings.  Particularly, how it interprets positive versus negative tweets remains unclear.

      Can you shed some light on this maybe?

  • http://twitter.com/hhaemmerle/status/103184392821809153 Helmut Haemmerle

    RT @ComMetrics: Sentiment analysis for online content: Honest? http://t.co/ODBaily

  • Pingback: The Marinoff Group

Previous post:

Next post: