Guest Author: Zishan Ansari from the eBook
The Future of Social Media 2013: Predictive Analytics
Zishan is a SAS certified predictive modeler with over 4 years’ analytics consulting experience for global players in banking, insurance and retail domain. Zishan is currently part of Target Corporation’s enterprise business intelligence team based in India that provides analytic solutions to problems in operational risk, loss prevention and profit protection area.
Source -http://provalisresearch.com/uploads/Sentiment.jpg If you are reading this, you probably are aware of how things in the social media world usually work. You probably have used/seen/heard of tools that tells you about the sentiment of your customers, by analyzing what they are writing over various social media platforms. And I am sure many of you would have also seen software providers making tall claims about how accurate their engines are in predicting the correct sentiment of a company’s customer base. There is no question about whether sentiment analysis is a powerful analysis technique or not, because it certainly is. The bigger question is whether it is the right thing for you or not? And if it is, then do you have the ability to apply it correctly or not?
If you are looking for an answer to the question whether investment on sentiment analysis is the right thing for your business, then read further. In case you have already spent loads of cash on buying a sentiment analysis package, still read further to know how you can best utilize it.
Why sentiments?
Being a part of a smart and proactive organization that listens to all its customers, you would want to build a strategy where you:
- Listen to all that your customers are saying about your brand or a newly launched product
- Take actions based on customer sentiments
- Redress customer grievances by offering him or her some brownie points
- And in turn be loved by your customers for being a great listener
Well, this all works really well when you are running a ‘Mom and Pop Store’. Why? Because:
- You know who your customers are
- You understand their language
- You understand their moods, their emotions
- You know their preferences
- You know what will please/displease them
How are things different in social media?
Simple answer to this would be – in more ways than one. It is important for you to understand these differences to figure out whether you need sentiment analysis or not. .
Not every person is good in expressing themselves in text, and definitely not in just hundred and forty characters. And this poses a big challenge.
Why? Because, there is no way to capture customer’s facial expression through social media, nor is there a way to capture voice modulation to figure out customer’s mood.
With deliberately excluding the discussion about misclassifications that majority of the tools anyways do, there is one more aspect of language that I think should be considered, and that is dialects. In this very world of ours, where spoken languages change with every two hundred to three hundred kilometers, it will only be illogical for us to expect everyone to express their sentiments in the language in which our software’s NLP engine is trained.
OK. So in that case will it be insane for us to demand for an NLP engine that has the functionality of identifying the sentiments from any language/dialect. No, it wouldn’t be completely insane. But it is very likely that you will be paying a bomb to get such a service.
Let us assume that you get an NLP engine that has all these powers. Tell me who writes a perfect language these days? If you think your customer does, then you are absolutely wrong. Even Shakespeare would have had a hard time in expressing himself in hundred and forty characters forget about an average Joe (Pardon me if you are Shakespeare or Joe!). We are living in an era where even native speakers are getting bad in grammar day by day. Languages are evolving, and so are the NLP engines. And, more evolved an engine is, more dollars you have to burn for it.
Knowing your objective
You will be solving the major part of the puzzle if you figured out what your objective is behind sentiment analysis. Because, after this you just have to pick the right approach to achieve your goal.
If you are managing a customer service center and you receive 100,000 e-mails every day, and would like them to be classified as Positive, Neutral or Negative then you can use software that has the capability to do the job.
That’s all the functionality you need to achieve your goal. Rest of the work will be done by the executives to ensure each customer is satisfied.
Looking beyond sentiments
Say if you had kids. You find out on a Sunday morning that your kids were annoyed about something and were not looking happy. Being a good parent you give them permission to go out with friends. If you are a more involved parent, you would yourself go out with your kids, to make their day. A parent very well knows what to do to change the mood of their kids.
As a kid I have really enjoyed that. Sometimes getting annoyed for no good reason can also get you treats, just because your parents wants you to be happy always.
But if you were a responsible parent, what else would you do? You would probably try to understand, why were your kids upset at the first place? Have they been getting annoyed quite often recently? What are the circumstances which makes them unhappy?
A responsible company is no different from a responsible parent in this aspect. It will do a root cause analysis of what actions led to all this. And they can take actions to ensure that such circumstances do not arise in future. An organization with a dedicated customer service team can probably do all what a responsible parent do, given that they know who the customer is. It can spend resources to get in touch with the customer, understand his/her concerns, identify the causes of unhappiness and then act accordingly. It can also go ahead and analyze what were the things that led to such circumstances.
So what should you do? Should you wait for a cheaper NLP engine with all the functionality? Or should you wait until your company buys a foolproof text-analytics software license?
It is important to be clear about what exactly is your objective to perform sentiment analysis. You can then combine the results from a not so very expensive tool with other free tools or with your own data visualization tools for analysis.
Stream Graphs
One such freely available tool is stream graph. It helps you in analyzing what people have been tweeting about a brand/product over a period of time.
Figure: (Source – http://www.neoformix.com/Projects/TwitterStreamGraphs/view.php)
The Stream Graph shows the usage over time for the words, which are highly associated with the search word. One of these series together with a time period is in a selected state and colored red. The tweets that contain this word in the given time period are shown below the graph. You can click on another word series or time period to see different matches. In the match list you can click on any word to create a different graph with tweets containing that word. You can also click on the user or comment icons and any URL to see the appropriate content in another window. If you see a large spike in one time period that hides the detail in all the other periods it will be useful to click in the area to the left of the y-axis in order to change the vertical scale.
The bad part is that the free version analyzes latest thousand tweets. The good part is that there are open source resources (http://www.processing.org/) available to help you develop such graphs. Probably you need to invest some time to develop that. Remember, there are no free lunches and there are no free graphs either.
You can combine the results of the sentiment analysis with the results of stream graph analysis to infer what circumstances probably led to positive or negative sentiments. So even if your sentiment analysis says that 80% of your customers have neutral opinion and your sales figures are way below your forecasted sales, a stream graph can help you identify what probably is causing the low sale. Stream graphs present one way of visualizing the data. There can be multiple such ways of analyzing it.
A good analyst will always find a suitable way of presenting the appropriate information that you would like to have in order to validate your sentiment analysis findings. And remember without validation, your results from sentiment analysis in the worst case can be as bad as that from a random classifier.