Sentiment Analysis and NLP on Amazon Reviews
Sentiment analysis model using python
Project Overview
I have created a classification model that is based on sentiment analysis and NLP. I followed the data science lifecycle in which I gathered the data and created a predictive model with it. Using data gathered from Amazon website, I decided to analyze board game items since that was something I enjoyed. I scraped ~6k rows of reviews for all types of board games in which I trained the classification model with.
Sentiment Analysis and NLP
Sentiment Analysis plays an important role in businesses today. In today’s world, there are large amounts of data that are collected and analyzed so businesses can use to improve their services and products. In order for companies to understand and improve their services and products, they are constantly analyzing an endless amount of consumer reviews.
“Sentiment analysis uses natural language processing and machine learning to interpret and classify emotions based on subjective feedback.”
Consumer reviews can come in different forms of data which include text or speech. Customer feedback can vary depending on the company but are typically in the form of number of stars, range of numbers, or a simple positive or negative rating. In addition, customers are able to type their feedback which come in the form of text. Because companies have saved all this data in their databases, it is impossible for them to manually analyze each review. To save time and money, this is where sentiment analysis comes in handy where it is possible to analyze reviews at scale.
NLP stands for Natural Language Processing and is used to help computers understand the human language, in the form of text or speech. Normally a computer does not know how to interpret and understand the human language — all they see are letters and words. The goal of NLP is to have computers read and decipher the natural language. As you already know, Python is a valuable tool and is the best choice for machine learning allowing NLP easy to implement.
Examples of NLP include:
- Email/spam filters
- Voice translators for voice assistants such as Cortana, Alexa, Siri
- Language translator
- Predictive text corrector
There are a lot more use cases for NLP, but it goes to show that NLP is actually very important! We won’t go over too much on how NLP works, but I highly recommend you do more research on how it works.
Web Scraping
Like all models, we first need to collect our data. As mentioned due to time constraint and learning experience, I only concentrated on one category: board games. Scraping board games was more than enough data to get me started on my project. I used BeautifulSoup to help parse HTML information from Amazon webpages. There were a few challenges when I first ran my scraper for the first time. This was one of the blockers when running my web scraper:
“Sorry, we just need to make sure you’re not a robot. For best results, please make sure your browser is accepting cookies.”
I encountered this many times thinking it was my code that was breaking, but after doing research Amazon webpage actually discourages web scraping. In order to bypass the error message, I decided I need to be “nicer” to the Amazon server. I scraped my data in intervals while also adding a VPN to prevent Amazon from blocking my IP address. Another blocker I encountered was that their HTML page structure makes it hard to parse through the webpage. Parsing through everything resulted in some bad data but it was manageable.
The order in which I scraped my data:
- Scrape general information for each item from all search page. This includes asin id (product id), name of the product, average # of stars, and number of ratings.
- Scrape the first few pages of reviews for each product id. This includes headline, feedback description, and user rating.
Preprocessing/NLP
Using the Python library Natural Language Tool Kit (NLTK), this makes handling NLP very simple and effective. NLTK helps with removing stop words, filtering and cleaning, and feature engineering such as unigrams and bigrams. Once we clean up the data, our reviews are stripped of unnecessary words, punctuation, and more.
Now that our reviews are all preprocessed, we can explore them. Below we have the most common positive and negative words in all reviews. NLP can be difficult and is nowhere perfect (yet). For example, there is a board game named “Sorry” but the text processor would list “Sorry” as a negative word.
Modeling
Before we do any type of modeling, let’s first address the class balance problem. User ratings contain values between 1–5 which I split the user rating into 2 binary classes, “GOOD” represents ratings of 4–5 and “BAD” represents ratings of 1–3. On the left plot, it shows the number of ratings per class and clearly there a lot more “GOOD” values than “BAD” values. To resolve this problem, I randomly sampled 250 from each class which is represented on the right plot.
Once we have our 2 binary classes, we split the data into testing and training data. We then use machine learning algorithms to train the data in which I tested a few of them in the model.
Based on the outcome, the best performing algorithm is Vader with an accuracy of 82.25%. Vader which stands for Valence Aware Dictionary and Sentiment Reasoner, is used for text sentiment analysis that relies on a dictionary of emotional intensity for text. For the other algorithms, it didn’t do so well compared to Vader since they don’t specialize in text data.
Github can be found here: https://github.com/ttam37/sentiment-analysis-amazon-reviews
Sources
- https://monkeylearn.com/sentiment-analysis/
- Flatiron School curriculum
