NLP
May 2024 – Jun 2024  ·  Associated with IBM

Sentiment Analysis
of IMDb Reviews
Using IBM Watson

User reviews are a goldmine of insights in the entertainment industry. This project leverages IBM Watson NLP & AI to extract emotions, trends, and audience preferences from thousands of IMDb movie reviews — enabling real-time, multilingual, and scalable sentiment intelligence.

View on GitHub ↓ See Results
Project Snapshot
IBM Watson
Dataset
50,000 IMDb reviews
Best Model Accuracy
85.6% — Multinomial NB
Classification
Positive / Negative / Neutral
Project Duration
May 2024 – Jun 2024
IBM Internship
Data Science Intern · Remote
Jun 2024 – Jul 2024 · 2 months
Main Contributor
Priyanka Gandhi A
Final Year ECE · Avinashilingam University

Collaborator
Reshma Sherin A
Final Year ECE · Avinashilingam University
50K
IMDb reviews analysed for sentiment patterns
85.6%
Best classification accuracy achieved
3+
ML models benchmarked & compared
7+
Emotions detected via Watson NLU
00

Work Experience

Data Science Intern
IBM · International Business Machines
Jun 2024 – Jul 2024
2 months
Remote
Internship
Role Overview

Worked as a Data Science Intern at IBM, applying the fields of artificial intelligence and data science in conjunction with the IBM Watson and IBM Cloud platforms for problem-solving and fostering innovation. Contributed to building scalable NLP pipelines, integrating Watson APIs, and delivering data-driven insights through advanced machine learning techniques.

Key Responsibilities
  • → Designed and implemented end-to-end NLP pipelines using IBM Watson NLU for sentiment classification, emotion detection, and entity extraction at scale.
  • → Applied BERT language models and deep learning techniques for advanced contextual text understanding, improving accuracy beyond classical ML baselines.
  • → Leveraged IBM Cloud infrastructure to deploy scalable AI services and automate data workflows, ensuring real-time analysis capabilities.
  • → Built data analytics dashboards using Microsoft Power BI to communicate model outcomes and business intelligence findings to stakeholders.
  • → Collaborated on SQL-based data querying and preprocessing pipelines, ensuring clean, reliable input for machine learning models.
  • → Managed version control and collaborative development workflows using GitHub, maintaining clean commit history and documentation standards.
Skills Applied at IBM
Machine Learning
Scikit-Learn
Neural Networks
Deep Learning
BERT (Language Model)
NLP
NLP Libraries
Sentiment Analysis
TF-IDF
IBM Watson
IBM Cloud
Artificial Intelligence
Microsoft Power BI
Data Analytics
Python (Anaconda)
GitHub
SQL
Data Science
Problem Solving
Troubleshooting
Machine Learning & DL NLP IBM / AI Platforms BI & Visualisation Dev & Data Engineering General
01

Skills Demonstrated

Natural Language Processing
Core NLP pipeline from raw text to structured sentiment features
Sentiment Analysis Emotion Detection Text Preprocessing Stopword Removal Lemmatization HTML Tag Stripping TF-IDF Vectorization Sarcasm & Context Awareness Multilingual NLP Language Detection NLTK WordNetLemmatizer
Machine Learning
Model building, evaluation, and comparative benchmarking
scikit-learn Multinomial Naive Bayes Random Forest Logistic Regression BERT (Language Model) Neural Networks Deep Learning Classification Report Confusion Matrix ROC / AUC Curves F1 Score Train-Test Split Feature Engineering SQL
IBM Watson & Cloud AI
Enterprise AI services for production-grade NLP at scale
IBM Watson NLU IBM Cloud Watson SDK (Python) Watson API Integration Artificial Intelligence (AI) Automated Workflows Real-time Analysis Scalable NLP Pipelines
Data Visualisation & BI
Translating model outputs into actionable business visuals
matplotlib seaborn IBM Cognos Tableau Power BI Audience Sentiment Trends Genre & Actor Analysis Competitive Benchmarking Streaming Platform Insights
Data Engineering & Dev
End-to-end pipeline from data loading to API deployment
Python Anaconda pandas numpy Data Cleaning Data Analytics Deduplication Flask / FastAPI REST API Deployment Streamlit Jupyter Notebook Git / GitHub Troubleshooting Problem Solving
02

Key Objectives

For Data Science & NLP

  • AI-Powered Sentiment Classification

    Used IBM Watson NLU to classify reviews into positive, negative, and neutral, while detecting deeper emotions like joy, anger, and sadness.

  • Sarcasm & Context Awareness

    Leveraged Watson's contextual NLP capabilities to improve accuracy on sarcastic and ambiguous reviews that fool simpler classifiers.

  • Multilingual Support

    Analysed reviews across multiple languages using Watson's built-in language detection and translation capabilities.

  • Scalability & Automation

    Automated the end-to-end workflow via Watson API integration, enabling real-time sentiment analysis at enterprise scale.

For Business Intelligence

  • Audience Sentiment Trends

    Identified sentiment shifts before and after movie releases to predict audience reception and box-office performance.

  • Genre & Actor Sentiment Mapping

    Mapped sentiment variations across different genres, directors, and lead actors to surface data-driven creative insights.

  • Competitive Platform Insights

    Compared sentiment scores across Netflix, Prime Video, and Disney+ Hotstar for cross-platform business intelligence.

  • Sentiment-Based Recommendations

    Used sentiment insights to improve movie recommendation systems — surfacing content that resonates with specific audience emotions.

03

Technology Stack

🤖
AI & NLP Core
IBM Watson NLU
Enterprise-grade natural language understanding for emotion detection, sentiment classification, entity extraction, and contextual awareness — powering the AI backbone of this project.
☁️
Cloud & API
IBM Cloud Platform
Hosted Watson services, handled API key management, and enabled scalable, production-ready deployment of the NLP pipeline via REST APIs with Flask/FastAPI wrappers.
🐍
Data Science
Python Ecosystem
Watson SDK, pandas, numpy, matplotlib, seaborn, NLTK, and scikit-learn — covering data ingestion, preprocessing, feature engineering, model training, and evaluation.
📊
Business Intelligence
IBM Cognos & Tableau
Transformed raw sentiment scores into executive-ready dashboards, trend visualisations, and competitive benchmarking reports for entertainment industry stakeholders.
🧠
Classical ML
scikit-learn Models
Benchmarked three classifiers — Multinomial Naive Bayes (85.6%), Random Forest (85.0%), and Logistic Regression (77.6%) — with TF-IDF vectorized features.
🚀
Deployment
Flask / FastAPI + Streamlit
REST API endpoints for real-time inference and an interactive Streamlit dashboard for live review analysis, word clouds, and per-film sentiment tracking.
04

Methodology Pipeline

1

Data Ingestion & Validation

Loaded 50,000 IMDb reviews from CSV. Validated schema (review + sentiment columns), checked for nulls (zero found), and identified 418 duplicate records for removal — resulting in 49,582 unique entries.

pandasdata_frame.info()isnull().sum()drop_duplicates()
2

Text Preprocessing

Four-stage cleaning: (1) HTML tag stripping with regex, (2) lowercase conversion and stopword removal via NLTK, (3) WordNet lemmatization for root-form normalisation, (4) noise removal — non-alpha chars, repeated letters, extra spaces.

re.sub()NLTK stopwordsWordNetLemmatizerCleanedText column
3

IBM Watson NLU Integration

Passed preprocessed reviews through IBM Watson NLU for emotion classification (joy, anger, sadness, fear, disgust), sarcasm detection, and multilingual sentiment scoring via Watson API — providing richer labels beyond binary polarity.

Watson NLU APIEmotionOptionsLanguage DetectionIBM Cloud SDK
4

Feature Extraction

TF-IDF vectorization converts cleaned text to weighted term-frequency matrices. Logistic Regression variant: max_features=750, ngram_range=(1,1), min_df=50, max_df=0.7 to balance vocabulary coverage and noise suppression.

TfidfVectorizerfit_transform()max_featuresngram_range
5

Model Training & Evaluation

Three classifiers trained on 80% split, evaluated on 20% held-out set. Metrics: accuracy, F1 score, precision/recall per class, confusion matrix heatmaps, and ROC-AUC curves. Multinomial NB emerged best at 85.6%.

MultinomialNBRandomForestLogisticRegressionclassification_reportroc_curve
6

Business Intelligence & Visualisation

Sentiment scores mapped to genres, actors, and release timelines in IBM Cognos and Tableau. Competitive benchmarking across streaming platforms (Netflix, Prime Video, Disney+ Hotstar) delivered actionable audience insights.

IBM CognosTableauPower BIseaborn heatmaps
05

Model Results

Model Accuracy F1 Score Precision Recall Notes
Multinomial Naive Bayes Best
85.6%
0.856 0.86 0.85 Fast, probabilistic, strongest on TF-IDF text data
Random Forest Classifier
85.0%
0.850 0.85 0.85 Ensemble, parallel training; robust but slower on sparse matrices
Logistic Regression
77.6%
0.773 0.79 0.78 C=0.0001, heavy L2 reg; trained on 77% sample — lower by design
05b

Live Inference Examples

Jurassic Park
✓ Positive
"A groundbreaking blend of stunning visuals and thrilling storytelling … remains a timeless adventure that defined the blockbuster experience."
model: MultinomialNB
Fantastic Four (2015)
✗ Negative
"Lacking coherence and depth … falters with weak characterization and a disjointed plot, failing to capture the essence of its source material."
model: MultinomialNB
The Dark Knight
✓ Positive
"An intense, gripping saga showcasing Heath Ledger's mesmerising performance … stands as a triumph in the superhero genre."
model: RandomForest
Suicide Squad
✗ Negative
"A disjointed mess with flashes of potential … suffers from an uneven plot, underdeveloped characters, and tonal inconsistencies."
model: RandomForest
The Matrix
✓ Positive
"Revolutionary in both visuals and narrative … a mind-bending sci-fi classic that redefined action cinema."
model: RandomForest
The Happening
✗ Negative
"A misfire, featuring a disjointed plot and cringe-inducing dialogue … falls flat, lacking both suspense and coherence."
model: MultinomialNB

All 6 spot-check reviews classified correctly — 100% accuracy on manual validation.

06

Future Roadmap

01 / Enhancement
Real-time IMDb Scraping
Integrate live scraping of freshly published IMDb reviews and wire them directly into the Watson NLU pipeline for on-demand, real-time sentiment inference without manual batch processing.
IBM Watson · Web Scraping · Automation
02 / Accuracy
Deep Learning with BERT
Replace TF-IDF with BERT / RoBERTa embeddings and fine-tune transformer models to push classification accuracy beyond 93%, capturing semantic nuances beyond bag-of-words approaches.
Transformers · HuggingFace · BERT
03 / Scope
Multilingual Expansion
Extend the full pipeline to non-English reviews using multilingual BERT (mBERT) or XLM-RoBERTa, enabling global audience sentiment coverage across Tamil, Hindi, French, and more.
mBERT · XLM-RoBERTa · Watson Translation
04 / Product
Interactive Analytics Dashboard
Build a full Streamlit or Dash dashboard with live sentiment charts, per-film emotion breakdowns, actor/genre heatmaps, and streaming platform competitive comparisons — ready for studio use.
Streamlit · Power BI · IBM Cognos
© 2024 Priyanka Gandhi A · All rights reserved ⭐ Star on GitHub