Sriwijaya Air Review Sentiment Analysis

Context

This is an NLP coursework project at ITS: sentiment analysis on Google Play reviews of the Sriwijaya Air Mobile app. I scraped 1,799 reviews with google_play_scraper, kept 1,756 after cleaning, and built the full pipeline end to end: preprocessing, feature engineering, model comparison, evaluation, error analysis, and a Streamlit inference demo.

Labels are derived from star ratings: 1 to 2 stars is negative, 3 is neutral, 4 to 5 is positive. That makes them weak labels, a rating is a useful proxy for sentiment but not a human annotation, and the whole evaluation has to be read with that in mind.

Problem

Airline app reviews are mixed-language (Indonesian with English fragments), short, and heavily imbalanced: 806 negative, 863 positive, and only 87 neutral reviews. The interesting questions were less “can a classifier fit this” and more operational: which model holds up on a held-out split, where exactly does it fail, and can the output be turned into something a product team could actually use.

Approach

• Preprocessing for mixed Indonesian and English text: lowercasing, tokenization, bilingual stopword removal, English lemmatization, Indonesian and English stemming, contraction expansion, and word normalization.
• Features: TF-IDF over unigrams, bigrams, and trigrams, plus simple numeric features (word count, character count, average word length).
• Models: Logistic Regression, Linear SVM, and Naive Bayes compared on the same stratified 80:20 split, with random oversampling applied to the training split only to help the small neutral class.
• Evaluation on the held-out test set with accuracy, macro F1, weighted F1, per-class F1, and a confusion matrix.
• Beyond classification: a rule-based keyword taxonomy over negative reviews (app performance, check-in, ticketing, promo, login, payment, membership, customer service) to rank what users actually complain about.

Evidence

Key Decisions

Oversampling only the training split. Balancing the whole dataset before splitting would leak duplicated neutral reviews into the test set and inflate every score; keeping the test set untouched kept the evaluation honest.

Selecting the best model by macro F1 instead of accuracy. With 87 neutral rows against 1,669 others, accuracy is nearly blind to the minority class; macro F1 is the metric that actually punishes ignoring it.

Treating the labels as weak labels explicitly. Rating 3 reviews mix complaints, suggestions, and mild praise, so neutral F1 of 0.15 is reported as a data problem to solve with manual annotation, not hidden behind the strong 0.88 negative and positive F1 scores.

Result

Logistic Regression was the best of the three models with 0.852 accuracy, 0.636 macro F1, and 0.845 weighted F1 on the held-out test set, against 0.841 accuracy for Linear SVM and 0.818 for Naive Bayes.

The aspect analysis turned the model output into product insight: negative reviews concentrate on promos, ticketing and booking, membership features, and the app failing to open or load. The pipeline ships with a model card documenting intended use and limitations, and a Streamlit demo for classifying new review text.

What I’d Improve

Manually annotate 100 to 150 reviews to measure how far rating-based labels drift from human judgment, add cross-validation with hyperparameter tuning, and test IndoBERT or another multilingual transformer once a stronger labeling baseline exists. The neutral class needs data work before it needs a bigger model.