Sentiment Analysis of Wargaku Surabaya Apps Reviews using Naïve Bayes and Support Vector Machine (SVM) Methods

Li’izah  Nur Fadilah; Anik Vega Vitianingsih; Yudi  Kristyawan; Anastasia  Lidya Maukar; Nina Kurnia  Hikmawati

doi:10.51454/decode.v6i1.1565

Authors

Li’izah Nur Fadilah Informatics Department Universitas Dr. Seotomo
Anik Vega Vitianingsih Informatics Department Universitas Dr. Seotomo
Yudi Kristyawan Informatics Department Universitas Dr. Seotomo
Anastasia Lidya Maukar Industrial Engineering Department President University
Nina Kurnia Hikmawati Information System Department Universitas Komputer Indonesia

DOI:

https://doi.org/10.51454/decode.v6i1.1565

Keywords:

Naïve Bayes, Natural Language Processing, Sentiment Analysis, Support Vector Machine , Text Mining, Wargaku Surabaya Reviews

Abstract

Digital public service applications in Indonesia are increasingly used to improve citizens’ access to government services, generating large volumes of feedback that are difficult to analyze manually. Moreover, many previous studies focus on polarity-based sentiment, which may not adequately capture specific user emotions. This study analyzes feedback on the Wargaku Surabaya application by classifying emotions into five categories: anger, disappointment, sadness, pride, and happiness. A total of 1,406 texts were collected (2021–2025), with 1,386 retained after preprocessing. Data were primarily sourced from Google Play Store reviews, supplemented by comments from Threads and YouTube. The research employs text preprocessing, TF-IDF weighting, and lexicon-based labelling with the generated labels reviewed on a subset of the dataset before model training. Emotion classification was performed using Naïve Bayes (NB) and Support Vector Machine (SVM), evaluated via a train–test split and confusion matrix. Results show that SVM achieved 84% accuracy, 85% precision, 84% recall, and an 84% F1-score, outperforming NB with 58% accuracy. These findings indicate that SVM is more reliable for multi-class emotion classification in digital public services.

References

Aida Sapitri, I., & Fikry, M. (2023). Pengklasifikasian Sentimen Ulasan Aplikasi WhatsApp Pada Google Play Store Menggunakan Support Vector Machine. Jurnal TEKINKOM, 6(1), 1–7. https://doi.org/10.37600/tekinkom.v6i1.773

Andana, M. H., Daffa, M., Fitria, N. U., & Mujiastuti, R. (2023). Webinar & Workshop Natural Language Processing in the Life of Artificial Intelligence. Masyarakat LPPM UMJ. https://jurnal.umj.ac.id/index.php/semnaskat

Aprinastya, R., Jazman, M., Syaifullah, S., Rahmawita, M., Siregar, S., & Saputra, E. (2024). Comparative Analysis of Naïve Bayes Classifier and Support Vector Machine for Multilingual Sentiment Analysis : Insights from Genshin Impact User Reviews. JUSIFO: Jurnal Sistem Informasi, 10(2), 117–126. https://doi.org/10.19109/jusifo.v10i2.24876

Artanto, F. A. (2024). Support Vector Machine Berbasis Particle Swarm Optimization Pada Analisis Sentimen Anggota KPPS. Jurnal Fasilkom, 14(1), 75–79. https://doi.org/10.37859/jf.v14i1.6795

Atmaja, F., Wahyuni, E. D., & Agussalim. (2025). Analisis sentimen berbasis aspek pada sistem layanan pengaduan masyarakat di Kota Surabaya menggunakan metode latent Dirichlet allocation dan naïve Bayes. JATI (Jurnal Mahasiswa Teknik Informatika), 9(1), 527–534. https://doi.org/10.36040/jati.v9i1.12438

Damayanti, E., Vitianingsih, A. V., Kacung, S., & Cahyono, D. (2024). Sentiment Analysis of Alfagift Application User Reviews Using Long Short-Term Memory ( LSTM ) and Support Vector Machine ( SVM ) Methods. DECODE: Jurnal Pendidikan Teknologi Informasi, 4(2), 509–521. https://doi.org/10.51454/decode.v4i2.478

Faisal, M. R., Kartini, D., Saragih, T. H., & Arrahimi, A. R. (2022). Belajar Data Science: Text Mining Untuk Pemula I. https://www.researchgate.net/publication/359619425

Kacung, S., Bagyana, C. P. P., & Cahyono, D. (2024). Analisis sentimen terhadap layanan Samsat Digital Nasional (Signal) menggunakan metode SVM. Jurnal Mnemonic, 7(1), 118–122. https://doi.org/10.36040/mnemonic.v7i1.9557

Komarudin, A., & Hilda, A. M. (2024). Analisis Sentimen Ulasan Aplikasi Identitas Kependudukan Digital Pada Play Store Menggunakan Metode Naïve Bayes. Computer Science (CO-SCIENCE), 4(1), 28–36. https://doi.org/10.31294/coscience.v4i1.2955

Maulida, N., Suarna, N., & Prihartono, W. (2024). Analisis Ulasan Sentimen Aplikasi Mobile Jkn Dengan Algoritma Support Vector Machine Berbasis Particle Swarm Optimization. JATI (Jurnal Mahasiswa Teknik Informatika), 8(2), 1651–1658. https://doi.org/10.36040/jati.v8i2.9105

May, I. P. A., & Fanida, E. H. (2022). Analisis efektivitas aplikasi Wargaku Surabaya dalam menunjang pelayanan publik masyarakat Kota Surabaya. Publika, 11(1), 1553–1568. https://doi.org/10.26740/publika.v11n1.p1553-1568

Motger, Q., Oriol, M., Tiessler, M., Franch, X., & Marco, J. (2025). What About Emotions? Guiding Fine-Grained Emotion Extraction from Mobile App Reviews. 2025 IEEE 33rd International Requirements Engineering Conference (RE). https://doi.org/10.1109/RE63999.2025.00012

Nur, D., Widiyanto K, M., & Puspitaningtyas, A. (2024). Pelayanan pengaduan masyarakat melalui aplikasi “Wargaku Surabaya” sebagai perwujudan e-governance Kota Surabaya. Triwikrama: Jurnal Ilmu Sosial, 4(3), 1–10.

Purnamasari, D., Bayu, A., Desy, A., Fanka, W. A. P., Reza, A., Safrila, M., Yanda, O. N., & Hidayati, U. (2023). Pengantar Metode Analisis Sentimen. Gunadarma Penerbit.

Putri, D. A., & Muthia, D. A. (2024). Implementasi metode lexicon based dan support vector machine pada analisis sentimen ulasan pengguna ChatGPT. IJCIT( Indonesian Journal on Computer and Information Technology ), 9(2), 80–86. https://ojs.bsi.ac.id/index.php/ijcit/article/view/23948

Riehl, K., Neunteufel, M., & Hemberg, M. (2023). Hierarchical confusion matrix for classification performance evaluation. Journal of the Royal Statistical Society Series C: Applied Statistics, 72(5), 1394–1412. https://doi.org/10.1093/jrsssc/qlad057

Rifaldi, D., Fadlil, A., & Herman. (2023). Teknik preprocessing pada text mining menggunakan data tweet “mental health.” DECODE: Jurnal Pendidikan Teknologi Informasi, 3(2), 161–171. https://doi.org/10.51454/decode.v3i2.131

Rizaldi, S. A. R., Alam, S., & Kurniawan, I. (2023). Analisis Sentimen Pengguna Aplikasi JMO (Jamsostek Mobile) Pada Google Play Store Menggunakan Metode Naive Bayes. STORAGE: Jurnal Ilmiah Teknik Dan Ilmu Komputer, 2(3), 109–117. https://doi.org/10.55123/storage.v2i3.2334

Sathyanarayanan, S., & Tantri, B. R. (2024). Confusion matrix-based performance evaluation metrics. African Journal of Biomedical Research, 27(4S), 4023–4031. https://doi.org/10.53555/AJBR.v27i4S.4345

Zeng, G. (2025). Invariance properties and evaluation metrics derived from the confusion matrix in multiclass classification. Mathematics, 13(16). https://doi.org/10.3390/math13162609