Multilingual Pizza Review Sentiment

Classify pizza reviews as negative, mixed or positive, when some are in English, some in Malay, and the labels come from a score someone typed next to their text. The dataset had 902 rows, and it was messier than it looked: invalid scores like −42 and 1010, language labels that were really fragments of review text, and the same review repeated with different scores.

With data that small and that noisy, the model is the easy part. Deciding what the target is, what to do with duplicates, and how to handle two languages without leaking anything between splits decides most of the result.

• Scores23 rows had missing, non-numeric or out-of-range scores and were dropped, leaving 879 valid ones.
• LanguagesEnglish (488 valid rows) and Malay (389) were kept. Chinese, “Guga” and “Nippon” had one usable row between them, and labels like “Come on ...” turned out to be corrupted records.
• Three classes, not five or a scoreScores 0 to 3 became Negative, above 3 to 7 Mixed/Neutral, above 7 Positive. Predicting exact scores meant fitting noise, and five classes left two of them with 42 and 33 rows.
• Duplicates, then the splitOf 51 groups of repeated reviews, 42 disagreed on the score and 15 still disagreed after mapping to three classes. Those contradictory groups were removed, the rest collapsed to one row each, and only then were 766 unique reviews split 70/15/15, with zero overlapping texts between splits.

Each strategy used the same split, labels and reference SimpleRNN, with a tokenizer fitted only on its own training text.

• A. Both languages, raw text

  Val macro F1 0.842English 0.822Malay 0.870

• B. Both, with a language token first

  Val macro F1 0.883English 0.920Malay 0.841

• C. A separate model per language

  Val macro F1 0.860English 0.822Malay 0.912

• D. Malay translated to English

  Val macro F1 0.763English 0.795Malay 0.713

A single token, lang_english or lang_malay, at the start of each review beat everything else overall. Separate models helped Malay but hurt English and doubled the moving parts. Translating Malay lost the most, likely the phrasing that carried the sentiment.

Five recurrent families were compared on the language-token input with everything else fixed. The plainest won: SimpleRNN reached 0.880 validation macro F1 against 0.857 for GRU and 0.848 for LSTM, with the smallest gap between training and validation. On about 540 short reviews, gating added capacity the data couldn’t use.

Improvements were then tested one at a time on that family: epochs, sequence length, capacity, bidirectional reading, regularisation, class weights, learning-rate schedule and extra training rows made by splitting reviews into fragments. Bidirectional reading and recurrent dropout of 0.2 earned their place; L2, layer normalisation, class weights and the fragment augmentation didn’t, and were left out.

• Network

  Value Bidirectional SimpleRNN, 64 units

• Input

  Value Language token + review, vocabulary 1,000, length 15, embedding 32

• Regularisation

  Value Recurrent dropout 0.2, dropout 0.3 after the RNN

• Training

  Value Adam 0.001, learning rate reduced on plateau

• Remove contradictions before splitting

  The same review with two labels teaches the model nothing, and if its copies land in different splits, the test partly measures memory. Resolving duplicates first, and then checking for zero shared texts across splits, closes both holes.

• Tell the model the language instead of translating it

  Translation was the obvious move and the worst performer. A one-token prefix keeps the original wording, lets one model share what the languages have in common, and costs nothing at inference.

• Let the simple model win

  LSTM and GRU are the textbook upgrades, but on this data they generalised worse. The comparison fixed everything except the family, so the result could be trusted and the smallest model kept.

• Keep only what validation supports

  Every addition was one controlled experiment judged on validation macro F1. Four plausible techniques didn’t help and aren’t in the final model, which keeps it explainable in one table.

• Labels come from scores, and people don’t score consistently: the duplicates proved it.
• Small splits mean the validation choices between close configurations could flip on another random split.
• Only two languages; the model knows nothing about code-switching within a review.
• The dataset’s source and licence weren’t given, so it isn’t in the repository and the notebook can’t be rerun as published.