Language Models and Languages: Why "Multilingual" Is an Illusion of Equality

When people see ChatGPT or Claude answer fluently in Swahili or Thai, the natural impression is that the model "knows" all languages equally well. It doesn't — and the gap isn't cosmetic. It runs straight through the architecture of how these models read and process text. What follows is a breakdown of where that gap actually comes from, how large it is, and what's being done about it. 1. The problem starts before the model "thinks" — at tokenization An LLM never reads text directly. Text is first cut into pieces (tokens) by a tokenizer, and only those tokens are what the model actually processes. Tokenizers are trained on corpora where English historically dominates — in Llama 3's training data, for instance, English and code made up 95% of the dataset, while every other language combined accounted for just 5%. Because of this skew, the tokenizer ends up "efficient" for English and noticeably less efficient for everything else: models generate text one token at a time, and for languages poorly represented in the tokenizer's vocabulary, this makes generation slower and more computationally expensive. How big is the gap in practice? One study covering 25 European languages and ten models put a concrete number on it: tokenization "fertility" ranges from roughly 1.2 tokens per word for English to roughly 3.1 for Greek and Maltese — a penalty of about 2.5x. Notably, the ranking of "efficient" vs. "inefficient" languages barely changes regardless of register — formal or casual text — with a correlation above 0.97 across styles. A similar picture shows up for languages using a different script. A comparison of several tokenizers on equivalent Polish and English text found that where Polish required 547 tokens, English needed only 377 on the same model (Mistral Small 3.2) — and this gap repeats across most tested models. 2. Why this isn't just about speed — it's about money and quality Tokenization inefficiency isn't only "responses in non-English are slower." Three effect