Working With Low-Resource and Multilingual Text

Most of what we know about language models, we learned in English. The tools and techniques were built for English. The intuitions that more data solves most problems and that a model's mistakes are random rather than structural were formed in a language that is very easily accessible. Once we step outside of this world and acknowledge the tons of different languages that are not that high in resources, the ground starts to shift. Suddenly, the techniques that felt reliable start failing in specific, patterned ways. I only recently started working with low-resource data myself, and the learning curve is real; problems that don't show up at all in high-resource settings turn out to be the whole game here.

The illusion of "just add more data"

The standard fix for a struggling model is more data. For high-resource languages, this usually works easily; scrape more text, fine-tune longer, watch the loss curve drop. However, for low-resource languages, it is often difficult because the data isn't available to add. Many languages have millions of speakers, but only a few thousand digitized sentences. Much of that text isn't useful enough for the model to learn.

Languages like Yoruba or Sindhi have tens of millions of speakers. But the digitized texts are thin and skewed toward religious or government documents, unrepresentative of everyday use. A model trained mostly on these data will struggle badly with casual conversation or news writing, no matter how many times you retrain it. Throwing more parameters at a problem doesn't help if the underlying data was never diverse enough to teach the model what it needs to know. The issue isn't volume. It's coverage.

Tokenization isn't neutral

Models don't see words and letters as we do. They see tokens before they can learn a pattern. The tokenizers are trained on the same biased data as everything else. A sentence in English might break into five or six clean subword tokens, because the tokenizers had enough resources to learn the pattern. The same sentence, translated into a low-resource language, can fragment into two or three times as many tokens, each carrying less semantic weight.

A common tokenizer might split a single Bengali word into two or three fragments, where the English equivalent is one token. For example:

English
  input:  "I love learning."
  tokens: ["I", " love", " learning", "."]

Bengali
  input:  "আমি and বাংলা"   (I and Bengali)
  tokens: ["আ", "মি", " and", " বাং", "লা"]

More tokens mean more computation, slower inference, and less effective context for the same amount of meaning. A model seems to be "worse" at these languages because it is working with a distorted map of them before training even starts.

Data scarcity is not data quality

Although "low-resource" sounds like one single category, data scarcity and data quality are two separate problems. A language can have a reasonable amount of text available, and if that text is riddled with mistranslation, inconsistent spellings, or heavy code-switching that no cleaning pipeline was built to handle, then it is still considered to be poorly served. Datasets scraped from social media, for instance, often contain sentences that shift between two or three languages mid-sentence.

Consider common idioms in English and Bengali, and how a model handles them:

• English idiom. "It's raining cats and dogs" is correctly understood as heavy rain.
• Bengali idiom. "আকাশ ভেঙে পড়া", meaning to feel devastated, is often mistranslated literally as "the sky is falling."

Without massive conversational datasets, non-English text tools struggle with cultural context. Before asking "do we have enough data," it's worth asking "is the data we have actually trustworthy?"

Evaluation metrics lie to you

Usually, BLEU or accuracy scores are the metrics used to evaluate a model's performance in low-resource settings. But the trap is that these scores are only as good as the data they're tested on. For low-resource languages, the test data is often insufficient to catch the problems. A model can score well on a benchmark and still fail in the real world.

Take Hindi as an example. A model evaluated only on formal, standardized Hindi that is found in news articles or textbooks might score impressively well.

• Benchmark sentence. "आपका दिन शुभ हो" (Have a nice day). Model translation: accurate, fluent, natural.
• Regional or colloquial input. "काम हो गया क्या?" (Is the work done or what?). Model translation: often stiff, overly literal, or just wrong.

The regional dialects, spoken by tens of millions of Indian people, barely show up in these benchmarks at all. The metric is only answering a question based on the context it was given. A high score can either mean the model works well or the model failed to recognize the gaps.

The ethics of representation

Building a dataset for an underrepresented language is a hard task. Somebody has to carefully decide what counts as "the language" when the dataset gets built.

For example, Arabic has more than 25 spoken dialects: Egyptian, Levantine, Gulf, Maghrebi, and more. Each is different enough that speakers sometimes struggle to understand each other. Most Arabic NLP tools are trained almost entirely on Modern Standard Arabic (MSA), which is the formal, written version used in news and official documents.

• MSA input. "كيف حالك؟" (How are you, formal)
• Egyptian Arabic input. "إزيك؟" (How are you, colloquial)

There is a high chance that models trained only on MSA might not even recognize the second sentence as a greeting. It doesn't mean that the model is flawed; rather, the people who built the dataset made a choice. That choice quietly sidelines the way most people actually speak day to day.

This is where dataset building brushes up against ethics when it comes to marginal languages. Deciding which dialect to standardize, which speech to clean out as "noise," which regional variant to leave behind. These are not just small, practical decisions; they're the judgment calls about whose voice gets represented.

Practical starting points

Despite having so many challenges, low-resource language work is not a lost cause. We only need to adjust our tools and methods for diverse languages. There are a few approaches that are worth considering:

• Transfer from a related language. We can train a model on a related, higher-resource language first, then fine-tune it on the low-resource one. A model that already understands English grammar has a better chance at learning German, as they are structurally similar.
• Translate existing text. Existing text can be translated into the target language and used as extra training data. This may not be perfect, and it can introduce awkward, "translated-sounding" phrasing.
• Collect from real speakers. Text can be directly collected from real speakers, with their knowledge and consent. It tends to outperform scraped data even if collected in smaller quantities. A few thousand sentences from actual conversations often teach a model more than a million scraped from mismatched sources.
• Standardize spelling before training. Many low-resource languages don't have one standardized spelling system. Cleaning and standardizing this before training, rather than after, saves a lot of headaches later.

Closing thoughts

Working with low-resource languages means confronting problems that simply don't exist in English-first NLP: tokenizers that fragment meaning, benchmarks that measure the wrong thing, and datasets that reflect one dialect while ignoring the rest. Simply put, low-resource languages need methods built for their actual constraints, not a smaller, repurposed version of what works for English. Getting this right matters beyond benchmarks; it decides whose language gets represented in the tools everyone increasingly relies on.