Curating large multilingual corpora for language models
Lessons from building 100B-token corpora across Turkish, Dutch, Greek, and Bulgarian — and why, for smaller languages, the dataset work is most of the model work.
Everyone wants to train the model; far fewer want to build the data. But for languages outside the English-dominated web, the corpus is the hard part, and model quality tracks the quality of the pipeline that produced its data far more than it tracks another clever architecture tweak.
Coverage is a data problem, not a model problem
A base architecture that does well in English will happily underperform in Turkish or Greek if it never saw enough clean text in those languages. That's the motivation behind the per-language corpora — Turkish at 100B tokens, Dutch at 200B, Greek at 150B, Bulgarian at 33B, and more. They exist so a model starts from real coverage instead of translationese, and so that a 3B-parameter multilingual model trained on top of a 6T+ token mixture has something genuine to learn from in each language.
Clean, dedup, then clean again
Web-scale text is mostly noise: boilerplate, near-duplicates, and documents mislabeled by language. The unglamorous passes — deduplication, language identification, quality filtering — do more for downstream quality than another epoch on dirty data. When the corpus is going to be trained on, a safety-classification pass matters too; what you don't filter, the model will happily memorize.
Build the retrieval data alongside
Language modeling isn't the only consumer. Rerankers and embedding models need hard negatives, not just positive pairs, and mining those — from code and from text, while keeping them honest — is its own dataset-building effort. It pays off directly: better negatives make better retrieval, and retrieval is what makes a model useful on top of a knowledge base.
Reproducibility over heroics
The most useful thing a dataset can be is reproducible. A one-off scrape that trained one good model helps nobody else; a documented pipeline that others can rerun, audit, and extend compounds. That's the bar I hold the corpora to — not "look what I trained," but "here's the data, and here's how it was made."