Apple: Mix, Don't Tune — Bilingual Pre-Training Beats Hyperparameter Search

Apple ML Research published "Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings" in October 2026, and its answer to a common problem is blunt: if you do not have enough data in your language, add data from another language instead of fiddling with regularization. Across about 1,000 pre-training runs, mixing English into Arabic training beat hyperparameter tuning at every model size the team tested, and the advantage widened as models grew.

The source is Apple's research page, which lists the paper as arXiv 2605.13225. The authors are Paul Jeha (DTU, work done while at Apple), Anastasiia Sedova, Louis Béthune, Skyler Seto, Jes Frellsen, Pierre Ablin and Natalie Schluter.

TL;DR: what the paper says

What is the data-constrained problem?

Most pre-training recipes assume you can see fresh text for every training step. For English or Chinese that is true. For most of the world's languages it is not: the whole usable corpus may be a few billion tokens, so a model large enough to be useful sees the same documents several times. Repeated epochs lead to memorization and a model that fits the training set better than it generalizes. We have covered the same pressure at the frontier in Babuschkin's pretraining feedback-loop argument and in the open-weight bilingual push behind Aleph Alpha's Kolibri.

The standard response is to regularize harder. Weight decay shrinks weights and limits what the network can memorize. The paper treats this as the baseline to beat.

Generated data flowing into a model, an image for how Mix, Don't Tune adds helper-language text to a small corpus

Why mixing beats tuning, according to the authors

The paper argues that mixing does two jobs at once. It regularizes by diversifying the training signal, so the model cannot just memorize the small target corpus. And it adds knowledge that the repeated target text cannot supply, such as world facts and reasoning patterns that English data carries.

The authors note something worth remembering when you evaluate your own runs: target-language validation loss underestimates the value of mixing. In their words, "validation loss captures only the first effect." Downstream task accuracy shows more of the second effect, which is why the downstream gains (2-13x data-equivalent) are bigger than the loss gains (2-3x).

What are the practical recommendations?

The research page boils the findings to three rules:

1. Mix in a high-resource language. If your target language is data-constrained, include a large language alongside it.
2. Tune the mixing ratio before the hyperparameters. The ratio is the lever that moved results most.
3. Transfer hyperparameters from a small proxy using muP. Maximal Update Parametrization lets settings found on a small model carry to a larger one, so you avoid searching at full scale. Apple lists its related work on completed hyperparameter transfer across modules, width, depth, batch and duration (ICLR, February 2026) on the same page.

Rule three matters for cost. Hyperparameter search at 1B parameters is expensive; searching on a 150M proxy and only tuning the mix is much cheaper.

How does this fit Apple's other research?

Apple has been publishing a steady run of efficiency-minded work. See SelfCompact for agent context, Minions for distributed inference on idle Macs, normalizing trajectory models and the machine-translation quality annotation study. The thread is practical: get more from smaller models and less data. Language coverage is a product concern for a company shipping on-device features to many markets, so a recipe for strong small models in under-resourced languages has obvious value, though the paper does not mention any product use.

What are people likely to ask next?

Is this just multilingual training with extra steps? Partly. Multilingual pre-training is not new, and many open models already train on dozens of languages. What the paper adds is a controlled comparison in the data-constrained regime: same target data budget, same model sizes, one arm that mixes and one that tunes. Most multilingual releases do not publish that ablation, so it is hard to know how much the helper language actually contributed versus the recipe around it.

Why Arabic? It is a language with large speaker numbers but far less web text than English, written in a different script with rich morphology. That makes it a fair stress test: if a helper language only worked for closely related, same-script languages, Arabic would expose it. The authors still test one pair, so the honest reading is "works for this difficult pair", not "works everywhere".

Does the helper language have to be English? The paper uses English because it is the largest corpus available. Nothing in the framing says it must be, but the page does not test alternatives. A practitioner could reasonably try a related higher-resource language as the helper and measure.

What mixing ratio should I use? The page does not give a universal number, and we are not going to invent one. The recommendation is to treat the ratio as the main thing to tune, using a small proxy model and muP to carry the findings upward.

Why this matters beyond Arabic

Many teams building products for regional markets, such as national-language assistants, domain-specific models for law or medicine, or on-device features for smaller languages, face the same shape of problem: a small, valuable corpus and a temptation to push regularization until the training curve looks healthy. The paper's message is that the cheaper and stronger lever is usually the data recipe. That fits a broader pattern across recent research, where careful data choices beat clever optimization tricks at small and medium scale.

It also changes how you should read validation curves. If you only watch target-language loss, mixing looks like a modest win; if you watch downstream accuracy, it looks like a large one. Teams that stop at loss may reject a recipe that would have helped.

How to run your own mix-versus-tune experiment

A minimal version of the paper's idea for a team with a small compute budget looks like this:

1. Pick a small proxy model, in the range of a few hundred million parameters, and fix a target-language token budget you actually have.
2. Train a baseline arm on the target data alone for several epochs, with a sensible weight-decay sweep.
3. Train mixed arms at several ratios of helper-language data, holding the target-language budget constant.
4. Evaluate every arm on target validation loss and on downstream tasks in the target language.
5. Compare the best mixed arm to the best tuned arm, and note how far apart they are in terms of "how much more unique target data would the baseline need to match".
6. Carry the winning ratio and hyperparameters to the larger model using muP, then verify with one or two full-size runs.

This mirrors the paper's logic without claiming its exact protocol, which lives in the full text.

Why the "data-equivalent" framing is useful

Raw accuracy gains are hard to compare across projects. The paper expresses benefit as the amount of extra unique target data you would need to get the same result without mixing. For a team deciding whether to spend months collecting or licensing more text, "this mix is worth two to three times your corpus on loss" is a concrete budget argument. The downstream figure, two to thirteen times, shows the range depends strongly on the task and the model size, which is itself a reminder to evaluate on the tasks you care about instead of trusting one headline multiple.

What are the limits?

• One language pair. Arabic with English is a good test (different script, rich morphology), but results for, say, Swahili with French, or a related-language helper, are untested here.
• Scale tops out at 1.43B. The trend widens with size, but frontier models are far larger, and extrapolating a curve is not a measurement.
• Page-level summary. We read Apple's research page; the full paper has the experimental details, and anyone relying on the numbers should check the exact definitions of data-equivalence there.
• No code. The page links no repository.
• Mixing costs compute. Adding English means more tokens per step. The paper's comparison is about data efficiency for the target language, so budget accordingly.

What can builders do with this today?

If you are training or continuing pre-training for a smaller language or a narrow domain, the actionable checklist is short:

• Start with a mix, not a purge. Include a large-language or large-domain corpus and sweep the ratio first.
• Evaluate on downstream tasks, not only target validation loss, since loss understates the mix's value.
• Search hyperparameters on a small proxy with muP and transfer them up.
• Treat strong weight decay as a fallback, not a first resort.

For fine-tuning rather than pre-training, the finding is suggestive but not directly proven: the paper is about pre-training from scratch.

Related reading

• Apple finds code-switched speech with language-selective encoder dimensions
• Aleph Alpha Kolibri sovereign bilingual MoE
• What is sovereign AI?
• Babuschkin on pretraining feedback loops
• Apple SelfCompact
• Apple Minions
• Apple translation quality study

Figures come from Apple's research page as of October 13, 2026; confirm details in the paper.