Apple Detects Code-Switching Straight From Speech Encoders: Whisper-Small Hits F1 0.798

Apple Machine Learning Research has published a paper that finds a cheap way to spot code-switched speech, meaning audio where a speaker mixes two languages inside one sentence, without transcribing it first. The method, described on Apple's research page (published October 2026), reads language information that already lives inside pretrained speech encoders. Using the Whisper-small encoder, it reaches a mean F1 of 0.798 across 14 language pairs (see the arXiv paper) in the CS-FLEURS benchmark.

The result is small in scope but practical: it is a data-curation tool. If you want a speech model that handles bilingual speakers well, you first need to find bilingual speech in huge collections, and this is a way to do that with little compute. This post explains the problem, the method, the numbers, and the limits.

Quick reference

Why code-switching is a data problem

Code-switching is everywhere in real life: Hindi and English in a single message, Spanish and English in a conversation, Mandarin and English in a meeting. Speech recognisers trained on mostly monolingual data stumble on it. Apple's research page also lists a related paper from August 20, 2026 on Mandarin-English code-switching recognition with iterative pseudo-labeling, which attacks the same scarcity from the training side.

Apple's framing is straightforward: code-switching is "underrepresented in pretraining corpora". To increase coverage you need to find relevant utterances in large speech collections so they can be added to training data. The obvious way is to transcribe everything and look for language alternation in the text. That depends on accurate multilingual transcription, which is exactly what is unreliable on mixed speech. The method here skips transcription.

How the method works

Two speech bubbles in different languages joined by a bridge, illustrating code-switching detection in a speech encoder

The paper's approach can be read in three steps, based on the abstract.

1. Find language-selective dimensions. Feed monolingual recordings in each language through a pretrained speech encoder. Identify hidden-state dimensions whose activations are selective for one language. These are directions in the encoder's internal representation that light up for, say, English frames but not French ones.
2. Read them frame by frame. For a new utterance, aggregate the frame-level activations of each language's selective dimensions over time.
3. Decide. If both languages' dimensions are active within the same utterance, predict code-switching. A small code-switched set is used to tune the decision rule, such as a threshold.

The appeal is that nothing is trained from scratch. The encoder is frozen, the dimensions come from cheap monolingual data, and only a rule is tuned. Apple calls it "lightweight", and the structure supports that.

For readers who build with models, the idea is related to probing and activation-based monitors, which look for meaningful directions in hidden states rather than output text. Our coverage of activation probes as misalignment monitors in Gemini 4 Argon's safety report uses the same intuition for a different purpose: the model already represents the thing you care about; you just need to read it out.

The numbers

On CS-FLEURS, across 14 language pairs, the Whisper-small encoder reaches a mean F1 of 0.798. The comparison points are:

Two cautions. First, the gaps are modest: about 2 F1 points over the CNN and about 5 over HuBERT. They are a fair argument that the encoder carries the needed signal, not a claim that code-switching detection is solved. Second, F1 around 0.8 means roughly one in five decisions is wrong in some way, so a pipeline using it should expect noise and plan to filter or sample candidates before training on them.

Apple's conclusion is that "pretrained speech encoders already encode the information needed to identify CS utterances", and that the method is "a lightweight tool for building more representative multilingual corpora". That is a careful claim: it is about corpus building, not about recognising or translating mixed speech itself.

What this changes for builders

• Data curation gets cheaper. If you maintain a speech collection, you can score utterances for likely code-switching with a single encoder pass and a tuned rule, instead of running full multilingual ASR on everything.
• Evaluation sets. The same detector can help assemble test sets of mixed-language speech so you can measure how your recogniser handles it. Our write-up of FineEnvs Multilingual's 12M open RL tasks for OCR and speech in 102 languages shows the scale of multilingual data work that this kind of filter could feed.
• On-device and small models. The encoder used is the small variant. For teams shipping on-device speech, such as the Tether TranscribePsy 250M on-device recognition model, a light detector is more realistic than a second large model.
• Training mixtures matter. Apple's earlier recent paper, covered in Mix, Don't Tune: bilingual pretraining beats hyperparameter search, makes a related point on the text side: what language mixture you train on can matter more than tuning knobs.

What people are asking

Is this a new speech model? No. It is a method applied to an existing encoder. Whisper-small is used as a frozen feature extractor, and the contribution is how to read its hidden states. That is why the approach is cheap and why it does not change what Whisper can transcribe.

Why compare against HuBERT trained for monolingual language ID? Because that is the natural off-the-shelf alternative: a classifier that says which single language a clip is in. Such a model is not built to notice two languages in one clip, so it makes a sensible, if easy, baseline. The CNN baseline shows what a small supervised network can do, and the encoder-based rule edges past both.

What does "language-selective" mean in practice? Neural network hidden states are vectors with hundreds of dimensions. Some dimensions respond far more to one language than another. Apple finds them using monolingual recordings, which are plentiful, instead of rare code-switched ones. The hard-to-get resource is therefore only needed in small quantity, for tuning.

Does a higher F1 mean the data will be good? Not automatically. A detector tells you which clips probably contain two languages. Whether the transcripts, accents and audio quality are good enough to train on is a separate question, and any pipeline should sample and listen to a batch of flagged clips before adding them.

How does this relate to evaluation of voice assistants? If you cannot find mixed-language speech, you cannot measure how a recogniser does on it. A detector gives you a way to build a test set from existing audio, which is often the first step before improving a model. See also how small, domain-specific evaluation sets drive results in Coinbase's fine-tuned Qwen3.5 9B fraud agent.

Limits and open questions

• We only have the abstract. The Apple page provides the abstract and bibliographic details. Readers should consult the full paper for per-pair scores, the decision rule, and error analysis.
• Benchmark scope. CS-FLEURS is a benchmark setting; real-world conversational code-switching, with accents, noise and partial words, is harder. Results on 14 pairs may not extend to low-resource pairs outside it.
• Encoder dependence. The headline is for Whisper-small. Whether larger Whisper models or other encoders do better is not stated in the abstract.
• Tuning data requirement. A small code-switched set is needed to tune the rule, so there is still a labelling step, just a very small one.
• No stated release. We did not find code or models on the page.

How you could try the idea

This is our sketch of how a team might reproduce the approach, not Apple's recipe.

1. Collect monolingual clips for each target language.
2. Run them through a frozen speech encoder (for example Whisper-small's encoder).
3. For each language, rank hidden dimensions by how selectively they fire.
4. For a new clip, compute per-frame activations of the top dimensions per language.
5. Flag the clip if both languages' activations exceed thresholds tuned on a small labelled set.

Start with one language pair you know well, hold out a test set, and report precision as well as F1, since a curation filter lives or dies on false positives.

Why this matters

Speech assistants are judged on how they handle real speakers, and real speakers mix languages. A cheap detector does not fix recognition itself, but it makes the unglamorous part of the work, finding and balancing data, easier. It also adds to a trend we keep seeing: the best small tool is often a probe on a model you already have. For Apple, whose products serve hundreds of millions of multilingual users, that is a sensible research direction, and its tri-modal masked diffusion paper shows the same lab pursuing speech and audio from other angles.

Related reading

• Mix, Don't Tune: bilingual pretraining beats hyperparameter search
• Apple tri-modal masked diffusion
• FineEnvs Multilingual: 12M open RL tasks
• Tether TranscribePsy 250M on-device speech recognition
• Gemini 4 Argon safety report and activation probes
• VoxCPM2 multilingual voice cloning
• Official: Apple Machine Learning Research page

Details reflect Apple's published abstract as of October 14, 2026.