Can LLMs Grade Machine Translation? Apple Finds Humans and Models Are Both Challenged

Using a language model to grade translations is tempting: it is cheap, fast and tireless. An Apple Machine Learning Research paper presented at EMNLP, Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged, tests how far that temptation can be trusted. The short version from the Apple page: agreement between LLMs and humans is unreliable in most settings, sometimes beats human-human agreement, and humans themselves struggle on fine-grained annotation and low-resource languages.

The authors are Hala Almaghout, Christian Federmann and Qin Gao, and the full text is on arXiv. If you use LLMs as judges anywhere in your pipeline, the lessons carry well beyond translation. We covered a related judge problem in our post on the AI referee and leaderboard paper, and the general toolkit in AI evals for engineers and PMs.

TL;DR: what builders ask first

Why translation quality annotation matters

Machine translation systems are tuned and compared using human quality judgments. Those judgments are expensive, slow and, as this paper shows, noisy. If an LLM could stand in, evaluation would scale to many more language pairs. The authors argue that before anyone relies on that, LLM annotation needs thorough testing across language pairs, domains and annotation granularity.

The two annotation schemes

MQM (Multidimensional Quality Metrics) asks annotators to mark errors with fine-grained categories and severities; a score is computed from them. ESA (Error Span Annotation) asks for severity-marked error spans and a directly entered score. MQM is more detailed and slower; ESA is lighter.

How the study measured agreement

Agreement on scores and error counts uses Krippendorff's alpha with interval distance, with 0.667 as the usual threshold for tentative agreement. LLM-human agreement is computed by leave-one-out substitution, replacing one human annotator with the model and comparing. For error spans, the paper reports F1 on character overlap, span overlap, category and severity-specific overlap, and no-error detection. Significance comes from a paired bootstrap over source segments with 10,000 iterations and false-discovery-rate correction.

This matters because the headline comparison is not LLM against a gold truth. It is LLM-human agreement against human-human agreement: does swapping in a model change how well annotators line up?

What happened with MQM

A grid of rows each with a small check, standing in for MQM error-by-error annotation of machine translation quality

On WMT23 English to German and Chinese to English, human-human agreement was significantly better than LLM-human agreement, with one exception: Gemini on Chinese to English matched it. Every value was well below 0.667. On error counts, models did comparably on major errors but fell clearly short on minor errors, especially for Chinese to English.

On the 70-pair long-context web set, the picture was mixed:

• English to other languages: Gemini and GPT-OSS were not significantly different from human-human agreement; DeepSeek and Qwen lagged.
• Other languages to English: humans agreed significantly better for most models.
• Non-English to non-English: LLMs often beat human-human agreement, but the authors note this mostly reflects unusually low human agreement in that direction.
• Human agreement itself reached 0.667 only for a few pairs, such as Russian to English and Chinese (simplified and traditional) to English.

For error spans on WMT23, human-human span overlap peaked at 0.51. Models struggled with rare categories, with some scoring zero on a category such as Currency Format, as well as with minor errors and recognizing error-free segments.

What happened with ESA

On WMT25 across 14 language pairs and 5 domains, nearly every pair and domain fell below 0.667. Human-human agreement significantly exceeded LLM-human agreement for most pairs. Counting pairs where the model beat humans versus where humans beat the model:

Span agreement told the same story: human span overlap was about 0.45 versus roughly 0.30 to 0.31 for the models, and DeepSeek's no-error agreement was 0.05.

The Egyptian Arabic failure

The most vivid example is English to Egyptian Arabic. LLM-human alpha was negative, while human-human alpha exceeded 0.90. The reason: about 60 percent of translations came out in Modern Standard Arabic rather than Egyptian Arabic. Human raters scored those as zero because they were the wrong variant. The LLMs rated them highly, apparently not penalizing a wrong language variant. The authors suggest this may reflect how well each variety is represented in training data.

This is a clean demonstration of a general judge failure: a grader can be fluent and confident and still blind to a requirement it was never trained to check.

Why "LLMs beat humans" is a misleading headline

Several results show LLMs equalling or exceeding human-human agreement. The paper warns against reading that as model strength. Where humans disagree with each other, the bar is low. Humans struggle with fine-grained MQM and with low-resource language pairs, so a model that agrees with the average human may simply be matching noise. Agreement above human-human is not agreement with truth.

The authors even turn this around: LLMs can be used as diagnostics to flag inconsistent human annotation rounds, which is a practical use that does not require the model to be right.

How this fits the wider LLM-as-judge debate

Judge models have become the default way to score open-ended output, from chat quality to code review. The usual defence is that a strong model agrees with human raters about as often as humans agree with each other. This paper is a careful test of exactly that defence in one domain, and the answer is that it holds only in pockets. It breaks where the task needs fine-grained distinctions (minor errors versus none), where the signal is rare (uncommon error categories), and where the model has weak coverage of the language or variant involved.

There is also a quieter lesson about the reference point. When human-human agreement is itself below the conventional reliability threshold, a high ranking for the model says little. A team that reports "our judge matches annotators" should also report how well the annotators match each other, and whether that number clears a bar anyone would accept. Without both, the claim is uninterpretable.

For translation vendors and localization teams, the practical upshot is cautious. A model-graded score can be a cheap first pass to rank systems on high-resource pairs and major errors. It should not be the sole basis for shipping decisions on low-resource languages, regional variants, or quality tiers where minor errors matter, such as legal or medical text.

What the authors recommend

• Use LLMs only after thorough, per-language-pair and per-domain agreement testing with quality controls.
• Build human-LLM collaborative pipelines, for example LLMs pre-filtering or flagging annotations.
• Improve LLM performance on minor errors, rare categories and span localization through prompt engineering or fine-tuning.
• Improve human reliability with clearer guidelines, better annotator training and processes that reduce fatigue.

Limits of the study

The authors list several caveats. The 0.667 alpha threshold may be unrealistic for fine-grained MQM. The long-context web set is internal and cannot be released for licensing reasons, which limits replication. Only four LLMs and a single zero-shot prompt were tested, with a heuristic step to re-align spans. WMT23 may have appeared in model training data, which could inflate LLM agreement. And the significance tests check for difference, not equivalence, so "not significantly different" is not the same as "interchangeable."

What this means if you use LLM judges

Even outside translation, three habits follow from this paper. First, measure judge agreement against human-human agreement, not just against one human. Second, break results down by language, domain and error type, because averages hide the Egyptian Arabic style failures. Third, treat the judge as a triage tool that routes disagreements to people.

If you build multilingual products, remember that evaluation quality is uneven across languages. A model that looks fine on English to German may be badly miscalibrated for a regional variety. Apple's group has published on several other fronts recently, such as Normalizing Trajectory Models for four-step image generation, but this paper is a reminder that evaluation, not generation, is often the weaker link. Related practical judge pitfalls appear in our guide on building evals with Claude Code.

Sources: the Apple Machine Learning Research page and the arXiv paper.

Related reading

• AI referee paper: leaderboard LLM judge
• AI evals for engineers and PMs, explained
• Build evals and hill-climb with Claude Code
• Apple Normalizing Trajectory Models
• Apple SelfCompact agents
• Gemma 4 26B A4B on Mac with MLX

Findings reflect the arXiv paper and Apple page as of October 12, 2026.