Last Translation Benchmark

Vilém Zouhar, Niyati Bafna, Mukund Choudhary, Maike Züfle, Sara Rajaee, Pinzhen Chen, Jannis Vamvas, Sara Papi, Ona de Gibert, Bhavitvya Malik, Eliya Habba, Orfeas Menis Mastromichalakis, Patrícia Schmidtová, Michelle Wastl, Sheriff Issaka, Leshem Choshen, Stella Biderman, Antonis Anastasopoulos, Jan Niehues, Rico Sennrich, Mrinmaya Sachan, Ondřej Bojar, Kenton Murray, Jörg Tiedemann, Alham Fikri Aji, Philipp Koehn, Christof Monz, Alexandra Birch, Sowmya Vajjala, Chalamalasetti Kranti, Cristina España-Bonet, Nobin Sarwar, David Kaczér, Shunta Asano, Malik Marmonier, Daban Q. Jaff, Vaisakhi Mishra, Hend Al- Khalifa, Gabriele Sarti, Sourajit Saha, Nils Rehlinger, Juan Daniel Cuervo Villa, Jonathan Tonglet, Saugata Purkayastha, Dominik Macháček, Jagannathan Ramanujam, Heejin Do, Zuzana Nadova, Fred Philippy, Fabian Retkowski, Maria Lymperaiou, Silvia Casola, Hanna Yukhymenko, Shubhashis Roy Dipta, Sangwon Ryu, Andrés Jerez, Ron Keinan, Shuaib Shuaib Yusuf, Avantica Vempati, Maria Carmen Staiano

A crowdsourced, peer-reviewed benchmark of difficult translation examples paired with objective verification rules.

How can we build a translation benchmark that actually challenges modern LLMs and provides interpretable failure analysis?

Standard machine translation benchmarks are saturated, and current evaluation metrics—whether automatic overlap scores or LLM-as-a-judge—are unreliable, opaque, and prone to self-bias. The Last Translation Benchmark (LTB) replaces generic scoring with a collection of human-authored, difficult-to-translate examples, each paired with handcrafted verification rules that define specific, objective success criteria. On this benchmark, even state-of-the-art models struggle significantly, with top-performing models achieving pass rates well below 50% on the official verifier metric.

Paper Primer

The benchmark is built inductively through crowdsourcing: contributors submit challenging inputs, identify failure modes in model outputs, and write verification rules that an LLM verifier uses to check future translations. This approach creates an asymmetric evaluation where the verifier is explicitly told what to look for, providing a reproducible and interpretable pass/fail scale that distinguishes between strong models better than generic metrics.

The LTB provides a more objective and stable evaluation than existing LLM-as-a-judge methods.

The benchmark shows significantly lower self-bias (e.g., Gemini 3.1 Pro at 6.9% vs. 20.2% in typical judge setups) and higher ranking stability across different verifier models. Models that score poorly on the verifier pass rate often receive "good" quality ratings from generic LLM judges, highlighting the benchmark's ability to expose failures that standard metrics miss.

Why not just use human evaluation as the gold standard?

Human evaluation is expensive, subjective, and difficult to scale or reproduce across different campaigns. The LTB's verification rules provide a cost-effective, objective proxy that allows for consistent, automated benchmarking over time.

Does knowing the verification rules allow models to "cheat" the benchmark?

Providing the rules to a model during translation (the "oracle" mode) does improve performance, but it does not solve the underlying difficulty. The benchmark is designed to be used in "blind" mode for comparing realistic model quality, while the "oracle" mode serves as a diagnostic tool to see if a model is capable of satisfying a requirement once it is explicitly identified.

The LTB shifts translation evaluation from opaque, holistic scoring to targeted, rule-based verification, providing a long-term goalpost for tracking genuine progress in machine translation.

The Saturation of Translation Benchmarks

We expose why existing translation benchmarks miss critical errors and propose a harder, rule‑driven benchmark.

As machine‑translation models improve, standard static benchmarks have reached near‑saturation, masking remaining errors. Moreover, automatic metrics are misaligned with human judgments and vulnerable to reward‑hacking, while human evaluation suffers from reproducibility and scalability issues. Together, these gaps prevent reliable tracking of progress.

We need benchmarks that expose the lingering failure modes of strong models, otherwise we cannot tell which model is truly better.

Google Translate outputs “Ty dvě nové sestry sdílejí světový rekord mužů na 100 m,” using the feminine noun “sestry” and violating the rule.

Gemini 3.1 Pro produces “Dvě nové zdravotní sestry sdílejí světový rekord mužů na 100 m,” also violating the rule.

GPT‑5.6 Sol yields “Ty dvě nové zdravotní sestry společně drží mužský světový rekord v běhu na 100 m,” again using a feminine form.

Human translation respects the rule: “Dva noví zdravotníci společně drží světový rekord v běhu mužů na 100 m.”

State‑of‑the‑art models mis‑handle gender agreement, revealing a concrete failure that standard benchmarks never expose.

Current benchmarks are too easy to separate top‑performing models; only a rule‑driven, challenging dataset can reveal true differences.

Prior Limitations in Translation Evaluation

We review prior translation benchmarks, metrics, and human evaluation challenges.

Translation benchmarks have long relied on static test sets sampled from natural text distributions. Recent LLM‑driven systems expose the limits of these benchmarks, prompting researchers to craft harder examples that often look unnatural.

**Example 1:** All state-of-the-art translation models fail on a simple English→Czech translation (LTBv1#4652) where the term for “nurse” is gendered, but the translations select the incorrect female gender despite the context clarifying that they are men. Human translation is correct.

Evaluation has traditionally used overlap‑based metrics such as BLEU and ChrF, which compare a system output to a reference at the surface level. Neural metrics like Comet and the emerging LLM‑as‑a‑judge aim to predict human judgments more faithfully, but they inherit interpretability and robustness issues.

Human evaluation avoids metric‑driven reward hacking but is costly, noisy, and hard to calibrate across studies, making reproducibility a major obstacle for large‑scale translation research.

Principles of Difficult Translation

Identifying why current benchmarks miss translation difficulty and introducing verification rules to expose model weaknesses.

Standard translation benchmarks measure average quality but fail to pinpoint why models still stumble on hard inputs; existing error taxonomies describe what went wrong but not what makes an input intrinsically difficult.

**Example 2.** A typical example for translation evaluation from WMT26 [17]. Not only is the source example easy, with marked errors being mostly subjective preferences, but also the evaluation is opaque: what does it mean for a translation to be 90% or 86% correct?

LTB pairs each challenging example with a set of verification rules; a translation passes only if it satisfies every rule, yielding a calibrated pass‑rate that directly reflects the model’s ability to meet the intended constraints.

The model outputs “Sestra předala zprávu doktorovi” (correct feminine “sestra”).

A competing model outputs “Nurse” translated as “ošetřovatel” (masculine), violating the gender rule.

The verifier checks the gender of the noun; the first translation passes, the second fails.

The verification rule catches a subtle gender‑agreement error that generic metrics would miss, exposing a concrete weakness in the model’s handling of gendered nouns.

The LTB Crowdsourcing Pipeline

We describe how the Last Translation Benchmark is built via crowdsourced, multimodal submissions and verification rules.

The benchmark creation pipeline turns raw crowd contributions into a rigorously vetted test set, ensuring each example is both hard for models and solvable by humans.

LTB is a continuously growing collection of translation examples that are deliberately chosen to be difficult for current models while remaining tractable for expert humans.

How does LTB differ from existing translation benchmarks?

Existing benchmarks consist mostly of naturally occurring sentences that current models already handle well; LTB deliberately curates examples that break a majority of models, using verification rules to guarantee difficulty while still being human‑translatable.

Verification rules are explicit linguistic constraints that a translation must satisfy; they serve as binary checks to flag model failures.

Why not just use a loss function instead of explicit verification rules?

Loss functions provide a smooth gradient but do not give a clear binary signal about specific linguistic violations; verification rules give deterministic pass/fail feedback that isolates the exact phenomenon a model is missing.

Select source and target language, optionally specifying dialects or scripts.

Provide the example input (text, audio, image, or video) and a correct reference translation.

Platform translates the input with a set of up to ten models.

Inspect model outputs, identify common failure modes, and write verification rules.

Run an LLM to check each model translation against every verification rule.

Ensure that at least eight of ten model translations fail a rule while the human reference passes.

Submit the example for reviewer approval; the reviewer verifies fairness and significance of failures.

Upon approval, add the example to the rolling benchmark release.

Platform runs ten models; eight output omit “No parking” or reorder words.

LLM evaluates each output against the rule; eight fail, two pass.

Human reference passes the rule, satisfying the difficulty criterion.

Reviewer checks that a native speaker could produce the same translation and approves the example.

This concrete trial shows how a single lexical rule can turn an otherwise easy sentence into a benchmark‑worthy challenge.

**Figure 1.** Screenshot of the contributor interface. The contributor writes input 1 and reference translation 2, and obtains the automatic translations 3. The contributor then writes verification rules 4. This contribution can be submitted if it is: translatable (the reference translation passes the rules 5), and difficult (most automatic translations fail at least one rule 6).

Crowdsourcing provides a scalable way to capture edge‑case translation difficulty that static benchmarks miss.

Dataset Composition and Difficulty

LTBv1 expands the benchmark with thousands of new, diverse translation examples.

LTBv1 contains 3456 accepted examples spanning 109 language pairs, markedly expanding coverage.

Table 1 lists 3456 examples across 109 main languages.

Each example is paired with about 1.9 verification rules, and 10 % include explicit translation instructions. Languages are split by resourcedness (Ultra‑High 30.5 %, High 28.3 %, Medium 19.0 %, Low 3.2 %, Minimal 15.2 %, Zero 3.8 %) and by family (Indo‑European 58.9 %, Afro‑Asiatic 8.5 %, Dravidian 8.4 %, Sino‑Tibetan 7.5 %, Other 16.8 %). We also release a 911‑example text‑only subset (LTBv1‑eval) selected for highest difficulty and balanced language pairs.

Headline Model Performance

Key performance gaps on LTB reveal human superiority over current LLMs.

Human translators achieve near‑perfect scores while the strongest open‑weight LLM falls short by several points.

Human verifier pass 99.9 % versus Qwen 3.7 Plus 91.3 %; human judge 99.0 % versus 81.2 % for the same model.

All models were run with the same LTB version, prompts, and verification rule set; only the model architecture and weights differ.

Impact of Verification Rules

Verification Rules evaluation shows how models perform under LTB constraints.

Verification pass rates reach up to 84 % on the hardest LTB examples.

Table 2 lists a maximum verification pass of 84.0 % across all evaluated models.

Human-in-the-Loop Evaluation

We probe whether LLMs lack knowledge of test phenomena or cannot satisfy verification rules.

We test whether state‑of‑the‑art LLMs fail because they do not know the targeted phenomena or because they cannot satisfy the verification rules that define those phenomena.

Think of the LLM as a referee that watches a translation play and checks whether each move obeys the rulebook supplied in the prompt.

How does an LLM‑as‑a‑judge differ from a human evaluator?

Unlike a human who can draw on world knowledge and intuition, the LLM judge follows the explicit textual rules given in the prompt; it cannot infer unstated constraints or cultural nuances unless they are encoded in the rule text.

Select 22 bilingual annotators fluent in both source and target languages.

Sample 317 translation examples covering 19 language pairs from the collected dataset.

For each example, present multiple candidate translations side‑by‑side (cESA protocol).

Annotators first mark error spans as minor or major.

Annotators then assign an overall quality rating to each translation.

Aggregate the span annotations and quality scores to compute human performance metrics.

**Table 3.** Averaged translation performance (same as Main Table 2) when LLMs-as-translators are shown the verification rules in their prompts. “Synthetic” rules are generated by the LLMs before translating.

Overall, the human study reveals that LLMs benefit from explicit verification rules but still fall short of human translators, whose judgments are informed by the same rules they authored.

Stability of Evaluation Metrics

Verifier pass rate remains stable across models and aligns better with human judgments.

Verifier pass rate rankings are highly stable across model choice.

Table 4 shows a Kendall $\tau_b$ of 86.9 for the VERIFIER approach, the highest among all methods.

**Figure 2.** Human evaluation of translation quality (cESA annotation protocol) consists of two stages: first without, and then with the verification rules shown at the top.

Human annotators first evaluated translations without seeing the verification rules, then repeated the task with the rules displayed. The gap between human scores and the second‑best model narrowed in the first pass but widened after the rules were introduced, suggesting that annotators prioritize different phenomena when explicit constraints are present.

Evaluator Self-Bias

We expose how LLM judges favor their own translations and how verification reduces this bias.

LLM‑as‑a‑judge evaluators display pronounced self‑bias, while the verification LLM’s bias is near zero.

Table 5 reports self‑bias percentages; the JUDGE column reaches 28.2 % for Gemma 4 versus VERIFIER values of 8.9 %.

Self‑bias is the tendency of an LLM used as a judge to rank its own translation outputs more favorably than those of other models.

How does self‑bias differ from a model’s overall performance bias?

Self‑bias measures only the preference for a model’s own outputs when it acts as a judge, whereas overall performance bias reflects systematic errors across all evaluated translations regardless of source.

**Table 5.** Model self-bias computed as the difference between self-ranking and ranking according to other LLMs. Positive values are self-preference, and negative ones are self-dispreference.

Taxonomy of Translation Difficulty

We define a taxonomy of translation difficulty based on LTB examples.

To understand why certain inputs are hard for translation models we built a difficulty taxonomy from the Last Translation Benchmark.

The taxonomy groups source‑side phenomena that make translation hard, each group reflecting a distinct skill a model must master.

Challenges that require fine‑grained linguistic knowledge of the source language.

Phenomena where meaning cannot be derived by composing word‑level semantics.

Issues arising from mismatches between source and target language conventions.

Rare or structurally odd inputs that confuse models.

Challenges that depend on world knowledge beyond pure language.

Qualitative Analysis of Difficult Examples

Models consistently miss the linguistic challenges posed by the Last Translation Benchmark.

Across twelve diverse linguistic phenomena, current models fail to meet the verification rules in the majority of cases.

Failures are observed in Examples 3‑12, covering phonology, tone, polysemy, sarcasm, false friends, code‑mixing, and poetic nuance.

The verification rules and LTB input format were held constant across all examples, isolating the linguistic challenge as the sole variable.

Linguistic and Atypical Challenges

Atypical linguistic examples expose systematic translation failures despite verification rules.

Atypical linguistic examples lead to universal translation failures across evaluated models.

In Examples 13–11, every model output diverges from the human reference, mis‑handling vowel constraints, garden‑path structures, atypical parts of speech, collocations, and culturally‑specific slang.

The verification rules were held constant across all examples, ensuring that failures stem from linguistic atypicality rather than rule variability.

This image is a screenshot of a social media post featuring a photograph of a small, round elephant standing on a dirt path in a wooded area.

Extralinguistic and Cultural Challenges

Models consistently break the extralinguistic verification rules in the LTB suite.

All evaluated LLMs violate at least one extralinguistic verification rule across the five LTB test cases.

Failures are observed in Example 12 (orthographic mismatch), Example 13 (vowel‑removal), Example 14 (cultural address), Example 15 (idiom translation), and Example 16 (idiom misinterpretation).

**Figure.** A visual prompt and evaluation example showing three individuals performing sign language gestures on a stage, with associated instructions and model responses for translation.

Failure Modes and Ethics

Ethical considerations and acknowledgment of benchmark limitations.

We first revisit the “Blockers” taxonomy, which groups failure modes that obscure the true difficulty of source sentences.

Irrelevant outputs (e.g., GPT‑5.4 Mini) overlap with hallucinations, refusals (Claude Haiku 4.5) arise from toxic‑content filters, incomplete outputs stem from token limits, instruction‑ignoring cases follow prompt injections, and wrong tokenization (LTBv1#517) yields entirely different parses.

**Table 7.** Manual review of transferred examples.

The Last Translation Benchmark is deliberately a stress test; verification rules and taxonomy tags are intentionally incomplete, leaving room for future extensions as models improve.

Ethically, all personally identifiable information is removed, problematic submissions are filtered, the dataset is released under CC BY 4.0, and contributors must cite sources and verify license compatibility.

Supplementary Data and Examples

Guidelines for verification rules, FAQs, and reviewer instructions.

Verification rules are the criteria an AI judge uses to pass or fail a translation; they should target the observed failure modes rather than be overly specific to a single model output.

Write each rule as a short, unambiguous instruction (e.g., “Check whether the translation conveys that the speaker feels no inner peace.”) that yields a clear pass/fail outcome.

At most two automatic translations may satisfy your entire rule set, and each rule should test a single conceptual issue to keep granularity high.

Beyond text, you may supply image, audio, video, or style instructions as additional context; the translation system will treat that context as the source to be rendered in the target language.

FAQ: All languages and scripts are accepted—write “Neapolitan” or “Serbian (Cyrillic)” as appropriate.

FAQ: You become a co‑author after ten accepted submissions.

FAQ: Each translation or verification click consumes one credit; good submissions may earn additional credits.

FAQ: Attribute any reused material with a comment (e.g., “Attribute: https://en.wikipedia.org/wiki/`Werner_Herzog`”).

FAQ: To craft difficult inputs, browse social media or advertisements for naturally ambiguous phrasing.

FAQ: Generative AI may inspire ideas, but its raw outputs are not accepted as final translations.

FAQ: “Last Translation Benchmark” is hyperbolic, reflecting the saturation of existing MT benchmarks.

Reviewer instructions: Approve examples that are genuine, exhibit a substantive mistranslation, and have a correct verification rule.

Return examples that are too easy, lack a meaningful error, or have rules that do not generalize.

Comment when returning to explain the deficiency and guide the contributor toward improvement.

Prompt 1 drives the translation step; it includes optional italicized sections and may specify a context type (image, audio, video).

Prompt 2 asks an LLM to score a translation on a 0‑100 scale using a detailed quality rubric.

Prompt 3 provides cESA‑style error‑span annotation guidelines for human annotators.

Prompt 4 requests a JSON list of concise verification rules, each a single evaluable sentence.

Prompt 5 describes how to transfer LTB examples across target languages, noting that verification rules must remain applicable.

Expanding LTB across languages shows that source‑side difficulty (idioms, wordplay) transfers well, while target‑side phenomena (gender) often do not.

Example: Hindi source “Haan toh phir koi doosra vaala book kar lete hai…unnees bees ka phark hai, itna sir phodke phayda nahi.” → English human translation “Okay then let’s just book another one… it’s such a minor difference, no point stressing about it so much.”

Verification rule for the example: the phrase “unees bees ka phark” must be rendered as “the difference is negligible” rather than a literal numeric translation.

Table 7 reports that 48 % of transferred examples survive automatic filters, with most rejections caused by bad verification rules.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers