UltraText Bench: A Comprehensive Bilingual Benchmark for Evaluating Visual Text Rendering in Image Generation
Deyuan Liu, Yihao Hu, Jingxuan Zhang, Xingying Li, Jun Xie, Jiacheng Liu, Jungang Li, Yu Huang, Xuanyi Liu, Yue Ding, Zecheng Wang, Lei Zhao, Mingda Wang, Zhenglin Cheng, Peng Sun, Tao Lin
UltraText Bench evaluates dense, bilingual text rendering across 24 scene categories using a structured per-region reference.
How can we reliably evaluate the ability of image generators to render dense, multi-region text in both English and Chinese?
Text-to-image models struggle to maintain character accuracy and spatial placement when prompts require dense, multi-region text. The authors introduce UltraText Bench, a bilingual benchmark that pairs 432 prompts with structured ground-truth references for text content, placement, and carrier. A vision-language model judge, Q-Judger, evaluates generated images against these references across four quality dimensions. Results across 24 model configurations reveal that high visual clarity often masks poor text fidelity, with performance dropping sharply as text load increases.
Paper Primer
The benchmark treats visual text generation as a reproduction task: the model receives all target strings in the prompt and must render them in specified regions. The core move is the use of a uniform, structured reference for every prompt, which allows the judge to verify not just if text exists, but if the correct content appears on the intended carrier at the requested location.
Model performance on dense text is highly sensitive to workload and configuration.
Composite scores for models like Qwen-Image-2512 drop from 86.50 at L1 (Hard) to 42.86 at L3 (Extreme) in English. A performance decline of over 50% across difficulty levels.
Clarity and fidelity are distinct, often conflicting, performance dimensions.
Z-Image-Turbo achieves higher clarity scores than its Base variant but suffers a 14.76-point drop in text fidelity. A significant trade-off between visual sharpness and character-level accuracy.
Why is this benchmark needed if existing tools already evaluate text rendering?
Existing benchmarks often focus on short strings or single-region rendering; UltraText Bench specifically targets sustained performance across 4–12 regions, testing whether models can maintain accuracy and layout consistency under high text density.
What is the role of the Q-Judger in this evaluation?
Q-Judger acts as an automated evaluator that applies a six-property rubric to compare generated images against the structured reference, providing consistent, multidimensional scores for fidelity, clarity, spatial quality, and scene integration.
Researchers should treat text fidelity and visual clarity as separate metrics, as high-quality rendering of characters does not guarantee the preservation of the requested content.
Introduction and Motivation
We expose the gap in generating dense, multi-region text and introduce UltraText Bench to measure it.
Image generators now excel at rendering short strings, yet producing dense visual text across many regions remains a persistent challenge.
We need a benchmark that forces models to reproduce long strings in many places, because real‑world documents require accurate, legible text everywhere.
UltraText Bench fulfills this need: it offers 432 bilingual prompts spanning 24 real‑world scene categories, three difficulty levels, and 4–12 text regions per prompt, each paired with a structured reference.
The shift from short‑string rendering to dense, multi‑region document generation defines the next frontier for image generators.
Prior Benchmarks and Methods
We position UltraText Bench among prior text‑rendering benchmarks by comparing key attributes.
Related work spans three strands: visual‑text generation methods, existing text‑rendering benchmarks, and automatic evaluation techniques.
Benchmark Comparison is a side‑by‑side view of how each benchmark differs in language support, required extra inputs, reference style, and evaluation method.
**Figure 3.** Task scenes and assessment in three benchmarks. Left: CVTG-2K (Du et al., 2025) matches OCR-recognized words to targets using Word Accuracy and normalized edit distance, and separately reports CLIPScore. Center: LongText-Bench (Geng et al., 2025) uses a VLM to extract text for comparison with target strings. Right: UltraText Bench uses a structured reference (Ref.) to rate the whole image: text accuracy (TA), completeness (TC), readability (TR), position correctness (PC), layout quality (LQ), and scene integration (SI). The scenes, abbreviated strings, and line marks are illustrations, not benchmark outputs or measured results; they do not exhaust each benchmark's scene types.
This table compares various text-related benchmarks across five columns: Benchmark, Language, Extra input, Target reference, and Evaluation.
Benchmark Taxonomy and Design
Defines what varies, what stays fixed, and how performance is measured.
The benchmark construction specifies which variables are swept, which are fixed, and which metrics are reported.
The taxonomy groups text‑heavy scenes into six domains, each containing four categories that differ in carrier, density, and typical layout.
Define the six‑domain, 24‑category taxonomy (see Table 2).
Assign three language‑specific difficulty levels (L1, L2, L3) that increase text load and region count.
For each domain × level × language cell, generate three prompts, yielding 432 prompts total.
Annotate each prompt with a structured reference R that lists every region’s target text, grid position, size, type, carrier, and importance.
Run the generated image through the Q‑Judger, which produces six scores (TA, TC, TR, PC, LQ, SI) based on the reference.
**Table 2.** UltraText Bench scene taxonomy. Six domains organize 24 categories by rendering challenge and typical layout. Notes: Density describes qualitative text packing, not the GT-character load in Table 3. Layouts are typical examples, not mandatory templates.
**Figure 5.** Realized difficulty statistics for the 432 prompts. Each row contains 72 prompts. Bars show minimum–maximum ranges and dots show means from Table 3. (a) Target-string character counts, including spaces and punctuation, on a log axis. Grey brackets show the language-specific design bands; bounds are inclusive, ZH bands overlap, and L3 bands have no upper bound (shown by open ends). Levels are assigned workloads, not disjoint character bins; all records satisfy their own band. (b) Region counts; no per-level region-count bands are specified. Both panels describe benchmark inputs, not model performance.
The table provides statistics on prompts, ground truth (GT) characters, and regions per prompt across different language and difficulty splits (EN L1-L3, ZH L1-L3).
The Evaluation Protocol
Defines how the Q‑Judger scores generated images across four reporting dimensions.
This section details the automatic evaluation pipeline that underpins all benchmark results.
The Q‑Judger is a Vision‑Language Model that receives a generated image and a complete structured reference, then returns six numeric judgments about content and layout.
The six raw scores are aggregated into four higher‑level dimensions that capture fidelity, clarity, spatial quality, and scene quality.
Compute Fidelity: $(10 + 100)/2 = 55$.
Clarity equals TR: $80$.
Spatial quality: $(70 + 60)/2 = 65$.
Scene quality is SI: $50$.
Composite = \$0.60 \cdot 55 + 0.30 \cdot 80 + 0.05 \cdot 65 + 0.05 \cdot 50 = 33 + 24 + 3.25 + 2.5 = 62.75$.
This concrete walk‑through shows how a low TA score can be masked by a high TC, yielding a mid‑range Fidelity that still reflects the severe text error.
**Figure 4.** Task definition and evaluation protocol. The generator receives only prompt $p$. Q-Judger, from Qwen-Image-Bench (Li et al., 2026a), applies the UltraText rubric to the image and complete reference $R$, returning six image-level scores that map to four reporting dimensions and a composite. Reference fields and scores are not paired one-to-one. Grid cells give coarse locations; regions may share a cell. Invalid responses remain unscored and reduce coverage. The bakery schematic simplifies the four-region record `A1_L1_EN_001`, reproduced in Appendix B.1.
Human evaluation involved ten participants who inspected the same images and dimensions, but quantitative inter‑rater agreement is omitted; qualitative examples appear in Appendix F.
Experimental Results
Key results show GPT Image 2 [Low] dominates, with notable fidelity‑clarity tradeoffs across models.
GPT Image 2 [Low] dominates the UltraText Bench Composite metric with a score of 99.35.
Table 5 lists GPT Image 2 [Low] as the shaded Composite leader at 99.35.
All models were run with their standard sampling configurations; resolution, aspect ratio, and API behavior follow each model’s default settings.
**Figure 1.** Scene coverage at L3. One selected output per category in taxonomy order, from GPT Image 2 at API quality Low. L3 denotes Extreme workload; EN/ZH denote English/Chinese.
Qualitative Diagnostic Examples
Qualitative ablations illustrate judge dimension overlaps on real examples.
Each of the seven cases pairs a whole image (left) with a native‑resolution crop (right), quoting the relevant reference segment and noting the observable outcome.
Case 1 shows dense Chinese glyphs that require local inspection; the judge gives TA = 10, TC = 100, TR = 85, PC = 100, LQ = 95, SI = 95, Composite = 68.10. The rows appear tidy at page scale, but glyph strokes are only verifiable when enlarged, highlighting dimensions TA and TR.
Case 2 presents a correct surface type placed at the wrong grid location; scores are TA = 10, TC = 100, TR = 95, PC = 10, LQ = 95, SI = 95, Composite = 68.90. The hour plate appears in the bottom‑left of the full image instead of bottom‑right, exposing a failure in the PC dimension.
Case 3 contains readable chat content but the judge assigns TA = 0, TC = 0, TR = 100, PC = 0, LQ = 100, SI = 100, Composite = 37.50. Visible text in the central column contradicts low TA/TC scores, revealing a discrepancy between visible content and automated fidelity scores for TC and LQ.
Case 4 illustrates carrier integration as a separate property; scores are TA = 85, TC = 95, TR = 95, PC = 90, LQ = 95, SI = 98, Composite = 92.00. The “DAILY BREAD” title follows the sign’s plane and lighting, so SI captures integration distinct from text accuracy.
Case 5 shows maximum ratings despite visibly degraded small text; scores are TA = 100, TC = 100, TR = 100, PC = 100, LQ = 100, SI = 100, Composite = 100. The bottom‑right test block contains deformed characters, yet all dimensions receive perfect scores, exposing a local limitation of the automatic metrics.
Discussion and Failure Modes
We discuss why dense text poses unique challenges, outline current limits, and point to future directions.
Recall the core premise: image generators handle short captions well but stumble on dense, multi‑region text, which UltraText Bench quantifies using a structured reference‑based VLM judge across English and Chinese.
Dense text failures arise from three intertwined issues: (1) discrete character errors that break whole words, (2) the need to allocate attention capacity across many local patterns, and (3) the scarcity of high‑quality text‑annotated training data.
**Figure 2.** Local success can hide whole-scene failures. This illustrative six-region scene contrasts a correct shop name with errors elsewhere. *Left.* The correctly rendered shop name, cropped. *Center.* The full scene, with five further requested regions marked. Each is missing, wrong, unreadable, misplaced, or legible but poorly integrated with its intended surface. *Right.* Dense text load and whole-scene evaluation cover the requested text.
Potential training directions for stronger dense‑text rendering include visual‑representation warm‑up (e.g., ERW) and explicit text‑specific feedback loops that operate at the region level, diagnosing character errors, omissions, placement, and integration.
UltraText Bench complements existing benchmarks: STRICT evaluates thousands of characters on a plain page, OCRGenBench covers dense pages but mixes generation and OCR tasks, and InfoTextBench targets bilingual dense text but uses different region definitions, making direct numeric comparison infeasible.
Our results inherit several limitations: each model was run under its default generation settings (varying resolution, aspect ratio, and API behavior), and we report descriptive means without confidence intervals, so fine‑grained cell differences should be interpreted cautiously.
Future work will explore using the per‑region annotations as training signals, extend the benchmark to video generation and editing (e.g., Vidu S2), and investigate temporal consistency of dense text across frames.
Dataset Construction
Details of how the UltraText benchmark prompts and annotations were built.
Dataset construction followed a five‑stage pipeline. First, 160 seed prompts from X‑Omni LongText‑Bench guided scene description, target‑string placement, and region annotation. The authors extracted three design lessons: scene organization, embedding target strings naturally, and fine‑grained region division.
Second, they generated 9 prompts for each of the 24 categories in both English and Chinese, at three difficulty levels, yielding 24 × 3 × 3 × 2 = 432 records. Prompts within a category were kept distinct—no two share a business name, location, or theme.
Third, automated scripts recomputed each record’s ground‑truth character load as the sum of region text lengths and verified that every record fell inside its language‑ and level‑specific band. Scripts also ensured every region carried an identifier and all six required fields, and that each ground‑truth string appeared verbatim in its prompt.
Fourth, manual revisions integrated target strings into the surrounding scene description, replacing generic directives like “The image must contain the following text” with concrete descriptions of the surface bearing the text.
Fifth, cross‑lingual difficulty was inspected by comparing region‑text distributions; no OCR measurement acted as a release gate, so English–Chinese comparisons should be read from the reported distributions rather than an OCR‑derived score.
Example of cultural adaptation: the English menu record `C1_L1_EN_001` lists “Cappuccino $5.25" and "Extra Shot +\$0.75”, while the Chinese counterpart `C1_L1_ZH_001` shows “干炒牛河¥38” and “菠萝油配丝袜奶茶”. Both excerpts are verbatim strings from the released JSONL files.
Annotation schema (A.1) defines each region with an id and six attributes: exact text, position (one of nine grid cells), size (large, medium, small, tiny), type, carrier, and importance (high, medium, low). Position, size, and importance draw from closed vocabularies; type and carrier are free‑form labels describing the role of the text and the surface it appears on.
The released JSONL format stores one prompt and its structured ground‑truth per line. Placeholders such as `TELCODE_SAFE`, `WEBLINK_SAFE`, and `HANDLECODE_SAFE` appear in the data, and each record includes indexed telephone forms like TELCODE‑A0001.
Illustrative prompt revision: the instruction “The image must contain the following text: DAILY BREAD” becomes “A walnut storefront sign bears the cream serif lettering ‘DAILY BREAD’.” The target string remains unchanged; only the surrounding description is altered to name its surface.
Category definitions (A.2) expand the 24 benchmark categories into six domains (Signage & Labels, Documents & Print, Commercial, Digital Interfaces, Structured Data, Creative & Special), each with four sub‑categories that specify typical scene layouts and text roles.
Example Record Details
Appendix B presents the full English and Chinese sign records and their region annotations used in the benchmark.
B.1 COMPLETE ENGLISH SIGN EXAMPLE provides the full record `A1_L1_EN_001`, a Level‑1 sign containing 423 ground‑truth (GT) characters distributed across four visual regions. The generator prompt describes a bakery façade, a hand‑carved wooden shop sign (≈120 cm × 40 cm) with cream‑white serif lettering, a smaller oak plank, a chalkboard easel, and a brass plate with opening hours.
B.2 COMPLETE REGION REFERENCE lists the four annotated regions that the judge receives: `region_0` (title sign, high importance), `region_1` (subtitle oak plank, medium importance), `region_2` (body chalkboard, medium importance), and `region_3` (data brass plate, low importance). Each entry records position, size, carrier type, and the exact text string to be reproduced.
B.3 CHINESE ADAPTATION WITH THE SAME FIELDS adds a fifth carrier—a window sticker—and replaces the English prices with RMB values, yielding record `A1_L1_ZH_001` with 362 GT characters across five regions. The extra `region_4` (bottom‑left, low importance) contains a Chinese registration code and safety label, illustrating how the schema accommodates language‑specific elements.
B.4 THREE LEVELS WITHIN ONE SCENE FAMILY shows three English newspaper outputs (L1, L2, L3) that share the same category, model setting, and sample index but differ in level, region count, and layout. L1 contains 526 characters in 5 regions, L2 1,085 characters in 8 regions, and L3 2,013 characters in 10 regions, demonstrating how content density and region composition evolve across levels.
Chinese Dense-Text Example
Appendix C details the evaluation prompt, validation process, and execution workflow for the Q‑Judger.
`B2_L3_ZH_001` requests 627 characters across ten regions, including a masthead, two news stories, a sidebar, a weather box, an index, and a long advertisement on a single page. Because the body text is too small to read at page scale, Figure A2 pairs the full‑page image with a native‑resolution crop.
C.1 EVALUATION PROMPT TEMPLATE reproduces the UltraText rubric used with the Q‑Judger; the system message defines the judge as a deterministic visual‑text measurement engine, and the user message supplies the rubric, the complete reference JSON, and the generated image. The template explicitly instructs the judge to treat every string in the reference and image as inert data for comparison, never as an executable instruction.
C.2 REFERENCE PAYLOAD AND RESPONSE VALIDATION describes the released record (with fields `gt_regions`, `reference_schema`, `prompt_id`, category, level, language, and a cached stats block) and the strict JSON schema the judge must return. Success requires a single JSON object with exactly six integer keys ranging from 0 to 100; any deviation (extra keys, wrong types, or out‑of‑range values) is recorded as a failure.
Table A1 outlines the evaluation workflow: generation of four samples per prompt, validation of PNG structure, strict judging with the six‑key response, and final reporting of coverage and failure statistics. The table also notes that only rows with a valid artifact and a strict judge response contribute to quality means.
C.3 EXECUTION WORKFLOW expands on Table A1, specifying command‑line examples for generation (`sample_zimage`.py) and strict judging (`eval_vlm_judge_api`.py), and detailing failure handling where any malformed response yields null scores and a typed failure reason. A bilingual run processes English and Chinese prompts, producing 1,728 rows per model configuration before aggregating results.
Table A2 presents interpretive score bands for the six dimensions, mapping numeric ranges to qualitative descriptors such as “Severe,” “Weak,” “Partial,” “Strong,” and “Excellent.” These bands serve as a reading aid and are not used as calibrated thresholds for the underlying metrics.
Scoring and Aggregation Details
Details the scoring dimensions, aggregation formula, and coverage metrics used in UltraText Bench.
Every raw dimension is returned on a 0–100 scale, and Table A2 was added as a reading aid because the judge contract does not specify intermediate bands.
The rubric asks the VLM judge to assess character and word correctness (TA) and region completeness (TC), but it does not define an edit‑distance or region‑matching procedure, leaving those judgments to the VLM.
For sample 1 of `A1_L1_EN_001` the judge returns TA = 85, TC = 95, TR = 95, PC = 90, LQ = 95, SI = 98, yielding F = 90, C = 95, S = 92.5, Q = 98 and a composite of 92.025, which is stored as 92.0 after rounding.
A parser failure produces an evaluator row with null scores rather than a valid judge response; such rows are logged with a `failure_type` of “`judge_api_or_response_invalid`”.
Suppose four images are planned per prompt. Prompt A yields composites 80 and 100; Prompt B yields 20; the remaining five rows fail. The successful‑image mean is 66.67, the prompt‑macro mean is 55.00, image coverage is 37.50 %, and prompt coverage is 100.00 %.
Every row of Table 5 reports the same format: the first five score columns are the equal‑language mean of EN and ZH prompt‑macro aggregates, while the six level columns retain language‑specific composites; missing cells indicate unavailable measurements, not zeros.
Region Reference Utility
Appendix E details the region reference system and shows benchmark scores across models.
The structured reference links each target string to a concrete region, specifying both what content should appear and where it belongs. This enables the evaluator to detect mismatches such as a correct title placed above an incomplete menu or a price detached from its item.
**Figure A1.** L1/L2/L3 newspaper examples. All outputs use GPT Image 2 [Low], sample 1. **Notes:** The reference grows from a masthead, a headline, a lede, and a footer into several articles surrounded by smaller supporting text. Character load and region count describe the input. These thumbnails show the page layouts; checking individual characters requires enlargement.
Benchmark scores reveal how model capabilities evolve as text density and region count increase. Qwen‑Image‑2512 attains an EN Composite of 86.50 at level L1 but drops to 42.86 at level L3, whereas Z‑Image‑Base falls from 82.92 to 26.34 across the same levels.
Even models with similar overall composites differ in their trade‑offs: Qwen‑Image‑2512 scores higher on Clarity (79.89 vs 64.86) while Boogu‑Image‑0.1‑Turbo leads on Fidelity (66.27 vs 59.30). These nuances expose which aspects of dense‑text generation each model handles better.
**Figure.** A newspaper front page featuring a headline about twelve technology companies signing a joint testing agreement to standardize display interfaces and documentation, accompanied by a photograph of the signing ceremony.
Materials and Limitations
Limits of released materials and human‑alignment details.
The galleries in Section F.1 display the full taxonomy of scene types; enlarged examples let readers inspect individual text strings and their visual carriers.
All core materials—dataset, evaluator code, and metadata—are bundled in the benchmark release; generated images and per‑image judge records are distributed separately, requiring a versioned archive to link them to specific model configurations.
The released EN and ZH prompt records, together with the evaluator, support the dataset counts, region inventories, six raw metrics, strict response validation, typed failure states, and the four‑dimension composite; GPT Image 2 [Low] adds 859/864 successful EN images (215/216 prompts) and 864/864 successful ZH images (216/216 prompts), with four EN images missing.
The GPT Image 2 cases shown are qualitative examples, not a statistically representative error‑rate sample; judge summaries omit `model_revision` and `model_artifact_sha256` fields, leaving checkpoint provenance incomplete.
Full reproducibility across configurations demands the exact model revision, resolution, sampling settings, API quality flags, and evaluation dates, alongside images and raw judge responses; comparing only shared prompts can reveal how coverage differences affect model pairs, but uncertainty estimates must account for prompt overlap.
**Table A1.** Execution workflow.
```bash python src/generation/sample_zimage.py \ --model_path <MODEL> \ --prompt_file data/<lang>_prompts.jsonl \ --output_dir <run> --steps <N> \ --num_images_per_prompt 4 # Step 3 command example: strict Judge. python scripts/eval_vlm_judge_api.py \ --sample_dir <images> \ --prompt_file data/<lang>_prompts.jsonl \ --output_file <judge.jsonl> \ --model_name <served_model> ```
Human evaluation involved ten participants assessing four dimensions—content, readability, spatial organization, and scene integration—yet the paper does not report any quantitative inter‑rater agreement.
Qualitative case studies pair whole images with native‑resolution crops; for example, the hour plate appears at the bottom‑left instead of the requested bottom‑right, and lettering in another figure follows the wooden sign’s plane, exposing alignment errors that automatic scores miss.
**Table A2.** Interpretive score bands. Notes: These qualitative descriptions are a reading aid rather than calibrated thresholds on character accuracy or region recall, and they form no part of the judge prompt.
```json {"text_accuracy":85,"text_completeness":95,"text_readability":95, "position_correctness":90,"layout_quality":95,"scene_integration":98} ```