ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation

Jiahao Zhao, Xiaomin Yu, Zhongxiang Sun, Fengwei Teng, Chengwei Qin, Xiaobin Hu, Jun Xu, Shuicheng Yan

ToolArtist unifies reasoning, external search, and native image generation into a single, autonomous multimodal policy.

How can we enable text-to-image models to generate images requiring complex external knowledge and multi-step reasoning?

Standard image generators struggle with open-world tasks because they lack the ability to autonomously acquire missing knowledge or refine their output through multi-step reasoning. ToolArtist solves this by post-training a Unified Multimodal Model (UMM) to treat reasoning, tool invocation, and image generation as a single, interleaved decision process. On the WISE benchmark, this fully agentic approach achieves an overall score of 0.79, outperforming existing systems that rely on fixed pipelines or partial agent control.

Paper Primer

The core mechanism is a two-stage post-training strategy that transforms a UMM into an autonomous agent. During Supervised Fine-Tuning (SFT), the model learns to interleave search and generation by converting teacher-agent trajectories into a unified token format where the model itself performs the drawing. During Reinforcement Learning (RL), the model uses Reason-Act-Draw GRPO (RAD-GRPO) to optimize the entire trajectory using dual rewards: an intent reward for caption accuracy and a quality reward for visual faithfulness.

ToolArtist achieves superior performance on knowledge-intensive image generation benchmarks compared to non-proprietary baselines.

On the WorldGenBench-Humanities benchmark, ToolArtist reached an average Knowledge Checklist Score (KCS) of 22.10, significantly higher than the 15.58 achieved by the Unify-Agent baseline. A ~42% improvement in KCS over the strongest comparable open-source agentic baseline.

Source-aware summaries are critical for factual grounding in image retrieval.

Ablating the LLM-generated summaries for retrieved images caused the WISE benchmark score to drop from 0.79 to 0.61, with the Biology category suffering a 0.50 point decline. A 23% relative decrease in overall benchmark performance when source context is removed.

Why is this approach considered "fully agentic" compared to previous retrieval-augmented generation methods?

Previous methods typically delegate image synthesis to a separate, external generator after a search agent finishes its work. ToolArtist integrates image generation as a native action within the same UMM policy, allowing the model to inspect its own generated images and perform iterative refinements or additional searches as needed.

What is the role of the RAD-GRPO objective in the training process?

RAD-GRPO provides outcome-level feedback that propagates across the entire trajectory. By combining intent and quality rewards, it optimizes the model's ability to translate user requests and retrieved evidence into accurate visual intents and high-quality images, overcoming the limitations of static SFT data.

Researchers can now treat image generation as a flexible, multi-step agentic task rather than a single-shot prompt-to-pixel operation. This framework provides a blueprint for using UMMs to unify reasoning and synthesis in open-world environments.

Introduction to ToolArtist

We expose why static Text-to-Image fails open-world tasks and introduce the fully agentic ToolArtist framework.

Text-to-Image (T2I) models generate visually impressive pictures, yet they falter on open‑world requests that require deep semantic understanding, multi‑step reasoning, and up‑to‑date world knowledge. Benchmarks such as WISE and WorldGenBench expose systematic factual errors and missing details in these scenarios.

Recent work adds agentic components, but either hard‑codes a fixed pipeline (prompt → search → synthesize) or hands off generation to a separate model after tool use. In both cases the image‑generation step is not a decision the agent can take, so the policy never learns the full reasoning‑to‑output loop.

ToolArtist treats reasoning, external‑tool invocation, and image synthesis as interchangeable actions of a single Unified Multimodal Model (UMM) policy.

We acquire this capability via a two‑stage post‑training pipeline. In supervised fine‑tuning, a teacher agent equipped with text‑search, image‑search, and a concealed image‑generation tool produces 7,132 high‑quality trajectories; the generated images are stored as visual‑caption spans so the UMM can reproduce them itself. Reinforcement learning then refines the policy with Reason‑Act‑Draw GRPO (RAD‑GRPO), which combines an intent reward (faithfulness to the request and evidence) and a quality reward (visual fidelity).

**Figure 1** Demonstrations of open-world image generation, covering cultural knowledge, specific IPs, historical knowledge, geographic knowledge and more. All showcased images are generated by ToolArtist.

The shift from static Text-to-Image to an agentic, tool‑using generation paradigm unlocks reliable open‑world image creation.

Open-World Image Generation

Defines open‑world generation and the iterative tool‑calling loop that powers ToolArtist.

Closed‑world T2I assumes the generator already knows everything needed to render an image, so it fails whenever the target depends on facts or visuals that reside outside the model’s parameters.

When a prompt asks for something the model has never seen, the system must first go out and fetch the missing knowledge before it can draw the picture.

The model issues a TextSearch query “who is the mayor of Smalltown?”.

The search tool returns the name “Alex Rivera” and a short biography.

Using the retrieved name, the model constructs a visual‑caption “portrait of Alex Rivera” and generates an image.

This tiny loop shows why external retrieval is essential: the model’s internal knowledge alone cannot produce the correct subject.

ToolArtist treats image creation as a dialogue: it reasons, reaches for tools, draws, and can go back to reason again until the picture satisfies the request.

Round 1: $r_1$ = “need reference image of a cliff”; $a_1$ = TextSearch(“cliff photo”).

$o_1$ returns a grayscale cliff photograph.

Round 2: $r_2$ = “combine cliff with castle”; $a_2$ = draw with caption “medieval castle perched on the cliff”.

$v_2$ yields an image that blends the retrieved cliff background with a generated castle.

The model can stop after the first draw if the result is satisfactory, or continue searching for more details (e.g., a specific architectural style) before finalizing.

How does this loop differ from a simple “prompt‑optimization” agent that only rewrites the text before calling an external generator?

Prompt‑optimization agents never see the intermediate images; they treat generation as a black‑box call. ToolArtist’s loop keeps the generated image inside the history, allowing the policy to inspect, critique, and refine it with additional searches before deciding to stop.

**Figure 2.** Comparison of Agentic Image Generation Paradigms. Top: The UMM-based Workflow paradigm introduces tool calling to the UMM, and the model also completes the final image generation. However, the workflow is predefined in advance. Middle: The Prompt Optimization Agent employs a search agent to rewrite user instructions, and finally feeds the optimized prompt to an external generator. Bottom: Our ToolArtist. Both tool calling and image generation are fully determined by the model itself.

Training the Tool-Using Model

We train the model in two stages: supervised data synthesis then reward‑guided policy refinement.

Open‑world image generation demands grounding in external knowledge, which vanilla T2I models lack. Our training pipeline injects that grounding by first synthesising tool‑augmented trajectories and then refining the whole policy with reward feedback.

**Figure 3.** Post-training data and policy trajectory. The top part shows how text search and image search return evidence. The bottom part shows how raw trajectories are rewritten in the convert stage into native multimodal generation trajectories for SFT.

We turn noisy tool‑augmented rollouts into clean token‑level supervision so the unified multimodal model can learn the full reasoning‑→‑search‑→‑generation pipeline in one pass.

Trajectory 1: tokens = [

Trajectory 2: tokens = [

Compute the cross‑entropy for each masked token using the current model $P_\theta$; sum over the four masked positions.

Average the loss over the two trajectories to obtain $\mathcal{L}_{\text{SFT}}$ for this mini‑batch.

The mask isolates exactly the policy‑generated caption and image tokens, so the model never learns to predict the instruction prefix or raw tool responses.

How does this SFT differ from ordinary supervised fine‑tuning on image‑caption pairs?

Standard fine‑tuning only sees a static caption‑image pair, whereas our SFT loss trains the model to emit the entire multimodal trajectory—including reasoning steps, tool calls, and the native image‑generation token stream—by masking out everything that is not part of the policy’s own output.

After SFT gives a solid base, we treat the whole policy as a single decision‑making agent and optimise it with a reward that judges both the generated caption and the final image.

Compute dual rewards: $\tilde{R}_1 = 0.5\cdot0.8 + 0.5\cdot0.6 = 0.7$, $\tilde{R}_2 = 0.5\cdot0.5 + 0.5\cdot0.9 = 0.7$ (they tie).

Mean $\mu = 0.7$, standard deviation $\sigma = 0$; add $\epsilon=1e-8$ to avoid division by zero, so $A_1 = A_2 = 0$.

Because $A_i=0$, the min‑clipped term reduces to $0$, leaving only the KL penalty $-\beta D_{\text{KL}}^{i,j}$ to drive a tiny regularisation step.

This illustrates that when trajectories have equal reward, RAD‑GRPO falls back to a conservative KL‑only update, preventing noisy policy swings.

The advantage is computed *across* the whole batch, so a single lucky trajectory cannot dominate the update unless it truly outperforms its peers.

Why not just apply standard PPO on the caption‑only policy instead of the full multimodal trajectory?

Standard PPO would treat the caption as the sole action sequence, ignoring the intermediate reasoning and tool‑call tokens that are essential for grounding. RAD‑GRPO’s token‑masked formulation propagates the reward through the entire trajectory, teaching the model when and how to invoke tools and when to emit native image tokens.

Stage I (rollout): the teacher agent interacts with TextSearch, ImageSearch, and an external image generator to produce raw multi‑turn trajectories.

Stage II (convert): the converter rewrites external image‑generation calls into a visual‑caption span plus native image tokens, filters out failures, and builds $D_{\text{SFT}}$.

Stage III (SFT): supervised fine‑tuning on $D_{\text{SFT}}$ using the masked cross‑entropy loss $\mathcal{L}_{\text{SFT}}$.

Stage IV (RAD‑GRPO): sample $B$ trajectories per query, compute dual + auxiliary rewards, normalise to obtain $A_i$, and optimise the token‑normalized GRPO objective $J(\theta)$ with KL regularisation.

Evaluation and Benchmarks

ToolArtist sets new records on knowledge‑intensive image benchmarks.

We evaluate ToolArtist on two knowledge‑intensive suites: WISE, which probes factual grounding across cultural, temporal, spatial, and scientific subdomains, and WorldGenBench‑Humanities, which checks continent‑level cultural fidelity via detailed checklists.

ToolArtist attains the highest non‑proprietary overall score on WISE, reaching 0.79.

Table 1 shows ToolArtist’s 0.79 overall surpassing prior agentic models (max 0.78) while remaining competitive on science categories.

Compared with frontier proprietary systems, ToolArtist trails on the “Time” and “Space” WISE sub‑metrics (0.75 vs 0.89), yet it leads on knowledge‑heavy science categories, scoring 0.81 in Physics and 0.79 in Chemistry. On WorldGenBench‑Humanities it dominates Africa, Antarctica, and Asia, while Qwen‑Image leads Europe, North America, and Oceania.

WISE asks whether a text‑to‑image model can pull in factual world knowledge instead of merely aligning words with pixels.

How does WISE differ from typical image‑generation benchmarks like MS‑COCO?

MS‑COCO measures caption‑image similarity on everyday scenes, while WISE explicitly tests factual correctness across diverse knowledge domains, requiring the model to retrieve or reason about real‑world facts.

WorldGenBench‑Humanities measures a model’s ability to render culturally and regionally accurate images by checking a detailed knowledge checklist for each prompt.

Why does WorldGenBench focus on checklist items rather than a single overall rating?

The checklist isolates distinct cultural dimensions, allowing researchers to pinpoint exactly which aspects (e.g., clothing vs. architecture) a model struggles with, which a single aggregate score would hide.

**Table 1.** Main results on WISE and WorldGenBench-Humanities. Models are grouped into frontier proprietary models, general image-generation models, unified multimodal models, and agentic image-generation models. Dashes indicate unavailable entries in the collected source table. Lightly emphasized and underlined scores denote the best and second-best non-proprietary results within each metric column; ties share the same marker.

**Figure 5.** A case on WorldGenBench-Humanities. Our model better grounds the Socotra resin-collection scene in traditional tools, clothing, storage containers, grazing goats, and coastline cues.

ToolArtist outperforms existing models on knowledge‑intensive visual tasks.

Training Dynamics and Analysis

Ablation analysis reveals how each component shapes ToolArtist’s performance.

ToolArtist tackles open‑world text‑to‑image generation by letting a unified multimodal model invoke external search tools, grounding its creations in factual evidence.

**Figure 4.** RAD-GRPO training dynamics. (a) Overall reward. (b) Entropy of actor model.

We next assess the contribution of source‑aware image‑search summaries by removing them while keeping the rest of the inference pipeline intact.

Removing source‑aware image summaries drops the average WISE score from 0.79 to 0.61.

Table 2 reports a full‑toolchain average of 0.79 versus 0.61 when summaries are omitted.

We compare ToolArtist against two baselines on a WorldGenBench‑Humanities prompt about a 1955 Socotra resin collector.

ToolArtist attains a KCS of 0.625, outperforming Unify‑Agent (0.000) and GenSearcher (0.250).

The checklist evaluation in Table 3 shows ToolArtist satisfies five of eight required visual criteria, yielding the highest composite score.

The retained SFT corpus comprises 7,132 multi‑turn search‑and‑generation trajectories, offering a rich distribution of topics and lengths.

Finally, we illustrate the full reasoning pipeline with a case study on Thomas Edison’s greatest invention.

The agent first queries textual sources to confirm that the practical incandescent light bulb is Edison’s most iconic invention, then retrieves images of both the phonograph and the bulb, crafts a caption that places Edison beside both artifacts in his laboratory, and finally generates a high‑detail historical photograph.

Qualitative Case Studies

Our RL checkpoint outperforms competing models on four WorldGenBench‑Humanities cases.

Recall that ToolArtist equips a unified multimodal model with search tools to ground open‑world text‑to‑image generation.

Unify‑Agent is a baseline multimodal model that processes inputs jointly but does not invoke external search tools.

GenSearcher is a baseline that augments generation with a pretrained image‑search module but does not integrate the search results into the generation loop.

Our RL checkpoint consistently beats both baselines, achieving up to 0.600 KCS—roughly 100 % higher than the 0.300 scores of the competitors.

Whitechapel 0.600 vs 0.300/0.300; Antofagasta 0.500 vs 0.300/0.200; Moorea 0.400 vs 0.100/0.000; Santo Domingo 0.400 vs 0.200/0.100.

**Figure 7** Case study on WorldGenBench-Humanities. Our model better grounds the Whitechapel scene in patched working-class dress, coal-smog atmosphere, and soot-darkened Victorian facades while preserving the shared street, housing, and gaslight cues.

**Figure 8** Case study on WorldGenBench-Humanities. Our model better grounds the Antofagasta port scene in period labor clothing, colonial masonry harbor buildings, a distant steam freighter, desert surroundings, and morning fog.

**Figure 9** Case study on WorldGenBench-Humanities. Our model better grounds the Moorea noni-harvesting scene in volcanic valley terrain, scattered ripe noni fruit, a local flower garland, and a distant Pacific view.

**Figure 10.** Case study on WorldGenBench-Humanities. Our model better grounds the Santo Domingo sugar-workshop scene in the wooden cane press, earthenware vats, visible steam from the sugar cooker, and Caribbean-style exterior buildings.

Implementation Details

Full system and user prompts for the teacher agent and ToolArtist, plus workflow and tool definitions.

This appendix supplies the exact prompts used to drive both the teacher‑agent data synthesis and the inference‑time ToolArtist agent, together with the surrounding interaction protocol.

The Teacher Agent Full Prompt begins with a `SYSTEM_PROMPT` that instructs the agent to search for textual facts and reference images, distill them into a concise visual description, and iteratively call the draw tool until the result is satisfactory.

The accompanying `USER_PROMPT` defines the three tools in JSON schema form, specifying required fields such as query for searches and prompt/images for drawing, and outlines the allowed multi‑round call pattern.

ToolArtist’s Full Prompt mirrors the teacher’s structure but omits the explicit “review” step, focusing on a single pre‑generation caption delimited by <|`extra_50`|>…<|`extra_51`|> and enforcing the same fixed‑token reference rule.

The reference‑image citation format is strict: only tokens like [IMAGE1], [IMAGE2] may appear in prompts; natural‑language mentions such as “Image 1” are prohibited.

The overall workflow is simple – think, call a tool, wait for the response, then either continue thinking or produce the final answer – and may repeat arbitrarily many times.

Finally, the LLM Reader Prompt at the end of the section specifies how downstream evaluation extracts core information from the document, demanding concise bullet‑point output without filler.

SFT Data Examples

Illustrates the precise ellipsoid data used in SFT rollouts, highlighting a sub‑millimeter axis difference.

The SFT rollout starts from a user question and produces a technical diagram that contrasts the reference ellipsoids of WGS 84 and GRS 80.

Record the WGS 84 semi‑minor axis value (6 356 752.3142 m).

Record the GRS 80 semi‑minor axis value (6 356 752.3141 m).

Compute the absolute difference: 6 356 752.3142 m − 6 356 752.3141 m = 0.0001 m, i.e., 0.1 mm (≈ 0.105 mm as reported).

Note that the semi‑major axis is identical for both ellipsoids (6 378 137.0 m), so only the semi‑minor axis varies.

In the diagram the tiny 0.1 mm gap is exaggerated for visual clarity, with a caption stating the illustration is not to scale.

Even a sub‑millimeter discrepancy matters for high‑precision geodesy, yet the visual exaggeration helps readers perceive the difference that would otherwise be invisible.

Additional SFT Examples

A concrete SFT example shows how the model enforces factual visual details.

This section showcases a concrete SFT data example with the characters Sigewinne and Bailu in an underwater nursery. The prompt explicitly requires a medical chart displaying Sigewinne’s Hydro Vision and a glowing blue potion.

Assemble the prompt and the five reference images into a single SFT data entry.

Feed the entry to the ToolArtist model, which generates a full‑resolution image.

Inspect the generated image to confirm that the wall‑mounted chart contains the Hydro Vision symbol.

Verify that the potion in the characters’ hands glows blue as described.

Check that both characters are rendered in the expected anime style and that the overall scene matches the underwater nursery description.

This example demonstrates that SFT data can tightly control specific visual facts while preserving the broader aesthetic of the scene.

**Table 7.** Checklist evaluation for the Santo Domingo case. Each row is one score point from the benchmark checklist. Blue rows mark the points satisfied by our model. ✓ means satisfied and ✗ means not satisfied.

This image depicts a historical scene set in a narrow, cobblestone street lined with brick buildings, reminiscent of 19th-century London. A woman dressed in period-appropriate clothing, including a long skirt and a bonnet, is walking briskly down the center of the street. The atmosphere is foggy and dim, illuminated by warm, glowing street lamps. In the background, industrial chimneys emit smoke into the overcast sky, suggesting an urban, industrial setting.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers