WithEveryone: Unified Planning and Identity Grounding for Group Image Generation

Hengyuan Xu, Qixun Wang, Yiji Cheng, Miles Yang, Zhao Zhong, Wei Cheng, Xingjun Ma, Yu-gang Jiang

WithEveryone uses layout-grounded identity supervision and structured planning to generate coherent group images with up to ten distinct people.

How can we generate images of large groups (5–10 people) while ensuring each person's identity is correctly bound to their specific location and reference image?

Generating group images with many specified people is difficult because models struggle to bind multiple reference identities to distinct locations, often resulting in duplicated faces or identity dilution. WithEveryone addresses this by predicting a structured layout plan before synthesis and using a Layout-Grounded ID Loss that supervises each identity within its specific annotated region. This approach achieves the highest target-context identity similarity on a five-to-ten-person benchmark while significantly reducing copy-paste artifacts compared to existing systems.

Paper Primer

The core mechanism hinges on offloading the spatial and identity-binding burden to a unified model's understanding side. The model predicts a structured Layout Chain of Thought (Layout CoT) that binds reference identities to specific bounding boxes and poses, which a deterministic renderer then converts into a visual condition for the flow-based image generator.

To ensure identity preservation, the model employs Layout-Grounded ID Loss (LG-ID Loss): it supervises generated faces by cropping them directly from the annotated layout regions, effectively removing the need for unstable embedding-based face matching during training. ID Representation Forcing further stabilizes this by requiring the model to predict a representation for each identity before image synthesis, creating an identity scaffold for the generator.

WithEveryone achieves superior identity preservation in large groups compared to proprietary and academic baselines.

On a 210-example benchmark of 5–10 person images, WithEveryone reached a target-context identity similarity (Sim(Tgt)) of 0.499. This outperforms GPT-Image 2 (0.462) while reducing copy-paste artifacts from 0.169 to 0.055.

The system effectively scales to larger groups with minimal identity collision.

WithEveryone covers 97.3% of requested identities with a duplicate rate of only 2.8%. This is the lowest duplicate rate among compared systems, indicating high distinctness even when generating ten people simultaneously.

Why does existing identity supervision fail when scaling to groups of five or more people?

Existing methods rely on embedding-based matching to assign generated faces to references; in large groups, these faces are initially indistinguishable during training, causing the loss to mispair identities and cancel out the supervision signal.

What is the primary source of identity gain in this framework?

The LG-ID Loss is the single largest source of identity gain, as it grounds supervision in the layout annotation rather than in noisy pixel-based matching, forcing the model to place the correct identity in the correct spatial location.

The Challenge of Group Generation

Scaling identity preservation to large groups requires explicit identity‑layout planning.

Identity‑preserving generation becomes unreliable when many specified people must appear, because the model must bind each reference to a distinct person and location. Existing embedding‑based identity losses break down for large groups, as early‑training faces are nearly identical and the matching becomes arbitrary. WithEveryone addresses this by injecting each identity as an addressed token, planning a structured identity‑layout chain‑of‑thought, and supervising each region with the Layout‑Grounded ID loss while Representation Forcing forces per‑identity predictions before synthesis.

**Figure 1.** **Overview of WithEveryone.** WithEveryone generates coherent group images from five to ten reference identities. A single model reasons about which references participate, plans identity–layout bindings, person regions, and poses, and generates the image conditioned on that plan, so identity and composition are decided in one context rather than by separate modules.

The WithEveryone Architecture

How the model plans identities and layouts before generating a group image.

The core difficulty is that a model must decide which reference identities appear where before any pixels are produced; without that plan, identity information gets lost in the flood of image tokens.

We run a single backbone that alternates between autoregressive text/reasoning tokens and latent image tokens, letting high‑level decisions condition the later diffusion steps.

How does this differ from a classic two‑stage pipeline where a planner and a generator are trained separately?

In the classic pipeline the planner’s output is a static conditioning vector that the generator treats as immutable. Here the planner’s tokens remain in the autoregressive stream, so the generator can attend to them at every diffusion step, preserving fine‑grained spatial cues.

Before any image is rendered, the model writes a structured “plan” that lists each selected identity together with its target coordinates, much like a human sketching a scene layout.

Why not generate the layout directly as a raster image instead of a token sequence?

Token‑level planning keeps the layout differentiable and tightly coupled to the transformer’s language modeling objective, allowing the same loss functions (next‑token prediction) to supervise both reasoning and spatial placement.

Each selected person gets a dedicated representation token whose hidden state is forced to match the target’s ArcFace embedding, ensuring the diffusion process has a concrete identity scaffold.

Is this just another way of feeding the reference embedding into the model?

No—the forcing loss explicitly penalizes any drift between the projected hidden state and the true target embedding, guaranteeing that the token carries the correct identity information rather than merely receiving it as input.

Instead of matching generated faces to references after the fact, we crop both prediction and target at the layout‑specified region, guaranteeing a one‑to‑one correspondence for every supervised identity.

Compute cosine similarity for identity 1: $\cos(\hat{e}_1, e^{\text{tgt}}_{1}) = \frac{0.7\cdot0.8 + 0.5\cdot0.6}{\|\hat{e}_1\|\|e^{\text{tgt}}_{1}\|}\approx 0.99$.

Compute cosine similarity for identity 2: $\cos(\hat{e}_2, e^{\text{tgt}}_{2}) \approx 0.98$.

Apply the loss formula: $L_{\text{ID}} = \frac{1}{2}\bigl[(1-0.99)+(1-0.98)\bigr] = 0.015$.

Even tiny misalignments are amplified by the cosine distance, so the loss pushes the predicted embeddings to match the target identities very tightly.

Why does cropping at the layout region avoid the “assignment” problem that other identity losses suffer?

Because the layout already tells us which pixel region belongs to which person, the crop operation guarantees a deterministic pairing; there is no need for a Hungarian matcher that can mis‑assign faces when many identities look similar.

The model is trained with four complementary losses, each supervising a different part of the autoregressive sequence.

**Figure 3.** Overview of WithEveryone. The model loads selected reference identities as ID tokens, predicts structured Layout CoT, renders a visual layout condition, and predicts identity representations before generating the target group image. Modality-switch tokens that mark the boundaries between autoregressive reasoning and flow-based image generation are used in the sequence but omitted from the figure.

Benchmark and Performance

We evaluate WithEveryone on a large‑group benchmark and show it dominates all baselines.

We assess identity preservation, copy‑paste artifacts, and overall generation quality across 210 real‑world group images.

A curated set of 210 group photos, each containing five to ten distinct reference faces, used to test how well models keep identities when scaling up.

GPT‑Image 2 serves as the strongest proprietary baseline; it combines a large diffusion model with a dedicated identity‑preserving finetune.

WithEveryone attains the highest target‑identity similarity (0.499) while covering 97.3 % of references with only 2.8 % duplicate rate.

Table 1 shows Sim(Tgt)=0.499, Coverage=0.973, Dup=0.028, outperforming all academic, open‑source, and proprietary competitors.

Copy‑Paste artifacts are markedly lower for WithEveryone (0.055) than for GPT‑Image 2 (0.169), and its CLIP‑I generation quality (0.861) matches the best proprietary system.

**Table 1.** Quantitative comparison on our 5–10-person benchmark. All models are scored on the same 210 examples, and the identity similarities are averaged over the three face encoders of the main protocol. Coverage is the fraction of references whose best-matching generated face reaches a similarity of 0.20, and Dup the fraction that collapse onto a face already claimed by another identity. Purple, light purple, and very light purple mark the first-, second-, and third-best results; for Copy-Paste only the three methods with the highest Sim(Ref) are ranked, since lower similarity means less copying naturally.

**Figure 5:** Scaling to more references. Lines show mean ArcFace similarity to the references; distributions are shown for With-Everyone and GPT-Image 2.

**Figure 6.** Qualitative comparison with proprietary models. Yellow, red, and purple squares mark faces that reproduce a person already generated elsewhere in the image, that are not recognizable as their bound reference, and that appear transplanted without adapting to the pose, lighting, or viewpoint of the scene. These author annotations are illustrative; their quantitative counterparts are the duplicate rate, Sim(Ref), and Copy-Paste.

WithEveryone significantly outperforms baselines in identity similarity and coverage for large groups.

Component Ablations

We revisit explicit identity‑layout planning and show how each ablation impacts that goal.

We revisit the need for explicit identity‑layout planning and show how each ablation impacts that goal.

Adding model‑predicted layout (Layout CoT) modestly improves reference similarity and composition.

Table 2 shows Sim(Ref) rises from 0.339 to 0.364 (+0.025) and Count from 0.771 to 0.828 (+0.057) when moving from P1 to P2.

Using ground‑truth layout further boosts all metrics, establishing an upper bound for planning quality.

Table 2 reports Sim(Ref) 0.412 (+0.048 over P2) and Count 0.958, the highest among all variants.

Introducing an ID token yields a clear gain in identity count.

Table 2 shows Count increases from 0.771 to 0.827 (+0.056) when moving from P1 to P4.

Representation Forcing adds modest similarity gains on top of the ID token.

Table 2 shows Sim(Ref) improves from 0.351 to 0.364 (+0.013) and Sim(Tgt) from 0.313 to 0.328 (+0.015) in P5 versus P4.

The LG‑ID loss provides the largest single improvement, raising reference similarity by over 0.16.

Table 2 shows Sim(Ref) jumps from 0.339 to 0.506 (+0.167) and Sim(Tgt) from 0.304 to 0.435 (+0.131) in P6 versus P1.

**Figure 15.** **LG-ID Loss weight ablation.** Increasing $\lambda_{ID}$ substantially improves Sim(Ref), whereas Layout Score and RLS remain in the same range without a stable ordering across weights.

Face‑size analysis shows that baseline methods lose similarity as faces shrink, but our method’s slope is near flat (–0.002), indicating that face‑resolution degradation does not explain the remaining group‑size drop.

Plan IoU experiments reveal that the model follows its own layout plan closely (IoU ≈ 0.81), yet replacing the plan with ground‑truth layout still yields large identity gains, pointing to plan quality—not execution—as the dominant error source.

Attention visualizations at the supervised identity‑prediction positions exhibit strong diagonal dominance (0.378–0.574) versus off‑diagonal values (0.104–0.121), confirming that Representation Forcing induces identity‑specific attention.

Training Details and Data

Appendix details the training pipeline, configs, metrics, and evaluation nuances.

The training pipeline proceeds in seven stages: (1) reference images and prompt, (2) reference identity selection and loading, (3) face layout and identity binding plus body‑region and pose planning, (4) rendering the plan into a visual layout condition, (5) a recaption that restates the request consistent with the canvas, (6) target identity representation prediction, and (7) target image generation. Each group‑image sample includes a target image, reference images, and identity correspondences, with short user prompts and detailed recaptions. Face detection and ArcFace provide embeddings, while YOLOv11 supplies person regions and keypoints, jointly supervising the structured reasoning stages.

WithAnyone uses a mixture‑of‑transformers backbone with 60 B parameters for understanding and 60 B for generation, initialized from HunyuanImage 3.5‑preview. Training runs on 400 K in‑house group‑image samples (see Figure 4) using Muon optimizer with learning rates $1\times10^{-5}$ (generation) and $3\times10^{-6}$ (understanding), packed sequence length 72 K tokens, on 128 H100 GPUs for 1600 iterations. Objective weights are $\lambda_{\text{FM}}=1.0$, $\lambda_{\text{RF}}=1.0$, and $\lambda_{\text{ID}}=0.5$.

All models are evaluated on the same 210 benchmark examples with identical detection, matching, and aggregation code; no example is excluded. Each evaluation example contains one generated image, 5–10 reference identities, a target group image, and reference‑to‑target identity correspondences. For models with Layout CoT we additionally parse face boxes, body boxes, pose keypoints, and identity–layout bindings from the generated plan; body‑level evaluation uses YOLOv11x‑pose with confidence 0.25 and input size 640, falling back to face‑level terms if body detection fails.

Identity metrics follow the WithAnyone protocol, averaging results from ArcFace, FaceNet, and AdaFace; generated faces are matched to references or targets via maximum‑similarity Hungarian assignment, yielding Sim(Ref) and Sim(Tgt). The Copy‑Paste metric computes $\text{Copy-Paste} = \frac{\theta_{gt} - \theta_{gr}}{\theta_{tr}}$ with $\theta_{ab} = \arccos(\cos(e_a, e_b))$, where a larger value indicates the generated face is closer to the reference than to its target appearance.

CLIP‑I and DINO‑I are cosine similarities between generated and target images in the CLIP and DINOv2 embedding spaces, while CLIP‑T measures similarity between the generated image and the user prompt in the joint CLIP space. These three metrics capture overall scene and prompt agreement and are read alongside the identity metrics rather than in isolation.

Layout Score aggregates seven sub‑scores in $[0,1]$, renormalizing weights if a component cannot be computed. Count uses $S_{\text{count}} = 1 - \min\!\left(1, \frac{|M - N|}{\max(1, N)}\right)$ to penalize missing or extra people. Coverage, uniqueness, and distinctness derive from cosine‑similarity matching with a $0.20$ threshold, yielding $S_{\text{cover}}$, $S_{\text{uniq}}$, and $S_{\text{distinct}}$ respectively. Anonymous leakage is measured by $S_{\text{noleak}}$, spatial validity by $S_{\text{spatial}}$, and plan adherence by $S_{\text{plan}}$, the mean IoU between generated face boxes and layout boxes.

Relative Layout Score (RLS) evaluates the textual plan against ground‑truth layout without using the generated image, combining relative count, position, and size terms as $\text{RLS} = 0.20\,S_{\text{rel\_count}} + 0.45\,S_{\text{rel\_pos}} + 0.35\,S_{\text{rel\_size}}$. Position scoring blends order consistency ($S_{\text{order-x}}$, $S_{\text{order-y}}$) and aligned‑center similarity ($S_{\text{align}}$) into $S_{\text{rel\_pos}}$, while size scoring compares normalized box areas. Plan IoU, in contrast, measures how well the generation side executes the supplied plan by averaging matched IoUs between planned and detected boxes.

Extended background reviews the evolution from diffusion U‑Nets to transformer‑based generators and flow‑matching schedules, noting that permutation‑equivariant token sequences enable interleaving heterogeneous conditioning signals. It also contrasts prior reference‑conditioned and identity‑preserving approaches, emphasizing that scaling to five‑to‑ten identities shifts the bottleneck to supervision and addressing rather than conditioning mechanisms.

Limitations include the difficulty of evaluating layout when prompts are underspecified, the modest benchmark size of 210 examples, and demographic biases inherited from face detectors. Responsible use cautions that identity‑conditioned generation can produce realistic images of real people without consent, so deployments should require consent and include provenance signalling.

**Figure 4.** Data statistics, pipeline, and example. We report the distributions of identity occurrence frequency, reference count, and real photographs versus collage and poster images; the data-construction pipeline; and an example training sample. The corpus contains 400K group-image samples, and the reference-count distribution covers the five-to-ten-person range used at evaluation time.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers