A Unified Multimodal Model for Scalable 3D Generation, Understanding, and Editing

Junliang Ye, Kenkun Liu, Guocun Wang, Yang Li, Yansong Qu, Chunshi Wang, Jingwei Xu, Yunhan Yang, Zibo Zhao, Jiachen Xu, Jiaao Yu, Lifu Wang, Zhihao Liang, Zhuo Chen, Chunchao Guo

A unified 3D multimodal framework integrating understanding, generation, and editing via an 87M-sample training corpus.

How can we unify 3D understanding, generation, and editing into a single scalable multimodal model despite the scarcity of high-quality, geometrically consistent 3D training data?

3D modeling remains fragmented because specialized systems for understanding, generation, and editing cannot share knowledge, largely due to a lack of large-scale, geometrically consistent training data. Hunyuan3D-Buffalo 1.0 unifies these tasks by training a 3D Vision-Language Model (VLM) and a Diffusion Transformer (DiT) on an 87-million-sample corpus, using a learned spatial-constraint pipeline to generate high-quality editing pairs. The framework achieves state-of-the-art performance on text-to-3D and editing benchmarks, demonstrating that unified training allows understanding and generation capabilities to mutually reinforce editing accuracy.

Paper Primer

The core mechanism hinges on a dual-module architecture: Hunyuan3D-VLM acts as the semantic engine for 3D perception, while a 3D-DiT handles high-fidelity synthesis. The system uses an MLP-Connector to inject VLM-derived semantic features into the diffusion process, allowing the model to perform instruction-following tasks like part-segmentation and editing within a single, unified interface.

To solve the data bottleneck, the authors developed Nano3D-v2, an agentic pipeline that automates the creation of editing pairs. It functions like a surgical editor: it identifies the optimal viewpoint, predicts a precise 3D bounding box to isolate the mutable region, and performs voxel-level modifications while freezing the surrounding geometry to prevent identity drift.

Hunyuan3D-Buffalo 1.0 significantly outperforms existing 3D generative baselines in human preference studies.

In a four-way blind user study, the model achieved a 56.6% overall preference rate, more than double the 18.4% rate of the strongest baseline (Omni123). 56.6% preference vs. 18.4% (Omni123) and 21.0% (TRELLIS).

Unified training induces cross-task synergy, specifically improving 3D editing performance.

Analysis shows that stronger generative priors lead to more natural edited geometry, while improved 3D understanding helps the model localize editing regions and preserve unedited areas. Direct correlation between training scale (up to 50M samples) and preference metrics.

Why is this approach more effective than simply combining separate 3D models?

Separate models fail to learn a unified semantic-visual-geometric representation. By training on a joint corpus, the model learns that 3D understanding (e.g., part localization) and generation (e.g., structural priors) directly improve the precision and consistency of 3D editing.

What is the scope of the "part generation" capability?

The model treats part generation as a native instruction-following task rather than a specialized pipeline. It can segment or remove parts (e.g., "remove the wheels") by conditioning the diffusion process on the source object and the specific semantic part concept.

The framework relies on the assumption that 3D editing can be effectively modeled as a conditional generation task where the source object's geometry acts as a frozen constraint, preventing the "identity drift" common in 2D-to-3D lifting methods.

Researchers can now treat 3D editing and part-level manipulation as native instruction-following tasks, moving away from per-instance optimization toward scalable, feed-forward 3D multimodal systems.

Introduction to Hunyuan3D-Buffalo

We expose data scarcity in unified 3D modeling and propose a massive multimodal corpus with a VLM‑DiT framework.

Unified multimodal models have excelled in 2D vision, yet extending this unity to 3D is hampered by a chronic lack of large, diverse, and geometrically consistent multimodal data—especially for editing. Existing 3D pipelines remain fragmented into separate understanding, generation, and editing systems, preventing the emergence of a shared semantic‑geometric representation. Consequently, it is unclear whether joint training can let these tasks reinforce one another.

**Figure 1.** Overview of the Hunyuan3D-Buffalo 1.0 framework, featuring a large-scale 3D multimodal dataset and a unified 3D multimodal architecture.

**Fig. 1:** Hunyuan3D-Buffalo 1.0 is an unified 3D multimodal framework that combines autoregressive modeling with diffusion-based generation, enabling 3D understanding, text-to-3D generation, 3D editing, and text-grounded part generation within a single architecture.

A single model that jointly learns to understand, generate, edit, and decompose 3D objects by coupling a vision‑language encoder with a diffusion transformer.

Shifting from task‑specific 3D models to a unified multimodal framework unlocks mutual reinforcement among understanding, generation, and editing.

Related Work

We survey recent 3D generation, editing, part generation, and unified multimodal modeling approaches.

Recent 3D generation work falls into three paradigms. Optimization‑based methods such as DreamFusion distill 2D diffusion priors into 3D without large‑scale 3D data. 3D‑native VAE approaches (e.g., 3DShape2VecSet, TRELLIS) enable feed‑forward generation in latent space, while low‑poly mesh generators like MeshAnything model vertices and faces autoregressively.

Text‑guided 3D editing seeks to modify assets via language while preserving untouched regions and multi‑view consistency. Early per‑instance optimization uses Score Distillation Sampling (SDS) or 2D diffusion priors, whereas training‑free methods (Nano3D, VoxHammer) edit latent or structured representations of frozen 3D priors. Training‑based approaches fine‑tune generators on paired or self‑generated edit data but still struggle with precision and efficiency.

3D part generation aims to output meshes where each semantic component is a separate part. Early pipelines relied on fixed taxonomies or 2D segmentation, leading to view‑dependent errors. Recent 3D‑native methods use multi‑diffusion‑path architectures or continuous feature fields, and CubePart introduces an open‑vocabulary, text‑grounded framework that unifies part generation with other 3D tasks.

Unified multimodal modeling for images has followed three architectural lines: token‑based autoregressive Transformers (Chameleon) that treat images as discrete VQVAE tokens; decoupled designs (MetaQuery) that pair a Multimodal Large Language Model with a Diffusion Transformer; and tightly integrated backbones (Bagel) that combine language modeling and flow‑based generation. In 3D, efforts like ShapeLLM‑Omni still rely on VQVAE tokenization and struggle with fine‑grained geometry.

Data Curation Strategy

We build three large, geometry‑consistent corpora to feed a unified 3D VLM‑DiT pipeline.

Large‑scale 3D multimodal modeling stalls because high‑quality, geometrically consistent data are scarce. To break this bottleneck we assemble a unified data engine that yields three complementary corpora.

Table 1 summarizes the scale: ~25 M understanding samples, ~50 M text‑to‑3D pairs, and ~12 M editing triples, each designed to preserve language, 2‑D vision, and 3‑D geometry jointly.

The pipeline turns a compact, hierarchical prompt into a high‑quality 3‑D asset, then attaches geometry‑aware captions while discarding any broken geometry.

Stage 1 samples the two hierarchical specifications and draws the attribute set (optional drops leave the first prompt with all four attributes, the second with material omitted).

Stage 2 assembles the natural language requests: “A realistic black metal office chair, upright.” and “A realistic knight wielding a sword and a shield, standing.”

Stage 3 runs each request through the Text‑to‑Image model, then the Image‑to‑3D model, producing two meshes and four canonical renders per mesh.

Stage 4 calls Gemini VLM on the renders; it returns six caption tiers for each asset and scores 10 for the chair (clean geometry) and 8 for the knight (minor topology glitch).

Stage 5 filters out the knight (score < 10) and packages the chair as (point‑cloud, caption dict, “Furniture”) for training.

The example shows how hierarchical sampling, optional attribute dropping, and strict geometry scoring together guarantee diverse yet geometrically reliable training pairs.

How does this hierarchical prompt taxonomy differ from a naïve random prompt generator?

Random generators ignore semantic constraints, often producing implausible or unrenderable requests (e.g., “a floating cube with a scent”). Our taxonomy enforces a mode‑category‑type hierarchy and attribute compatibility tables, so every sampled prompt maps to a feasible 3‑D scene that the downstream generator can actually render.

**Fig. 2:** Pipeline of constructing text-to-3D training corpus.

The 3D understanding corpus interleaves pure text, image‑text, and point‑cloud‑text dialogues, preserving language and 2‑D perception while adding ~15 M geometry‑grounded instruction samples.

The editing corpus supplies (source, edited, instruction) triples generated by the Nano3D‑v2 pipeline and refined with a VLM‑based filter, ensuring source‑target geometry stays consistent.

Without strict geometry‑quality filtering the model would learn to associate textual cues with malformed meshes, degrading both generation fidelity and editing precision.

Constructing Editing and Part Data

We describe Nano3D‑v2, a five‑stage pipeline that builds high‑quality 3D editing pairs automatically.

Existing 3D‑editing pipelines either lift 2D edits back to 3D—suffering identity drift and view inconsistency—or constrain edits with hand‑crafted 3D boxes, which remain low‑quality and hard to scale.

Instead of editing in 2D then reconstructing, or manually bounding a region, Nano3D‑v2 learns where to edit in 3D and refines only that locality, keeping the rest of the asset untouched.

Anchor View Selection renders eight views; the front view shows the face most relevant to the instruction, so it is chosen as the anchor.

Editing Planning computes a 2D mask highlighting the front‑face pixels that changed, and the Transformer predicts a 3D box covering voxels (x = 1..4, y = 1..4, z = 4).

Voxel Editing runs FlowEdit inside this box, setting the top‑layer voxels to a higher density to form a protrusion, then merges all voxels outside the box unchanged.

LATTICE refines the protrusion surface, adding sub‑voxel detail, while NaTex inpaints the texture to match the surrounding material.

Alpha blending over a 1‑voxel band smooths the transition, yielding a clean edit without affecting the cube’s other faces.

Localizing the edit prevents the whole model from drifting, and the refinement stages restore high‑frequency geometry that raw voxel editing would lose.

How does Nano3D‑v2 differ from a 2D‑to‑3D lifting pipeline?

Lifting pipelines edit only in 2D and then reconstruct, which provides no explicit correspondence between source and target voxels, leading to identity drift and view inconsistency. Nano3D‑v2 predicts a 3D edit region first and performs voxel‑level modification directly, guaranteeing that unchanged voxels stay identical and that multi‑view consistency is preserved.

Render eight canonical views of the source asset; a VLM selects the anchor view with highest semantic relevance to the instruction and edits that view with Qwen‑Image.

Compute a pixel‑wise difference between the source and edited anchor images to obtain a 2D mask; feed the mask and the source voxel grid into an autoregressive Transformer that outputs a tight 3D bounding box.

Condition the TRELLIS voxel Transformer on the edited anchor image and the predicted box; perform FlowEdit inside the box and merge unchanged voxels from the source outside the box.

Pass the merged voxel to LATTICE for sub‑voxel geometry refinement, then to NaTex for texture inpainting; apply a 7‑voxel alpha‑blend along the box boundary.

Render multi‑view pairs of source and edited assets; a VLM filters out structurally broken results, re‑derives concise edit instructions (Addition/Replacement/Removal), and verifies integrity, instruction‑geometry alignment, non‑edited region consistency, and visibility.

**Fig. 3.** Pipeline of constructing 3D editing training corpus (Nano3D-v2).

**Figure 4.** Examples of editing pairs in the training corpus created by Nano3D-v2.

**Fig. 5:** Examples of multi-round editing by Nano3D-v2.

Unified Architecture and Training

Unify 3D understanding, generation, and editing by linking a VLM to a diffusion transformer.

Multimodal 3D data are scarce, so a single model must reason about geometry, appearance, and language without over‑fitting any one modality.

A Vision‑Language Model that ingests colored point clouds through separate shape and appearance streams, turning them into a compact token sequence that can be fused with text and images.

The structural pathway encodes (x,y,z,n) → shape tokens s₁, s₂.

The semantic pathway encodes (r,g,b) → appearance tokens a₁, a₂.

VecSet merges (s₁,a₁) and (s₂,a₂) into two 128‑dim vectors.

Q‑Former compresses the two vectors into a fixed 512‑token sequence.

The 512 tokens are concatenated with the text tokens “a red cube next to a green cube”.

The compact 512‑token representation preserves both geometry and color while fitting the VLM’s fixed‑length input budget.

How does 3D‑VLM differ from a standard Vision‑Language Model that only sees images?

Standard VLMs treat an image as a 2‑D raster and lack explicit geometry. 3D‑VLM adds a structural pathway that consumes raw point‑cloud coordinates and normals, giving the model direct access to shape and part boundaries that images cannot convey.

A diffusion‑based transformer that synthesizes 3D content, driven by high‑level semantic embeddings from the VLM.

Why not simply concatenate VLM embeddings with DiT latents instead of using an MLP‑Connector?

Direct concatenation would mismatch dimensionalities and break the pretrained DiT’s internal scaling. The MLP‑Connector learns a linear‑plus‑nonlinear projection that respects both spaces, preserving the DiT’s diffusion dynamics while still injecting high‑level semantics.

During editing or part generation, the original object's latent representation is fed alongside the noisy diffusion latent so the model can keep unchanged regions intact.

Why does concatenating the source representation help preserve geometry better than feeding it only at the first diffusion step?

Providing the source at every self‑attention layer continuously re‑injects the exact geometry, preventing drift that would accumulate if the model only saw the source once at the start.

**Fig. 6: Hunyuan3D-Buffalo 1.0 pipeline.** The framework unifies 3D QA and grounding, text-to-3D generation, and 3D editing through a shared Hunyuan3D-VLM backbone, which connects language, 3D representations, and generative Hunyuan3D DiT modules for multi-modal understanding, generation, and editing.

**Fig. 7:** Overview of the training pipeline. The shared trunk proceeds through 3D-VLM pre-training, text-to-3D pre-training, and omni pre-training. It then branches into three continued pre-training paths: 3D editing, text-to-3D, and part generation.

3D Understanding Evaluation

Hunyuan3D‑VLM sets new state‑of‑the‑art on part‑centric 3D perception benchmarks.

Hunyuan3D‑VLM outperforms all baselines on part‑understanding Q&A, reaching 85.47 SBERT and 89.06 SimCSE.

Table 3 shows the model’s scores dominate those of GPT‑4Point, PointLLM, ShapeLLM, Part‑X‑MLLM, and UniVerse3D.

UniPart‑Bench measures how well a model can perceive, describe, and reason about individual parts of a 3D object, not just the whole shape.

How does UniPart‑Bench differ from generic 3D perception benchmarks?

Most 3D benchmarks evaluate only whole‑object geometry or coarse categories. UniPart‑Bench adds per‑part bounding boxes, dual‑level textual annotations, and part‑aware QA, forcing a model to understand and generate language about individual components.

The evaluation covers all eight UniPart‑Bench tasks, reporting IoU for box localization and SBERT/SimCSE/BLEU‑1/ROUGE‑L/METEOR for textual outputs. Compared with strong 3D multimodal baselines—GPT‑4Point, PointLLM, ShapeLLM, Part‑X‑MLLM, and UniVerse3D—our unified VLM‑DiT architecture consistently leads, especially on the fine‑grained part‑understanding Q&A and overall captioning tracks.

**Table 3.** Results of Hunyuan3D-VLM on various tasks.

**Table 4.** Detailed all-task evaluation of Hunyuan3D-VLM on UniPart-Bench [102]. The benchmark covers pure box listing, multi-part grounding, single-part grounding, box-to-text generation, and part-level question answering.

Hunyuan3D‑VLM delivers the strongest part‑centric 3D perception performance across localization, description, and reasoning.

Text-to-3D Generation Results

Human study shows our model vastly outperforms baselines on text alignment and geometry.

The paper unifies 3D understanding, generation, and editing by training a large multimodal VLM‑DiT on a massive curated corpus. Here we evaluate the generation capability on text‑to‑3D.

A diffusion‑based pipeline that first predicts a coarse shape then refines it with voxel‑level detail.

A latent‑space generator that maps a text embedding to a global 3D latent, which is then decoded into a mesh.

A hybrid VLM that first predicts a point cloud from text, then converts it to a mesh via a learned surface‑reconstruction module.

Our method is preferred over all three baselines across text alignment, geometry quality, and overall preference.

Table 5 shows preference rates of 55.2 % (text), 57.1 % (geometry), and 56.6 % (overall) for our model, far above the strongest baseline.

**Fig. 8:** Qualitative text to 3D results.

3D Editing Performance

Ablation study shows how 3D‑VLM conditioning drives superior editing on Edit3D‑Bench.

Unified 3D modeling hinges on a single model that can understand, generate, and edit shapes from language. This section tests how each editing component contributes to that goal.

We evaluate on Edit3D‑Bench, which provides paired source‑target meshes for addition and removal operations. Performance is measured with Chamfer Distance (lower is better) and F1 score (higher is better).

**Table.** Comparison of different methods on 3D editing tasks.

Hunyuan3D‑Buffalo 1.0 reduces average Chamfer Distance by 86.7 % relative to the strongest baseline (Omni123).

Table 5

The model improves average F1 by 2.39× over the best competing editor (Steer3D).

Switching the conditioning from CLIP to 3D‑VLM lowers average CD by ≈42 % and raises average F1 from 0.6336 to 0.6515.

Qualitative inspection (Fig. 9) confirms that our edits stay localized: added accessories appear at the correct semantic spot, and removed parts disappear without distorting the rest of the mesh.

**Fig. 9:** Qualitative shape editing results. Our method significantly outperforms all baselines in both geometric consistency before and after editing, and responsiveness to editing instructions.

We further probe the link between generation and editing by scaling text‑to‑3D data. Adding 1 000 chicken‑head samples during pre‑training (Fig. 10) enables successful head replacement without any dedicated editing examples.

**Fig. 10:** Scaling up text-to-3d data facilitates 3D editing. Model A is our base model; when instructed to edit an object by replacing its head with a chicken head, it fails to produce a satisfactory result. Model B is built upon Model A by adding only 1,000 additional chicken samples for the text-to-3D task during the **Omni pre-training stage**—crucially, without introducing any new editing data. After incorporating this text-to-3D data, the model can successfully replace the head with a chicken head. This suggests a clear direction: to improve 3D editing, the text-to-3D generation capability should be maximized as much as possible. Since constructing text-to-3D data is far less costly than constructing 3D editing data, scaling up text-to-3D data is a relatively more feasible path toward stronger 3D editing.

Conclusion

We wrap up the unified 3D model, its achievements, and open research directions.

Figure 11 illustrates that a natural‑language query can reliably extract arbitrary parts—from a complex octopus to a simple wheel—demonstrating open‑vocabulary, text‑grounded part generation within our unified framework.

**Fig. 11:** Qualitative part generation results.

**Fig. 12:** Qualitative shape editing results.

**Fig. 13:** Qualitative text to 3D results.

Overall, the unified multimodal 3D model sets new performance records on understanding, generation, and editing, confirming that a single framework can excel across diverse 3D tasks.

Future work must resolve the open problem of single‑stage, high‑quality geometry generation, which currently relies on multi‑stage DiT pipelines that hinder scalability.

Improving the captioning quality of text‑to‑3D and editing datasets—currently sourced from large‑scale multimodal LLMs—will reduce training noise as those models mature.

Extending the model to end‑to‑end texture editing requires new data and a joint representation that can handle both geometry and appearance.

Our editing data pipeline, based on Nano3D‑v2, still yields inconsistent interior regions within masked boxes; a more precise construction method is needed to avoid quality loss.

Exploring Transfusion‑style architectures that deeply fuse multimodal signals could further boost 3D generation, mirroring successes in image and video domains.

Finally, scaling both the volume and fidelity of 3D datasets remains essential for continued progress toward truly unified 3D models.

We thank the Omni123 team for providing comparative results and Zehuan Huang for his internship contributions.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers