ZipTok3D: High-Fidelity 3D Tokenization with Compact Token Prefixes

Mingda Lin, Weijie Wang, Zeyu Zhang, Bowen Cui, Yefei He, Haoyu Zhao, Yuanyu He, Donny Y. Chen, Feng Chen, Bohan Zhuang

ZipTok3D enables high-fidelity 3D reconstruction from extremely short token sequences using nested prefixes and iterative refinement.

How can we represent 3D shapes using extremely compact token sequences without sacrificing reconstruction fidelity?

Existing 3D tokenizers struggle to maintain geometric fidelity when compressed to very few tokens, as they force global structure and fine-grained details to compete for limited latent capacity. ZipTok3D solves this by organizing geometry into progressively informative prefixes and unfolding them through a parameter-shared Transformer that iteratively refines the 3D representation. This approach allows a single token to achieve reconstruction quality comparable to 32-token baselines on ShapeNet, reducing sequence length by 32×.

Paper Primer

The core mechanism hinges on two controls: nested dropout during training, which forces the leading tokens to capture essential object-wide geometry, and a parameter-shared decoder that repeatedly updates the spatial representation. This is like a sketch artist who first draws the rough outline of a subject and then passes over the same canvas multiple times to add increasingly fine details, rather than trying to render the entire image in one stroke.

ZipTok3D achieves high-fidelity reconstruction at significantly lower token budgets than fixed-budget autoencoders.

On ShapeNet, a one-token ZipTok3D prefix matches the CD and F1 metrics of a 32-token COD-VAE baseline. 32× reduction in latent sequence length on ShapeNet; 8× reduction on TRELLIS.

Iterative refinement provides consistent gains across diverse object categories without adding step-specific parameters.

Ablation studies show that repeated application of the shared Transformer block is the primary driver of reconstruction improvement in the few-token regime. Refinement improves reconstruction for over 90% of objects in the 1-to-5 pass transition.

Why does this approach outperform standard fixed-budget tokenizers?

Standard tokenizers optimize for a fixed budget, leaving the model to implicitly resolve the competition between global and fine-grained information. ZipTok3D explicitly forces the leading tokens to retain object-wide geometry through nested dropout, ensuring the decoder always has a valid, compact representation to work from.

Does this method require different decoder parameters for different token budgets?

No. The decoder uses a single, parameter-shared Transformer block that is applied for a variable number of refinement steps, allowing the same checkpoint to operate across different token budgets and refinement depths.

Introduction and Motivation

Compact token sequences are essential, but existing 3D tokenizers collapse when forced to be very short.

Existing 3D tokenizers either scatter geometry over spatially anchored tokens or compress it into a fixed set of global tokens, and both approaches suffer severe reconstruction loss when the token budget is forced to be extremely low. This creates a fundamental tension: shorter latent sequences reduce downstream modeling cost but inevitably sacrifice geometric fidelity. We therefore need a representation that packs essential shape information into the first few tokens while still allowing detailed reconstruction.

Current 3D tokenizers spread object geometry across many tokens, so when we compress them to a handful of tokens the reconstruction collapses; the goal is to force the most informative geometry into the leading tokens so that even an extremely short sequence can recover the full shape.

**Figure 1.** Qualitative reconstruction at different token budgets. Ground-truth shapes are shown in the upper left. VecSet uses 512 global tokens, while COD-VAE uses 32. ZipTok3D reconstructs the same objects from the first token of its ordered latent sequence with five shared refinement steps.

The key trade‑off is between how few tokens we allocate and how faithfully the 3D shape can be reconstructed.

Related Work

Survey of 3D tokenization and refinement methods, highlighting their trade‑offs.

Hierarchical point‑cloud feature extractor that builds a multi‑scale representation of 3D geometry.

Uses irregular grids to store latent features, allowing adaptive spatial resolution.

Octree‑based latent representation that fuses features across hierarchical cells.

Builds sparse voxel hierarchies to encode geometry with a compact set of occupied cells.

Attaches latent vectors to occupied cells of a sparse grid, enabling efficient retrieval.

Compresses voxel‑based latents into coarse anchors, trading spatial detail for memory savings.

Sparse geometry‑and‑appearance latent codes that jointly encode shape and texture.

Obtains a set of global latent vectors via cross‑attention over point features, forming a compact token set.

Progressively compresses point features into a small latent code that is decoded into triplane geometry via a variational auto‑encoder.

Extends permutation‑invariant token sets to multi‑view scene modeling, enabling view‑consistent generation.

Organizes tokens in a hierarchical tree, allowing coarse‑to‑fine decoding.

Quantizes latent features at multiple scales, preserving detail while reducing code size.

Samples points preferentially on high‑curvature regions to preserve surface detail.

Condition a flow‑matching surface field on a compact set of learned shape codes.

Uses quantized shape codes for multimodal autoregressive scene modeling, linking geometry to language.

Accelerates VecSet‑based generation via diffusion distillation and an efficient implicit decoder.

Studies block‑wise diffusion for efficient text‑to‑3D generation, processing groups of tokens together.

Randomly truncates latent sequences during training, imposing an ordering that makes early prefixes more informative.

Learn token sequences where a short prefix can already reconstruct a coarse image, with later tokens adding detail.

Allocate more tokens to complex inputs and fewer to simple ones, matching capacity to content difficulty.

Learns coarse‑to‑fine video prefixes with a generative flow decoder, enabling flexible video length decoding.

Improves utilization of later tokens in nested‑dropout representations, reducing waste in the tail of the sequence.

Allocates octree tokens according to shape complexity, using more tokens where geometry is intricate.

Adapts supervoxel size to local detail while following a fixed generation order, enabling efficient 3D synthesis.

Orders tokens by semantic salience; early prefixes encode overall shape, later tokens add instance specifics.

Learns nested reconstructive prefixes of global latents and decodes them via a shared iterative refinement module, without spatial partitioning.

Applies the same self‑attention and feed‑forward block recurrently, allowing the model to adapt its depth.

Extends the Universal Transformer idea to image and video generation, using weight‑shared loops and intra‑loop self‑distillation.

Recovers fine geometric detail from coarse predictions via a cascade of coarse‑to‑fine refinement stages.

Shares operations across recurrent levels to reduce parameter and memory costs while progressively increasing point density.

COD‑VAE compresses a point cloud into a tiny latent code that a decoder expands into a triplane representation.

VecSet gathers a fixed‑size set of global vectors from a point cloud via cross‑attention, forming a compact token representation.

The ZipTok3D Architecture

ZipTok3D packs geometry into a short prefix and iteratively refines it into detailed triplane features.

The low‑token reconstruction problem is two‑fold: a short latent prefix must retain enough geometry, and the decoder must turn that compact code into a full 3‑D field without exploding compute.

Nested dropout forces the first K tokens of the latent sequence to carry all information needed for reconstruction, so the model can be truncated at any budget without losing fidelity.

How does nested dropout differ from standard token dropout?

Standard dropout removes random individual tokens, which can break the sequential structure; nested dropout removes a contiguous suffix, guaranteeing that the remaining prefix is still a coherent, ordered representation that the decoder can consume.

Think of the prefix as a concise summary of the shape: it packs the essential geometry into $K$ tokens so that a later decoder can rebuild the full object from this short code.

Step 1: Retain the prefix $Z_{:4} = (z_1,z_2,z_3,z_4)$ and discard $z_5\ldots z_8$.

Step 2: The selection module $S_\omega$ pairs $Z_{:4}$ with the initial triplane tokens $T^0$, yielding $H^0_4$ and a set of redundant tokens $R_4$.

Step 3: The shared transformer block will later refine $H^0_4$ while $R_4$ stays unchanged.

Step 4: After $L$ refinement steps the final triplane $T^L_4$ is reconstructed and queried by the occupancy MLP.

The prefix length $K$ directly controls how much geometry is packed into the early tokens; smaller $K$ forces those tokens to be more information‑dense.

Why not simply use the full latent sequence $Z$ at inference?

Using the full $Z$ would fix the token budget and prevent the model from adapting to different compute constraints; the reconstructive prefix lets the same network run at multiple budgets without retraining.

Iterative refinement repeatedly applies the same transformer block to the triplane state, gradually unfolding the coarse geometry encoded in the prefix into finer detail.

Iterative refinement loop – the same transformer block $g_\theta$ is applied $L$ times.

How does this differ from stacking $L$ separate transformer layers?

Stacking $L$ distinct layers would introduce $L$ × more parameters and break the weight‑sharing property; iterative refinement reuses the same parameters, so the model gains depth without increasing memory or over‑fitting risk.

Triplane features store 3‑D information on three orthogonal 2‑D planes, allowing fast lookup of spatial features by projecting a query point onto each plane.

**Figure 2** Overview of ZipTok3D. Nested dropout concentrates geometry into a reconstructive prefix $Z_{:K}$, while the parameter-shared block progressively unfolds it into triplane features over $L$ refinement steps. Blue and green denote latent and triplane tokens, respectively; hatched tokens indicate the masked suffix.

Experimental Results

ZipTok3D concentrates geometry into a tiny prefix and refines it iteratively, achieving high‑fidelity reconstructions with far fewer tokens.

ZipTok3D improves query IoU by +0.06 points over COD‑VAE‑32 on TRELLIS while using 32× fewer tokens.

Table 1 reports a +0.06 IoU increase for ZipTok3D with $K\!=\!4$ compared to COD‑VAE‑32.

**Table 1.** Reconstruction from Extremely Short Prefixes

**Figure 3.** Qualitative reconstructions on ShapeNet (top) and TRELLIS (bottom). Columns show ground truth, COD-VAE with $K = 32$ and $K = 64$, VecSet with $K = 512$, and ZipTok3D with $K = 1$ and $K = 2$. Both ZipTok3D settings use five refinement steps.

ZipTok3D achieves comparable fidelity to baselines with significantly fewer tokens.

Ablation and Sensitivity Analysis

Analyzes how prefix length and refinement depth affect reconstruction quality.

We probe the joint impact of prefix length $K$ and refinement depth $L$ on three surface metrics: Mesh F1@0.02, Mesh CD, and query IoU.

**Figure A.** Surface-metric interactions across prefix lengths $K$ and refinement depths $L$ on ShapeNet (top) and TRELLIS (bottom). The left panel reports mesh F1@0.02; the right panel reports mesh CD.

**Figure B.** Complete ShapeNet (left) and TRELLIS (right) sweeps for refinement depths $L = 1, \dots, 6$ and all supported prefix lengths. Rows report query IoU, mesh F1@0.02, and mesh CD. Cell values are dataset means; CD is reported in units of $10^{-3}$. To orient all metrics so that better values receive darker colors, we define the residual from the optimum as $r = 100 - m$ for percentage-valued IoU and F1 and as $r = m$ for CD. Within each dataset-metric panel, we apply standard min-max normalization to $-\log r$. If $r_{\min}$ and $r_{\max}$ are the minimum and maximum residuals over its displayed $(K, L)$ cells, the resulting color score is $q = \log(r_{\max}/r) / \log(r_{\max}/r_{\min})$. Thus, $q = 1$ denotes the best cell and $q = 0$ the worst. Darker colors indicate larger $q$.

When refinement is shallow ($L\le 3$), increasing $K$ yields up to a 12 % boost in Mesh F1@0.02.

Observed on the $K$‑sweep heatmaps (Figure 6) for $L=1$ and $L=2$.

At $K\!=\!1$, moving from $L\!=\!1$ to $L\!=\!3$ improves IoU for 100 % of ShapeNet objects.

Table 4 counts objects whose IoU strictly increases.

At $K\!=\!1$, moving from $L\!=\!1$ to $L\!=\!3$ improves IoU for 98.5 % of TRELLIS objects.

At $K\!=\!4$, moving from $L\!=\!1$ to $L\!=\!3$ improves IoU for 88.2 % of ShapeNet objects.

Table 4 shows the improvement rate for the $K\!=\!4$ setting.

At $K\!=\!4$, moving from $L\!=\!3$ to $L\!=\!5$ improves IoU for 91.2 % of ShapeNet objects.

Table 4 reports the later‑refinement transition.

At $K\!=\!1$, moving from $L\!=\!3$ to $L\!=\!5$ reduces Mesh CD for 99.8 % of ShapeNet objects.

Table 4 CD‑decrease column for the $K\!=\!1$ row.

Adaptive Budget Analysis

Analyzes budget distribution and success rates of ZipTok3D versus COD‑VAE under fixed token budgets.

We evaluate the adaptive budget behavior of ZipTok3D and the COD‑VAE‑32 baseline on the ShapeNet and TRELLIS datasets, restricting attention to the subset of objects for which both methods meet all metrics.

The table reports average IoU, Chamfer Distance ($\text{CD}$), F1 score, inference rate, and the mean prefix length $K$ and refinement depth $L$ for each method; ZipTok3D consistently attains higher IoU and lower $\text{CD}$ while using far fewer tokens.

The remaining 304 ShapeNet objects and 846 TRELLIS objects have no candidate satisfying the strict all‑metric criterion, so they are excluded from the oracle averages reported in Table C.

Qualitative inspection shows ZipTok3D preserves thin components, large openings, and curved surfaces (e.g., open shelves, chairs, vessels, aircraft) while COD‑VAE often loses these details; on TRELLIS, ZipTok3D recovers multi‑level building layouts and repeated roof structures.

Experimental Setup Details

Details on datasets, preprocessing, baselines, and training configurations.

ShapeNetCore‑v2 provides 55 categories; we follow the split from 3DShape2VecSet, using 48,597 objects for training, 2,592 for validation, and 1,283 for testing.

During training we sample independent axis scales uniformly from $[0.75, 1.25]$, renormalize shapes to the unit cube, and add Gaussian noise ($\sigma=0.005$) to surface points before clipping to $[-1,1]$.

TRELLIS‑500K supplies 500,777 assets; after merging polygonal components and watertight conversion we keep 100,000 surface points and 500,000 volume and near‑surface queries per object.

Assets are filtered by a 900‑second timeout and a 32 GiB memory cap; failures in mesh conversion, normalization, or sampling lead to exclusion.

With seed 42 the TRELLIS assets are split 97 %/2 %/1 % into training, validation, and test, yielding a test set of 2,613 objects across ABO, Objaverse‑XL GitHub, and Sketchfab.

Baseline comparisons include 3DILG, VecSet, and COD‑VAE; all use the shared ShapeNet test split and COD‑VAE metric implementation.

For TRELLIS, COD‑VAE checkpoints are fine‑tuned from ShapeNet weights and selected by highest mean query IoU on the validation split.

Reconstruction token budgets are 512 for 3DILG, 32/64/512 for VecSet, 2/32/64 for COD‑VAE, and 1/2/4 for ZipTok3D.

Stage‑1 tokenizer training samples 2,048 surface points and 4,096 volume plus near‑surface queries per example, optimized with AdamW ($\beta_1=0.9$, $\beta_2=0.999$, $\epsilon=10^{-8}$, weight decay $0.01$, LR $10^{-4}$) and gradient clipping at $0.5$.

Training runs on four A800 GPUs with an effective batch size of 672 (56 examples per GPU, gradient accumulation over three steps); ShapeNet uses 1,000 epochs with 50‑epoch linear warmup then cosine decay, while TRELLIS uses 300 epochs at constant LR.

Prefix VAE training mirrors Stage‑1 hyperparameters but uses a global batch size of 376 and learning‑rate reductions at epochs 60, 70, 80, 90; checkpoints are chosen by highest query IoU on the ShapeNet validation split.

EDM training employs AdamW with weight decay $0.05$, batch size $256$, gradient clipping $1.0$, a 1,000‑epoch schedule with 40‑epoch linear warmup and cosine decay to $10^{-6}$, and hyperparameters $p_{\text{mean}}=-1.2$, $p_{\text{std}}=1.2$, $σ_{\text{data}}=1$.

Architecture Specifications

Describes the Stage‑1 tokenizer, triplane initialization, and shared refinement architecture.

This section details the three core components of ZipTok3D’s architecture: the Stage‑1 tokenizer, the triplane initialization and selection stage, and the shared refinement module that iteratively restores high‑fidelity geometry.

The tokenizer converts raw point clouds into a compact latent bank that serves as the foundation for downstream refinement.

The selection block quickly isolates the most informative triplane tokens, keeping a high‑scoring subset for refinement while preserving the rest for later restoration.

A single Transformer block is reused across refinement passes, updating the retained prefix and selected triplane tokens without adding per‑pass parameters.

Together, these components enable ZipTok3D to compress 3‑D geometry into a short prefix and recover detailed structure through efficient, shared refinement.

Training Objectives

Supplementary objectives detail auxiliary losses, the prefix VAE, and the class‑conditional EDM.

This section spells out the three auxiliary objectives that complement the main reconstruction loss: (1) COD‑VAE‑style losses that keep the early triplane stable and provide an uncertainty signal, (2) a prefix‑VAE that learns a compact latent code for a variable‑length prefix of stage‑1 tokens, and (3) a class‑conditional diffusion model that denoises latent arrays conditioned on object category.

Two COD‑VAE‑derived losses keep the early triplane representation well‑behaved and teach the model to predict uncertainty for each query.

The prefix VAE compresses a variable‑length prefix of stage‑1 tokens into a low‑dimensional Gaussian, enabling flexible generation of partial token sequences.

The class‑conditional diffusion model denoises a latent array while respecting the object category, allowing generation from a short prefix without autoregressive loops.

Evaluation Protocols

Supplementary evaluation details the metrics, efficiency protocols, and statistical analysis used to assess ZipTok3D.

In S4.1 the reconstruction protocol samples exactly 2,048 points from each target object's surface pool, making the task deterministic. Query IoU is obtained by decoding the representation at all 500 k stored volume‑query locations, while mesh quality is measured by extracting the 0.5‑probability isosurface from a dense 1,283‑cell occupancy grid and computing Chamfer Distance, precision, recall, and F1 at a 0.02 distance threshold.

S4.2 evaluates class‑conditioned generation on five ShapeNet categories, generating 2,000 shapes per class and reporting three CD‑based metrics: minimum matching distance (MMD‑CD), coverage (COV‑CD), and 1‑nearest‑neighbor accuracy (1‑NNA‑CD). For each category the reference set consists of all test‑split objects, while a deterministic subset of five times the reference size is used for MMD‑CD and COV‑CD, and an equal‑size subset for 1‑NNA‑CD, ensuring fair comparison across methods.

S4.3 describes the efficiency measurement setup: all timings run on a single NVIDIA H20 GPU with batch size 16 in FP32 mode, disabling TF32 and Flash SDP. For reconstruction, full inference includes point encoding, tokenizer decoding, and the 1,283 occupancy queries (processed in 131 k‑point chunks); decoder latency isolates the stage‑1 prefix decoding and triplane generation, omitting point encoding and queries.

S4.4 reports that shorter prefixes do not reduce the cost of dense occupancy queries nor the number of shared refinement calls, so ZipTok3D’s end‑to‑end throughput is lower than that of single‑pass COD‑VAE variants despite its shorter latent sequence. Peak memory consumption remains comparable because the encoder and dense‑field query dominate allocation.

S4.5 performs a bootstrap analysis with 20 000 resamples of 2 613 objects each, using seed 123456. For every metric the difference (ZipTok3D − COD‑VAE) is summarized by the 2.5th and 97.5th percentiles; negative CD values indicate a favorable outcome for ZipTok3D.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers