4DAnyone: Create Anyone in 4D from a Casual Monocular Video

Yudong Jin, Tao Xie, Qihang Zhang, Zehong Shen, Zhen Xu, Yujun Shen, Hujun Bao, Xiaowei Zhou, Yinghao Xu

4DAnyone generates reconstruction-grade multiview videos from monocular input by using skeleton-conditioned diffusion with scalable context management.

How can we reconstruct a 4D human avatar from a single uncalibrated monocular video by generating multiview-consistent video frames?

Reconstructing 4D human avatars from casual monocular video is difficult because existing video diffusion models struggle to maintain cross-view consistency when scaled to the tens of target views required for high-fidelity 3D reconstruction. The authors introduce 4DAnyone, which uses sparse 3D skeletons for geometric guidance and two architectural innovations: Reference Context Packing (RCP) to compress appearance references into fixed-length tokens, and Target Context Routing (TCR) to rotate view groupings during denoising for global structural coherence. This framework outperforms prior methods in both novel-view video quality and downstream 4D Gaussian Splatting (4DGS) reconstruction, enabling high-fidelity 4D avatar creation from uncalibrated monocular input.

Paper Primer

The core challenge is a bounded-attention-context bottleneck: when generating many target views, the model must split them into groups, which causes appearance guidance to weaken as references grow and structural drift to emerge because disjoint groups cannot exchange information. 4DAnyone solves this by treating the generation process as a scalable, geometry-aware routing problem.

4DAnyone is a skeleton-conditioned diffusion framework: it uses depth-buffered 3D skeletons to provide precise structural guidance, then employs RCP to compress reference views into an $O(1)$ context and TCR to dynamically rotate target-view groupings during high-noise denoising steps. Think of TCR like a rotating relay: by shifting which views are grouped together at each step, the model propagates global structure across the entire set of target views before locking in local details.

4DAnyone achieves superior multiview consistency and 4DGS reconstruction quality compared to implicit camera-conditioned and dense-geometry baselines.

Quantitative evaluation on DNA-Rendering and DyMVHumans datasets shows consistent gains in PSNR, SSIM, and LPIPS across both generated video consistency and downstream 4DGS rendering metrics. The method enables reconstruction-grade 4D human avatars from monocular video without requiring specialized multi-camera hardware or dense metric depth estimation.

Why use 3D skeletons instead of dense depth maps for geometric conditioning?

Dense depth is difficult to estimate reliably from in-the-wild monocular video and often introduces conflicting geometric constraints. 3D skeletons provide sparse but robust structural cues that modern human mesh recovery methods can extract reliably, leaving appearance details to be learned by the video model.

What is the specific role of the switching timestep $t_s$ in Target Context Routing?

The switching timestep defines the transition from a high-noise phase, where TCR rotates view groupings to propagate global structure, to a low-noise phase, where it fixes adjacent groups to stabilize fine appearance details.

4DAnyone demonstrates that reconstruction-grade 4D human synthesis is achievable from monocular input by prioritizing geometric accuracy over dense depth and managing attention context through explicit routing and compression.

Introduction and Motivation

We frame the challenge of reconstructing 4D humans from monocular video.

Reconstructing 4D humans from a single, uncalibrated video is attractive for embodied AI, immersive content creation, and virtual reality, yet existing pipelines demand dense multiview captures from calibrated camera rigs.

We need a way to turn a casually captured video into a volumetric 4D representation without expensive camera setups.

4DGS represents a dynamic scene as a collection of anisotropic Gaussians that can be rendered in real time.

Camera‑controlled video diffusion models can synthesize plausible novel‑view videos, but they break down when the number of target views $N$ grows to the tens required for 4DGS reconstruction.

The failure stems from a bounded‑attention‑context problem: reference‑context length scales as $O(N)$, exhausting memory, while disjoint target‑context groups cannot exchange information, causing structural drift.

**Fig. 1.** Given a casually captured monocular video, typically with mild camera motion and unknown camera intrinsics and poses, 4DAnyone generates multiview-consistent human videos, enabling 4DGS reconstruction rendered from free viewpoints. See the project page and Fig. 7 for more results.

The shift from monocular input to multiview‑consistent generation is the key enabler for reconstruction‑grade 4D modeling.

The 4DAnyone Framework

We describe how 4DAnyone builds consistent multiview videos using skeleton conditioning, reference packing, and target routing.

Generating dozens of consistent target videos from a single monocular clip is costly, and naïve joint denoising quickly exceeds memory limits.

**Figure 2.** **Overview of 4DAnyone.** Given a source video, an HMR model (GVHMR [Shen et al. 2024]) estimates a 3D skeleton sequence, which is rendered into depth-buffered skeleton videos for $v$ target views. A 3D-aware skeleton encoder produces residual skeleton tokens that are added to the noisy target latents. The source tokens, RCP reference tokens, and skeleton-conditioned target tokens are concatenated along the view dimension and processed by the DiT with video attention, multiview attention, and text cross-attention. Generated target videos are packed back into the RCP context for subsequent generation rounds, while TCR improves consistency during grouped target-view generation. The final multi-view videos are used to train a 4DGS model with FreeTimeGS [Wang et al. 2025a].

We give the diffusion model a sparse but reliable pose cue by rendering a depth‑aware 3D skeleton and injecting it as a residual into the latent tokens.

Assign depths: head $z=2$, left hand $z=1$, right hand $z=3$ (larger $z$ is farther).

Rasterize each joint as a single pixel; the pixel with smallest $z$ wins the occlusion test.

Resulting depth map shows the left hand (closest) covering the head pixel, while the right hand remains visible.

Feed the three‑pixel image $S_i$ into $g_{\phi}$, which outputs a $64$‑dimensional residual added to $z_i^{t}$.

The example shows that even a minimal skeleton resolves front‑back ambiguity, which a plain 2‑D keypoint map could not.

Why not use dense depth maps instead of a sparse skeleton?

Dense depth requires accurate camera intrinsics and suffers from noisy per‑pixel estimates in wild videos; a sparse skeleton sidesteps these issues while still providing enough structural guidance for multiview consistency.

RCP compresses all previously generated reference videos into a fixed‑size token bank, so the DiT sees a constant amount of appearance context regardless of how many references exist.

Apply $P_1$ (no compression) to the source, yielding $64$ tokens.

Apply $P_2$ (kernel/stride $(1,4,4)$) to each reference, reducing each to $4$ tokens.

Apply $P_4$ (kernel/stride $(1,8,8)$) to a third reference, reducing it to $1$ token.

Concatenate: $64 + 2\times4 + 1 = 73$ tokens total, independent of how many additional references we might later add.

Even as we generate more reference videos, the token count grows only by the fixed $P_2$ and $P_4$ slots, guaranteeing $\mathcal{O}(1)$ memory growth.

How does RCP differ from simply appending all previous reference frames as extra tokens?

Appending raw frames would increase token count linearly with the number of references, quickly exhausting GPU memory. RCP compresses each reference with a stride‑scaled patchify, keeping the total token budget constant while still preserving coarse appearance cues.

TCR reshuffles target‑view groups during high‑noise steps so that global structure can flow across groups, then freezes adjacent groups in low‑noise steps to refine details without drifting.

Step 9 ($t>t_s$): rotate indices by 1 → ordering [2,3,4,5,6,7,8,1]; groups = {2,3,4,5} and {6,7,8,1}.

Step 8: rotate by 2 → ordering [3,4,5,6,7,8,1,2]; groups = {3,4,5,6} and {7,8,1,2}.

Step 5 ($t\le t_s$): fix groups as originally defined → {1,2,3,4} and {5,6,7,8} for the remaining steps.

Within each fixed group, denoise jointly, sharing the same $C_R$ and skeleton conditions.

The example shows that early rotations let information cross the whole set of views, while later fixation locks neighboring views together for coherent detail refinement.

Why not keep the same grouping throughout the entire diffusion process?

Static groups would never exchange global pose information, so early high‑noise steps could lock each group into a different inconsistent body configuration, leading to visible drift across views.

**Fig. 3.** Progressive inference with Reference Context Packing and Target Context Routing. We first generate reference videos and pack them with the source video into a fixed RCP context. During target-view generation, this fixed RCP context is shared by all groups. TCR partitions $v$ target views into $m$ four-view groups, cyclically regroupes them at high noise levels to propagate global structure, and fixes adjacent groups at low noise levels for stable detail refinement.

**Algorithm 1** Target Context Routing **Require:** Target viewpoints $\mathcal{V}$, fixed RCP context $C_R$ built from the generated reference views, switching timestep $t_s$, denoising schedule $\{t_T, \dots, t_0\}$, skeleton conditions $\{S_i\}_{i \in \mathcal{V}}$ 1: Initialize $z_i^T \sim \mathcal{N}(0, \mathbf{I})$ for all views $i \in \mathcal{V}$ 2: **for** $n = T, T-1, \dots, 1$ **do** 3: $\quad$ **if** $t_n > t_s$ **then** $\quad \triangleright$ High-noise: rotate groups 4: $\quad \quad \mathcal{G} \leftarrow \text{RotatingGroups}(\mathcal{V}, 4, n)$ 5: $\quad$ **else** $\quad \triangleright$ Low-noise: fix adjacent groups 6: $\quad \quad \mathcal{G} \leftarrow \text{AdjacentGroups}(\mathcal{V}, 4)$ 7: $\quad$ **end if** 8: $\quad$ **for** each group $G \in \mathcal{G}$ **do** 9: $\quad \quad$ Jointly denoise $\{z_i^n\}_{i \in G}$ conditioned on $C_R$ and $\{S_i\}_{i \in G}$ to obtain $\{z_i^{n-1}\}_{i \in G}$ 10: $\quad$ **end for** 11: **end for** 12: Decode $\{z_i^0\}_{i \in \mathcal{V}}$ into target-view videos $\{V_i\}_{i \in \mathcal{V}}$ 13: **return** $\{V_i\}_{i \in \mathcal{V}}$

Training proceeds in three stages: (1) skeleton‑conditioned foreground videos, (2) full‑scene multiview videos, and (3) in‑the‑wild monocular clips, each adding progressively harder appearance and pose variations.

We combine a standard latent flow‑matching loss with a perceptual LPIPS term to balance pixel‑level fidelity and high‑level visual quality.

RCP and TCR together give a constant‑budget, globally consistent multiview generation pipeline.

Results and Evaluation

4DAnyone beats prior methods in video consistency and 4DGS reconstruction quality.

We train on the multi‑view datasets MVGameHuman, SynCamVideo, DNA‑Rendering and on the monocular sets Pexels and TedTalk, detecting 2D keypoints with Sapiens2‑1B and building 4DAnyone on top of the 5 B‑parameter Wan2.2‑TI2V‑5B diffusion model. Training follows the three‑stage curriculum (≈0.5 d, 1 d, 1.5 d) at 704×1280 resolution on 128 H20‑3E GPUs; inference uses 20 denoising steps and partitions target views into four‑view groups (tₛ / T = 0.2).

Sliding view grouping in Target Context Routing yields the highest video‑consistency score among all ablations.

Table 3 shows the “Full (Sliding)” configuration achieving the best PSNR.

Quantitative results (Table 2) confirm that 4DAnyone surpasses MV‑Performer, TrajectoryCrafter, and ReCamMaster† on both generated‑video consistency and 4DGS reconstruction across DNA‑Rendering and DyMVHumans. Qualitatively, the method preserves geometry and appearance even for unseen back‑view content, whereas baselines exhibit distortions or misaligned camera control.

**Figure 8.** Representative MVGameHuman samples. Each row shows one frame from four evenly spaced cameras in a synchronized 24-camera sequence captured with our in-house game data engine. MVGameHuman provides diverse actors, clothing, motions, lighting conditions, virtual scenes, and backgrounds for training multi-view human video generation.

**Fig. 9.** **Single image to 4D avatar.** Given a single input image, we first generate a source video via pose-driven video generation (Wan-Animate), then produce multi-view target videos with 4DAnyone, and finally reconstruct a 4DGS avatar via FreeTimeGS.

4DAnyone outperforms baselines in both video consistency and 4DGS reconstruction quality.

Ablations and Failure Modes

We evaluate how each component of the pipeline affects reconstruction quality and speed.

Recall that 4DAnyone builds a 4‑D human by first generating multi‑view videos from a monocular input, then lifting them into a 4D Gaussian Splatting model.

GVHMR + sparse vertex‑to‑keypoint regressor achieves a mean error of 3.5 mm, far better than the 14 mm error of a nearest‑vertex baseline.

Measured on held‑out DNA‑Rendering scenes.

**Table 6.** Multi-GPU inference configurations. Configurations are shown for different camera setups.

The full 4DAnyone pipeline finishes in roughly 39 minutes on a single RTX 4090.

Stage 1 (GVHMR + skeleton rendering) ≈ 2 min, Stage 2 (video generation) ≈ 7 min, Stage 3 (FreeTimeGS training) ≈ 30 min.

**Table 7.** **TCR switching-time sweep.** Gen. Video Consistency when varying the number of sliding denoising steps.

Using $t_s/T=0.2$ (four sliding steps) yields PSNR ≈ 22.63 dB, the point where consistency gains saturate.

Table 7 entry for $t_s/T=0.20$.

**Fig. 10.** **Failure cases.** Left: skeleton guidance is uninformative for the large flowing fabric, which is generated inconsistently across views and yields a degraded reconstruction (red box). Right: HMR mis-estimates the en-pointe pose (green circle in the source view) as flat feet, and all generated views inherit the pose error (red box).

Removing either Reference Context Packing or Target Context Routing noticeably harms consistency, confirming that explicit skeleton‑geometry conditioning is essential for high‑quality 4D reconstruction.

Related Work

We situate 4DAnyone among prior camera‑controlled diffusion and 4D reconstruction approaches.

Camera‑controlled video generation splits into implicit and explicit conditioning approaches. Implicit methods such as CameraCtrl, MotionCtrl, VD3D, CamCo, ReCamMaster, and CAT4D embed camera cues in learned representations but lack hard geometric constraints, leading to drift under large viewpoint changes.

Explicit conditioning methods like Gen3C, TrajectoryCrafter, and WVD incorporate dense 3D geometry to improve accuracy, yet they depend on reliable depth or coordinate estimates that are hard to obtain from in‑the‑wild videos, especially for dynamic scenes.

Pose‑driven animation techniques (ControlNet, MagicAnimate, Animate Anyone, UniAnimate, Wan‑Animate, and 3DiMo) use 2D or implicit motion cues for motion transfer, but they target animation rather than reconstruction‑grade novel view synthesis.

4DAnyone leverages sparse, precise 3D skeletons as explicit conditioning, achieving reconstruction‑grade multiview consistency without estimating dense depth or source‑camera parameters.

Multi‑view diffusion models generate geometrically consistent views for downstream reconstruction. Early work like Zero‑1‑to‑3 introduced viewpoint‑conditioned image generation, while later systems (MVDream, CAT3D, SV3D, Zero123++, SV4D, CAT4D, Diffuman4D) extend the paradigm to dynamic content, yet they face scaling bottlenecks such as GPU memory limits and cross‑window drift.

Traditional 4D human avatar reconstruction relies on dense multi‑camera captures and neural representations such as radiance fields or Gaussian splatting, which demand expensive rigs, whereas monocular methods using parametric body priors struggle to hallucinate unseen appearance, limiting visual quality.

Recent approaches like UP2You and MV‑Performer improve static or reconstruction‑only scenarios but do not combine generative hallucination with dynamic view synthesis. In contrast, 4DAnyone first synthesizes multiview observations via skeleton‑conditioned diffusion and then applies standard 4DGS pipelines for high‑fidelity 4D human reconstruction without specialized hardware.

Training and Implementation Details

Implementation specifics for the model, datasets, and training pipeline.

Multiview self‑attention reorders tokens to shape $(f, v \cdot h \cdot w, d)$ so that tokens from different viewpoints at the same timestep can attend to each other directly. The layer shares architecture and weights with the temporal self‑attention in the base Wan2.2 DiT, inheriting pretrained temporal coherence as a starting point for cross‑view consistency.

The standard patchify layer uses a $1\\times2\\times2$ kernel/stride; the RCP 2× and 4× variants expand the spatial kernel to $1\\times4\\times4$ and $1\\times8\\times8$, yielding respectively one‑quarter and one‑sixteenth the token count. They are initialized by tiling the pretrained kernel and dividing by the area ratio (4 for 2×, 16 for 4×) to preserve activation variance.

The skeleton encoder $g_{\\phi}$ consumes a depth‑buffered RGB skeleton video and produces a DiT‑resolution residual added to noisy latent tokens. It consists of ten $\\text{Conv3d}$ layers (five strided, five non‑strided) with SiLU activations, followed by a $1\\times1\\times1$ projection that is zero‑initialized so the residual starts at zero.

MVGameHuman provides 38 k synchronized multi‑view human videos at $2560\\times1440$ resolution, captured by 24 virtual cameras per sequence and covering 318 actors. The dataset exhibits diverse clothing, motion, lighting, and background conditions, illustrating the multi‑view coverage required for training.

All three fine‑tuning stages use Wan2.2‑TI2V‑5B at $704\\times1280$ resolution with a learning rate of $1\\times10^{-5}$ and an LPIPS weight $\\lambda=0.25$. The stages take roughly 0.5, 1, and 1.5 days on 128 H20‑3E GPUs, and training remains stable with 32 or more GPUs.

To reduce memory use, a body‑part‑aware sampler extracts $256\\times256$ crops focusing on the full body, face, and hands, with sampling probabilities 0.2, 0.2, 0.1, and 0.1 respectively; the remaining 0.4 is uniform. Face and hand crops are centered on detected boxes, while full‑body crops are centered on a random point within the body box.

**Table 4.** Training data sampling. Source cameras are uniformly sampled from the listed options.

**Table 5. Stage-specific training settings.** "Indep. Src Prob" denotes the probability of independently sampling the source frame range. Skeleton: B=body, H=hands, F=feet, Fi=fingers.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers