HOMIE: Human-Object Centric Video Personalization via Multimodal Intelligent Enhancement

Yiyang Cai, Nan Chen, Rongchang Xie, Junwen Pan, Chunyang Jiang, Cheng Chen, Wen Zhou, Zhenbang Sun, Wei Xue, Wenhan Luo, Yike Guo

HOMIE integrates MLLM reasoning into video diffusion models to improve human-object interaction and subject fidelity.

Existing video personalization methods struggle to balance high subject fidelity with precise human-object interactions, especially when handling abstract concepts like logos or complex multi-view references. HOMIE addresses this by integrating a Multimodal Large Language Model (MLLM) directly into the video diffusion backbone, using global guidance to inject semantic knowledge without replacing the text encoder or requiring costly re-alignment. This unified approach achieves state-of-the-art performance in both inter-subject and intra-subject personalization tasks, significantly improving logo placement accuracy and multi-view consistency.

Paper Primer

The core mechanism hinges on two modules: Global Multimodal Guidance (GMG) and Modality-Reference Embedding (MRE). GMG performs a feature-wise affine transformation on video latents using pooled MLLM features, while MRE assigns learnable identity tags to tokens to distinguish between distinct subjects and associate intra-subject references.

HOMIE significantly improves textual fidelity in intra-subject personalization tasks.

OCR accuracy on the evaluation dataset.

The model effectively reasons about abstract concepts like logo placement without explicit prompt instructions.

Qualitative comparison of logo attachment to objects (e.g., COSTA logo on a paper cup). Outperforms Kling and UniVideo in rendering logo details.

Why integrate an MLLM instead of just using a stronger text encoder?

The authors argue that MLLMs provide richer visual and linguistic context for reasoning about inter- and intra-subject relationships, which standard text encoders lack. By keeping the original text encoder and using the MLLM only for reference relationship extraction, the model preserves its base controllability while gaining advanced reasoning capabilities.

What is the scope of this framework?

HOMIE is designed for Human-Object Centric Video Personalization (HOCVP), covering both inter-subject (multiple distinct subjects) and intra-subject (multi-view images or OCR maps of a single subject) scenarios.

HOMIE avoids the "copy-paste" artifacts common in multi-subject generation by using MRE to explicitly distinguish between modality types and identity-level references, preventing the model from treating reference images as redundant or independent entities.

Researchers can now achieve high-fidelity human-object interaction by augmenting existing diffusion transformers with MLLM-derived semantic guidance, bypassing the need for full-model re-alignment or 3D multi-view priors.

Abstract and Introduction

We expose the core limitations of current HOCVP methods across inter- and intra-subject settings.

Current human‑object centric video personalization methods falter when scaling to many subjects or when dealing with abstract objects, and they lack mechanisms to exploit intra‑subject cues such as OCR maps or multi‑view images.

Given reference images of people and objects, the model generates videos where those entities interact according to a textual prompt.

Method

HOMIE injects multimodal guidance into a DiT video model via two novel fusion mechanisms.

Existing video generators struggle to fuse rich visual references with textual prompts: naïve concatenation leaves a distribution gap and makes temporal control brittle.

HOMIE treats video latents, reference‑image latents, and high‑level multimodal features as separate token streams and injects the latter into the former through two lightweight, learnable fusion modules.

How does GMG differ from simply concatenating $F_{M}$ to the token sequence?

Concatenation would force the attention matrix to treat multimodal tokens like any other token, which can drown their signal and break temporal consistency. GMG first pools $F_{M}$ into a single global query/key, then applies a learned affine modulation only to the video‑token queries and keys, leaving reference‑token attention untouched.

The model receives three streams—raw video, reference images, and a textual prompt—and aligns them in a common latent space before any attention occurs.

GMG extracts a single global summary of the multimodal feature and uses it to scale and shift the video‑token queries and keys, thereby steering attention toward semantically relevant regions without disturbing reference‑token dynamics.

Pooling across the temporal axis averages the four rows, yielding $\tilde{Q}_{M}=[(0.5,0.5)]$.

The scaling network outputs $\gamma_{q}(\tilde{Q}_{M})=[0.2, -0.1]$ and the shift network outputs $\beta_{q}(\tilde{Q}_{M})=[0.05, 0.03]$.

Original video query matrix $Q_{v}$ (four tokens) is $[[0.1,0.2],[0.2,0.1],[0.0,0.0],[0.3,0.3]]$.

Applying the affine transform: each row is multiplied element‑wise by $(1+\gamma_{q})=[1.2,0.9]$ and then $\beta_{q}$ is added, producing $\tilde{Q}_{v}=[[0.17,0.21],[0.29,0.12],[0.05,0.03],[0.41,0.30]]$.

The modified queries now carry a subtle bias toward the global multimodal direction while retaining their token‑specific variation.

GMG’s affine modulation injects semantic guidance uniformly across all video tokens, avoiding the “dilution” problem that occurs when multimodal features are simply concatenated.

Why does GMG only modulate video queries/keys and not reference queries/keys?

Reference tokens are meant to preserve their original self‑attention relationships with the video latents; altering their queries/keys would corrupt the precise alignment that the reference embeddings encode. By leaving $\{Q_{r},K_{r}\}$ untouched, GMG ensures that reference information is used as a stable anchor while the video stream receives the semantic push.

MRE tags each token with a learnable vector that tells the model “what kind of token this is” and, for reference tokens, “which subject identity it belongs to”.

How is MRE different from simply adding a one‑hot modality flag?

A one‑hot flag provides only a binary indicator and cannot capture nuanced relationships (e.g., similarity between two reference subjects). MRE learns dense vectors that can encode both modality type and fine‑grained identity information, allowing the attention mechanism to treat “same‑subject reference” differently from “different‑subject reference”.

**Figure 3.** Overview of HOMIE. HOMIE employs a multimodal-input paradigm with video tokens, image tokens, and multimodal features. Prior to self-attention, Global Multimodal Guidance (GMG) fuses high-semantic MLLM features into video tokens, boosting cross-modal interaction. Furthermore, Multimodal-Reference Embedding (MRE) enables the model to discriminate cross-modal tokens and inter-subject tokens, as well as bind reference tokens from intra-subject references.

The authors curate a high‑quality, subject‑driven video corpus that supplies both inter‑ and intra‑subject reference pairs for training the multimodal modules.

Implementation Details

Implementation details describe the feature extraction, guidance, and staged training that enable efficient high‑resolution video generation.

The core challenge is to fuse multimodal cues from a large vision‑language model (MLLM) with a video diffusion backbone while keeping compute tractable and preserving fine‑grained identity control.

We pre‑extract a compact visual embedding from a powerful MLLM and feed it to the video model, letting the heavy vision processing happen once per training step.

How does this differ from replacing the text encoder with an MLLM, as done in UniVideo?

Replacing the encoder forces the entire control pipeline to be re‑trained, dramatically increasing GPU hours. Our approach keeps the original lightweight text encoder and only adds a frozen visual embedding, so only the multimodal branch learns new control signals.

**Table 1.** Training details for the three stages.

Why not simply add the multimodal embedding after the final DiT block?

Appending it only at the end would let early layers ignore visual cues, limiting the model’s ability to shape coarse‑to‑fine structure. By injecting at every layer, GMG lets low‑level texture formation and high‑level motion planning both benefit from the same guidance.

Stage 1: train on a single‑reference video dataset (300 K samples) at $480\times832$ resolution with learning rate $1\mathrm{e}{-5}$ for 3000 steps.

Stage 2: continue on a multiple‑reference dataset (same resolution) for 2000 steps, keeping the learning rate unchanged.

Stage 3: fine‑tune on a high‑resolution ($720\times1280$) multiple‑reference dataset for 500 steps with a reduced learning rate $2\mathrm{e}{-6}$.

Start at LR $1\mathrm{e}{-5}$ → after 3 k steps we switch to the multiple‑reference set.

Learning rate stays $1\mathrm{e}{-5}$ while steps drop to 2 k, reducing over‑fitting risk on the larger reference pool.

Finally LR is halved to $2\mathrm{e}{-6}$ and steps shrink to 0.5 k, allowing a brief high‑resolution fine‑tune without destabilising earlier weights.

This progressive reduction in both learning rate and step count concentrates capacity on the new resolution while preserving the representations learned in earlier stages.

In the first stage we expose the model to videos that contain a single, clearly identified subject, forcing it to learn a strong, identity‑preserving embedding.

Why not train on multiple subjects from the start?

Mixing many identities early confuses the model’s ability to form a stable subject prototype; the single‑subject phase isolates identity learning before introducing interaction complexity.

After identity is stable, we train on videos that contain several reference subjects in a fixed order, teaching the model to reason about interactions.

How does fixed ordering differ from the random ordering used in Stage 1?

Random ordering only teaches “who is present”. Fixed ordering adds a temporal or relational cue, enabling the model to learn directed interactions rather than unordered co‑occurrence.

We finish with a short fine‑tune on higher‑resolution video so the model can render crisp details without relearning earlier knowledge.

Why not train the whole pipeline at high resolution from the beginning?

High‑resolution video dramatically increases memory and compute; starting low lets the model learn semantics cheaply, then the brief high‑res fine‑tune transfers those semantics to a larger canvas.

**Table 6.** Comparison of training costs

Evaluation Details

Our method uniquely supports intra‑subject distinction while others do not.

Our method uniquely supports intra‑subject distinction, a capability absent in all prior SOTAs.

Table 7 shows a green checkmark only for our method in the “Intra‑subject Distinction” column.

**Table 7.** Details of backbones and technical implementations compared to previous SOTAs.

Additional Results

Additional qualitative comparisons highlighting HOMIE’s advantages over prior methods.

We extend the evaluation with a broad set of qualitative comparisons that stress‑test HOMIE’s ability to generate natural human‑object interactions and correctly place logos.

1. You are an expert multimodal AI assistant specialized in highly controllable visual content generation. 2. You will receive A set of images depicting specific humans and objects and a text description detailing the target patterns of human-object interaction. 3. Your task: analyze the precise relationships between the visual entities in the images and the interaction logic defined in the text prompt. Specifically, evaluate how the distinct human and object features from the visual inputs should be spatially, physically, and semantically integrated to accurately reconstruct the interaction described in the text. The output hidden states are kept at a maximum length of 1024 and a hidden state dimension of 2048

Qualitative examples span inter‑subject scenarios (e.g., a rabbit holding a handbag alongside a fox) and intra‑subject cases that leverage OCR maps and multi‑view inputs to preserve textual fidelity across viewpoints.

All video samples, including the scenes described above, are publicly available on the anonymous demo webpage for further inspection.

Experiments and Conclusion

We report HOMIE’s quantitative gains, ablations, and user‑study results confirming its superiority.

HOMIE attains the highest motion‑smoothness (MS) score among open‑source baselines.

Table 1 shows Ours (Wan2.2‑14B) reaches 0.997 MS, far above the next best open‑source method SkyReels‑V3 (0.272 MS).

Table 1 aggregates video quality, text‑following, and subject‑consistency metrics across 14 methods. HOMIE’s OCR accuracy surpasses all baselines by 21.8 % and its motion‑smoothness (MS) is the highest among open‑source approaches, while maintaining competitive scores on AES, GMEScore, Face‑Sim, DINO‑I, and Obj‑Sim.

Table 2 evaluates intra‑subject HOCVP with multi‑view references. The Ours (Wan2.2) variant achieves the top DINO‑rec score (0.582) and the best DINO‑acc (0.560), confirming its ability to faithfully reproduce multiple viewpoints.

Table 3 presents an ablation study. Removing the GMG module drops GMEScore to 0.688 and OCR Acc. to 0.508, while omitting MRE reduces GMEScore to 0.690 but improves OCR Acc. to 0.863, highlighting the complementary roles of GMG and MRE in the full model.

**Figure 2.** Framework design of MLLM-facilitated video personalization.

**Figure 4.** **Inter-subject** qualitative comparison between HOMIE and previous SOTA methods, including more subjects and abstract content personalization. Prompts that are relevant to subjects and interaction are highlighted. (Zoom in for the best view)

**Figure 5.** Intra-subject qualitative comparison between HOMIE and previous SOTA methods. Two representative intra-subject exemplars (OCR maps and multi-view references) are provided. Prompts that are relevant to subjects and interaction are highlighted. (Zoom in for the best view)

**Figure 6.** Qualitative Ablation Studies (Zoom in for the best view.)

**Figure 7.** User study results.

Training consumes roughly 11 K GPU‑hours on A100 GPUs, using a three‑stage tuning paradigm and a two‑layer MLP to align Qwen3‑VL‑2B‑Thinking features with 3D VAE tokens.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers