ViViD: Virtual Try-On Video with Diffusion Models 204

ViViD, a video virtual try-on technology developed by Alibaba, can arbitrarily replace the clothing of the person in a video. ViViD is capable of generating natural and realistic virtual try-on videos, transferring garments onto the person in the video while keeping the rest of the video unchanged.

ViViD Demo Website: https://becauseimbatman0.github.io/ViViD

ViViD Paper Address: https://arxiv.org/abs/2405.11794

With the rapid development of e-commerce, consumers can conveniently access various types of clothing through online platforms such as Taobao and JD.com. However, relying solely on clothing images on e-commerce websites makes it difficult for consumers to imagine how the garment would look on themselves. While image-based try-on technology has become relatively mature, video-based try-on—especially when it comes to interaction with body movements—is still rarely seen.

Virtual try-on video involves transferring a garment onto a target person. Applying image-based try-on techniques directly to video in a frame-by-frame manner leads to temporally inconsistent results, while previous video-based try-on solutions could only produce outputs with low visual quality and blurriness.

ViViD, a video virtual try-on technology jointly developed by a research team from the University of Science and Technology of China and Alibaba Group, aims to address this issue by synthesizing videos, allowing consumers to see the realistic effect of clothing worn on themselves. This video framework leverages diffusion models to achieve high-quality and consistent try-on videos.

Specifically, we designed a garment encoder to extract fine-grained garment semantic features, guiding the model to capture garment details and injecting them into the target video through the proposed attention feature fusion mechanism. To ensure spatiotemporal consistency, we introduced a lightweight pose encoder to encode pose signals, enabling the model to learn the interaction between garments and human poses, and inserted hierarchical temporal modules into the text-to-image stable diffusion model to achieve more coherent and realistic video synthesis results.

In addition, we collected a new dataset, which is to date the largest, most diverse in garment types, and highest-resolution dataset for the video virtual try-on task. Extensive experiments show that our method can produce satisfying video try-on results. The dataset, code, and weights will be made publicly available.