ViViD: vídeo de prueba virtual con modelos de difusión 202

ViViD, a virtual try-on video technology developed by Alibaba, allows you to freely replace the clothing of a person in a video. ViViD can generate natural and realistic virtual try-on videos, transferring garments onto the person in the video while keeping the rest of the video unchanged.

ViViD demo website: https://becauseimbatman0.github.io/ViViD

ViViD paper address: https://arxiv.org/abs/2405.11794

With the rapid development of e-commerce, consumers can conveniently browse all kinds of clothing through online platforms such as Taobao Mall and JD Mall. However, relying solely on the clothing images on e-commerce websites, it is hard for consumers to imagine how the garment would look on themselves. Although image-based clothing transfer technology has become relatively mature, video clothing transfer—especially when it comes to interaction with body movements—remains rarely seen.

Virtual try-on video works by transferring garments onto a target person. Directly applying image-based try-on techniques to the video domain frame-by-frame leads to temporally inconsistent results, while previous video-based try-on solutions can only produce low visual quality and blurry outcomes.

ViViD, a video virtual try-on technology jointly developed by the University of Science and Technology of China and the Alibaba Group research teams, aims to address this issue through video synthesis, allowing consumers to see the realistic effect of garments worn on themselves. This video framework leverages Diffusion Models to achieve high-quality and consistent try-on videos.

Specifically, we design a garment encoder to extract fine-grained garment semantic features, guiding the model to capture garment details and injecting them into the target video via the introduced attention feature fusion mechanism. To ensure spatiotemporal consistency, we introduce a lightweight pose encoder to encode pose signals, enabling the model to learn the interaction between garments and human pose, and insert hierarchical temporal modules into a text-to-image stable diffusion model to obtain more coherent and realistic video synthesis results.

In addition, we have collected a new dataset, which is so far the largest, most garment-type-diverse, and highest-resolution dataset for the video virtual try-on task. Extensive experiments demonstrate that our method can produce satisfactory video try-on results. The dataset, code, and weights will be made publicly available.