PIA: Personalized Image Animator 206

PIA is a personalized image animation method implemented through a plug-and-play module in a text-to-image model, enabling personalized image animators to generate videos with high motion controllability and strong text-image alignment, compatible with various personalized T2I models without requiring specific adjustments.

The proposed PIA animates them with realistic motions based on different text prompts while preserving their unique styles and high-fidelity details, achieving a perfect balance between appearance and motion, alleviating appearance-related image alignment challenges, and focusing on motion-related guidance alignment.

PIA project address: https://pi-animator.github.io/

Recent advances in personalized text-to-image (T2I) models have revolutionized content creation, enabling non-professionals to generate stunning images with unique styles. While promising, adding realistic motions to these personalized images through text poses significant challenges in preserving unique styles, high-fidelity details, and achieving motion controllability through text.

In this paper, we introduce PIA, a personalized image animator that excels at conditional image alignment, motion controllability through text, and compatibility with various personalized T2I models without requiring specific adjustments. To achieve these goals, PIA is built upon a base T2I model with trained temporal alignment layers, allowing any personalized T2I model to be seamlessly converted into an image animation model.

A key component of PIA is the introduction of a condition module, which utilizes conditional frames and inter-frame affinity as inputs, transferring appearance information under the guidance of affinity prompts for single-frame synthesis in the latent space. This design alleviates internal appearance-related image alignment challenges and allows for greater focus on motion-related guidance alignment.

PIA Implementation:

• Temporal alignment layer-based T2I model: trained to convert personalized T2I models into image animation models.

• Condition module: utilizes conditional frames and inter-frame affinity to transfer appearance information in the latent space under the guidance of affinity prompts, directing frame synthesis.

PIA Features:

• Plug-and-play, enabling animation based on text prompts.

• Preserves original styles with high-fidelity details without distortion.

PIA is contributed by the Shanghai Artificial Intelligence Laboratory.