PortraitGen: A Multimodal Tool for Portrait Video Editing 228

小编 · 2024-09-24

PortraitGen is an innovative AI portrait video editing tool from the University of Science and Technology of China. It can elevate 2D portrait videos to a 4D effect and supports multimodal editing, allowing changes to a character's clothing, expression, and lighting environment in as little as 30 minutes. PortraitGen also features a face-aware editing module for smoother expression edits. Users can easily achieve text-based character modifications by providing input video and reference images.

PortraitGen: A Multimodal Tool for Portrait Video Editing

PortraitGen is an innovative AI portrait video editing tool developed by the University of Science and Technology of China. It can upgrade 2D portrait videos to 4D effects. It supports multimodal editing and can change the clothing, expressions, and lighting environment of characters in videos within just 30 minutes. PortraitGen also features a face-aware editing module that enhances the smoothness of expression editing. Users can easily achieve text-modified character effects by inputting videos and reference images.

Main Features of PortraitGen:

Rapid Editing: PortraitGen can complete video editing in just 30 minutes, with edited 3D portraits supporting smooth playback at 100 frames per second.
Multimodal Input: Users can edit through text, image, or 3D model input, supporting various editing methods, including text-driven and image-driven stylized editing.
Face-Aware Editing Module: This module is designed to improve expression quality while preserving personalized facial features, resulting in more natural and refined editing effects.
Lighting Adjustment: Users can adjust the lighting of video frames based on provided lighting descriptions, making the entire video more harmonious and visually appealing.
Technical Advantages: PortraitGen adopts neural Gaussian texture technology, which differs from the traditional spherical harmonics coefficient method. It can store learnable features for each Gaussian point, thereby achieving richer editing effects.
High Frame Rate: Edited videos support smooth playback at 100 frames per second.
Clothing Change Effect: Achieved through style transfer and virtual try-on.

Technical Details of PortraitGen:

SMPL-X Coefficients: Used for monocular video tracking.
Neural Gaussian Textures: Constructing 3D Gaussian feature fields.
Face-Aware Editing: Improving expression quality while preserving personalized facial structures.
Style Transfer: Used for image-driven editing.
InstructPix2Pix: Used for text-driven editing.

Use Cases of PortraitGen:

Film and Video Production: Quickly generate high-quality portrait videos.
Advertising Industry: Flexibly transform video styles.
Social Media: Personalized video content creation.
Education and Training: Produce instructional videos.
Virtual Try-On: Applications in the fashion industry.

PortraitGen is an advanced AI portrait video editing tool developed by the research team at the University of Science and Technology of China. It was released on Arxiv on September 23, 2024. PortraitGen modifies the appearance of characters in videos through text commands or reference images, achieving multimodal editing functions including but not limited to clothing changes, lighting adjustments, and altering character features.

How to Access PortraitGen:

Project Page: https://ustc3dv.github.io/PortraitGen

Github Page: https://github.com/USTC3DV/PortraitGen-code

Paper Page: https://arxiv.org/abs/2409.13591