Light-A-Video: Adjust Video Lighting Without Retraining 146

Light-A-Video is a simple application built on a per-frame image relighting model, primarily addressing issues of light source consistency and relighting appearance. The project's goal is to enhance the naturalness of lighting effects in video, making the generated video frames more consistent in terms of light sources and appearance, and solving problems faced by traditional video relighting methods such as high training costs, data scarcity, poor temporal consistency, and limited application domains.

Light-A-Video Features:

No Training Required: Light-A-Video adopts an unsupervised learning approach, eliminating the need for large-scale datasets or complex training processes.

CLA Module: Through a dual-stream attention mechanism, it combines the high-frequency details of the original frames with the stable background lighting averaged across frames, ensuring the stability of the light source and reducing flickering.

PLF Strategy: This strategy progressively fuses the relighting effect, ensuring consistency and smooth transitions in lighting during the video denoising process, thereby avoiding unnatural effects caused by per-frame processing.

Broad Compatibility: This method is compatible with multiple video generation models, such as UNet and DiT, demonstrating its versatility across different application scenarios.

Lighting Consistency: Lighting consistency in videos is enhanced by introducing two core modules:

• Consistent Light Attention (CLA): Ensures consistency of background light sources by enhancing cross-frame interaction in the self-attention mechanism.

• Progressive Light Fusion (PLF): Leverages the principle of independence in light transport to smooth lighting transitions through linear blending, ensuring lighting consistency between video frames.

Technical Workflow:

• Video Encoding and Noise Injection: The input video is encoded into the latent space, and noise is added to prepare for subsequent processing.

• Per-Frame Relighting: Each frame is processed using a pre-trained image relighting model.

• CLA Module Application: Lighting effects are stabilized by enhancing cross-frame interaction, maintaining consistency across video frames.

• PLF Strategy Implementation: The relighting effect is progressively integrated into the video denoising process, ensuring smooth temporal transitions.

• Video Decoding: The final relit video is generated, maintaining high quality and temporal coherence.

Light-A-Video Application Scenarios:

Light-A-Video provides more efficient and flexible video lighting adjustment tools for fields such as video editing, film special effects production, virtual reality, advertising, and game development:

• Full Video Relighting: Suitable for relighting an entire video.

• Video Foreground Relighting: Combines background generation to relight the video foreground.

Light-A-Video was jointly developed by researchers from Shanghai Jiao Tong University, University of Science and Technology of China, The Chinese University of Hong Kong, The Hong Kong University of Science and Technology, Stanford University, and other institutions. It is the first model capable of achieving high-quality, temporally coherent video relighting without pre-training.