RelightVid: Relight Videos Based on User-Provided Text Descriptions 218

What is RelightVid?

RelightVid is a video relighting technology—the video version of IC Light. It can adjust the lighting of a video based on conditions such as user-input text descriptions, background videos, or HDR environment maps, allowing the video to present realistic effects under different lighting conditions. Through a specialized model architecture and training approach, it achieves high-quality video relighting while ensuring coherence between video frames, avoiding flickering or jumps.

RelightVid Features

• Text-conditioned relighting: Users input text descriptions such as "sunlight filtering through leaves, creating dappled shadows," and RelightVid can simulate the corresponding video effect under those lighting conditions.

• Background-video-conditioned relighting: Uses a background video as the lighting condition, dynamically adjusting the lighting of the foreground object so that the foreground and background lighting remain consistent.

• HDR environment map-conditioned relighting: Uses HDR environment maps to precisely control lighting and achieve high-quality relighting results.

• Full-scene relighting: Relights both the foreground and background simultaneously, making the entire scene match the lighting condition.

• Foreground-preserving relighting: Changes the lighting of the foreground object while keeping the background unchanged—suitable for scenes where the foreground object needs to stand out.

How RelightVid Works

• Diffusion model extension: Built on a pre-trained image relighting diffusion model (such as IC-Light), the architecture is extended to support video input, with temporal layers introduced to capture inter-frame temporal dependencies and ensure temporal consistency in relighting.

• Multi-modal conditional joint training: Background videos, text prompts, and HDR environment maps can be used simultaneously as conditions. Encoders embed these conditions into the model, enabling cooperative editing through a cross-attention mechanism.

• Illumination-Invariant Ensemble (IIE): Enhances the input video in terms of brightness and averages the predicted noise, improving the model's robustness under different lighting conditions and preventing changes to the albedo.

• Data augmentation pipeline (LightAtlas): Generates high-quality relighting data pairs based on real videos and 3D rendering data, providing the model with rich lighting prior knowledge and enhancing its adaptability to complex lighting scenarios.

Strengths and Limitations

Strengths:

• Temporal consistency: By introducing temporal layers, RelightVid effectively ensures temporal consistency in video relighting, avoiding issues such as flickering or jumps.

• Multi-modal conditional control: Supports control through multiple modalities, allowing users to flexibly adjust lighting effects as needed.

• High-quality relighting results: Produces high-quality relighting effects through multiple techniques, making videos more realistic.

Limitations:

• Computing resource requirements: Being based on deep learning models, it requires substantial computing resources to run, limiting its use on low-end devices.

• Handling of complex scenes: When processing very complex scenes, lighting results may become inaccurate or unnatural.

• User knowledge requirements: Users need a certain level of video editing and lighting knowledge to take full advantage of its features.

How to Use RelightVid

• You can directly access RelightVid's online demo: https://huggingface.co/spaces/aleafy/RelightVid.

• Upload a video on the demo page and input the conditions as needed—such as a text description, a background video, or an HDR environment map—to perform relighting.

FAQ

1. What is the core difference between RelightVid and traditional video relighting methods?

A: Traditional methods are often built by directly extending image editing models, lacking modeling of video temporal coherence, and have difficulty fusing multi-modal inputs (such as text or dynamic lighting). RelightVid explicitly captures inter-frame lighting dependencies through trainable temporal layers and a lighting cross-attention mechanism, and supports multi-condition inputs such as text, background video, and HDR maps, enabling more natural dynamic lighting editing.

2. How is the balance between realism and diversity achieved in dataset construction?

A: This is achieved through dual-source data augmentation:

Wild videos provide real lighting scenes (such as kitchen window light, rainy-night street lamps) to ensure realism;

3D rendering data generates diverse lighting combinations (such as lava fire light, softbox soft light) through random HDR maps and camera trajectories, enhancing the model's generalization ability.

Combining the two makes the dataset contain both real cases and extreme lighting conditions that traditional methods find hard to obtain.

3. In the technical implementation, how is temporal consistency of the relighting results ensured?

A: Through a dual mechanism of dynamic HDR condition input and temporal layers:

• Dynamic HDR maps serve as time-series inputs, constraining smooth variation of lighting parameters (such as intensity, color) across each frame;

• Temporal layers introduce inter-frame feature interaction in the diffusion model's U-Net, preventing abrupt lighting changes between adjacent frames.

• In addition, the lighting cross-attention mechanism further propagates lighting features across frames, strengthening global consistency.

Project Links

Project website: https://aleafy.github.io/relightvid/

GitHub repository: https://github.com/Aleafy/RelightVid

arXiv paper: https://arxiv.org/pdf/2501.16330

Online demo: https://huggingface.co/spaces/aleafy/RelightVid