CreatiDesign: A Unified Multi-Condition Diffusion Transformer for Creative Graphic Design 90

小编 · 2025-06-12

CreatiDesign is a high-precision, multimodal, editable graphic design generation model developed jointly by Fudan University and ByteDance. It generates graphic designs with high precision while handling multiple elements—including hero visuals, decorative accents, and text—addressing the shortcomings of traditional methods when dealing with multi-condition inputs. Through fine-grained decoupling and independent control, CreatiDesign makes design more controllable and consistent.

CreatiDesign: A Unified Multi-Condition Diffusion Transformer for Creative Graphic Design

CreatiDesign is a high-precision, multimodal, editable graphic design generation model jointly developed by Fudan University and the ByteDance team. It can generate graphic designs with high precision while handling multiple elements, including primary visual elements, auxiliary decorations, and text content, addressing the shortcomings of traditional approaches when dealing with multi-conditional inputs. Through fine-grained decoupling and independent control, CreatiDesign makes design more controllable and consistent.

This model adopts an innovative multimodal attention mechanism and can also automatically synthesize data, thereby surpassing the existing state-of-the-art across multiple evaluation criteria. It not only improves design efficiency but also allows users to flexibly modify existing designs, such as adding new elements or adjusting text, while maintaining the overall coherence and integrity of the design.

CreatiDesign Model: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Design.webp

Model Architecture

CreatiDesign adopts the MM-DiT (Multimodal Diffusion Transformer) framework, achieving collaborative control of various different conditions through a simplified architecture. The main features are as follows:

Multi-subject image conditions: Users can input multiple primary visual elements (such as product images, LOGOs, etc.) and freely arrange them. These elements are processed by an encoder to generate primary visual tokens for subsequent generation.
Semantic layout conditions: Semantic descriptions of auxiliary elements or text are converted into semantic feature tokens by a T5 text encoder, while spatial position information is processed via Fourier transform and concatenated with the semantic features to form layout tokens.
Global description: Users can also input an overall description, which is encoded by T5 into a global description token to guide the overall content and style.
Multimodal attention mechanism: In each Transformer layer, a multimodal attention (MM-Attention) mechanism is employed to deeply fuse tokens from different modalities, enabling joint modeling and control of multiple conditions.

Independent Controllability

CreatiDesign proposes two dedicated attention mask mechanisms to enhance the independent controllability of each condition:

Subject Attention Mask: Subject tokens only interact with image tokens within designated regions and are completely isolated from layout tokens, global description tokens, and image tokens in unrelated regions, ensuring high-fidelity reproduction of subject content.
Layout Attention Mask: Each layout token only interacts with image tokens within its designated region, preventing semantic interference between layout elements.

Dataset

CreatiDesign constructed a large-scale multi-conditional annotated dataset containing 400,000 samples to meet real-world design requirements. In addition, a fully automated graphic design data synthesis pipeline was proposed, including subject generation, text layer rendering, and foreground-based image generation.

Application Scenarios

CreatiDesign can generate various graphic designs, such as movie posters, brand promotions, product advertisements, and social media graphics. It also supports multi-turn editing, allowing users to flexibly insert new text or subjects into existing design results, or modify text content. The model can precisely edit designated regions while keeping non-edited regions unchanged.

Advantages

Compared with previous controllable diffusion models, CreatiDesign excels in unified modeling of multi-conditional heterogeneous information, fine-grained decoupling and region-independent control of multiple conditions, as well as large-scale high-quality data support. It addresses the issues of "imbalanced capabilities" or lack of precise controllability that existed in previous models when handling graphic design.