Magi: Automatically Transcribe Manga into Text and Generate Scripts 111

The Magi model can automatically transcribe comics into text and generate scripts. It can also reorder and associate content by recognizing panels, text blocks, and characters on comic pages, outputting coherent dialogue and accurately generating text transcripts for each comic, thereby ensuring the coherence of the output text and making the entire process more efficient. The Magi model was developed by the Visual Geometry Group in the Department of Engineering Science at the University of Oxford.

Magi is suitable for researchers, developers, or comic enthusiasts who need to automatically generate text from comic images. It provides these users with convenient and fast tools to help them easily digitize comic content and indexing.

Magi GitHub source code: https://github.com/ragavsachdeva/magi

Magi paper: https://arxiv.org/abs/2401.10224

Main features of the Magi model:

• Panel detection: Accurately identifies individual panels on a comic page — the independent rectangular frames drawn by the artist — which is the first step in understanding the layout and content of a comic page.

• Text detection: Locates text regions within each panel and extracts text blocks containing important dialogue or narration, which forms the basis for understanding panel content.

• Character detection: Detects the characters appearing within each panel, obtaining visual information such as their bounding box, pose, and expression, which is crucial for subsequent analysis.

• Character recognition: Based on the detection results, identifies the identity information of each character, such as name and title, which helps track characters throughout the story.

• Character clustering: Groups images of the same character in different poses together based on facial features, clothing features, and other information, thereby distinguishing different characters on a page.

• Speech assignment: Determines which character on the page a text block belongs to and correctly assigns speech to the corresponding character, which is essential for generating accurate scripts.

• Reading order: Reorders and links scattered text blocks based on panel position and reading order, outputting coherent dialogue content, thereby ensuring the logical coherence of the output text.

How to use Magi?

1. Upload a comic image.

2. The system automatically recognizes and extracts relevant information from the comic image to generate text.

3. Users edit and modify the text as needed.

Through these features, Magi can automatically convert comic pages into detailed text scripts, solving the problem of automatic understanding and script generation for comic pages. It includes character dialogue along with corresponding action or situational descriptions, making the digitization of comic content easy and fast, allowing readers to fully experience a comic story through reading the text.