SignLLM: The First Multilingual Sign Language Generation Model 189

SignLLM, the first multilingual sign language generation model, can convert input text or prompts into corresponding sign language gestures in video form. SignLLM generates and ai-translates multiple sign languages through advanced technology. SignLLM introduces the first multilingual sign language dataset, named Prompt2Sign, and develops various sign language generation models based on this dataset. This dataset transforms a large amount of video into a format convenient for model training, optimizing the training of translation models such as seq2seq and text2text. Based on these datasets, we present the benchmark results of SignLLM, showing that our model achieves state-of-the-art performance in sign language generation tasks for eight sign languages.

Inspired by the powerful translation capabilities of large language models (LLMs) trained on extensive multilingual text corpora, SignLLM aims to leverage off-the-shelf LLMs to handle SLT. In this paper, we normalize sign language videos to reflect the linguistic features of spoken language, and propose a novel SignLLM framework that converts sign language videos into language-like representations to improve the readability of off-the-shelf LLMs.

SignLLM contains two key modules:

1. A vector-quantized visual symbolization module that converts sign videos into a sequence of discrete character-level symbol tokens.

2. A codebook reconstruction and alignment module that converts these character-level tokens into word-level symbol representations using an optimal transport formulation. A symbol-text alignment loss further bridges the gap between symbol and text tokens, enhancing semantic compatibility. We achieve state-of-the-art gloss-free results on two widely used SLT benchmarks.

SignLLM GITHUB address: https://signllm.github.io/