Stability AI Releases Stable Code 3B Model 205

2024-01-17

Today, we announce our first large language model release of 2024: Stable Code 3B. This new LLM is the successor to our previously released Stable Code Alpha 3B, and as the first major Stable Code release, it provides a new state-of-the-art model designed for code completion with a range of additional features.

Stability AI Releases Stable Code 3B Model

Today, we are announcing the release of our first large language model in 2024: Stable Code 3B. This new LLM is the successor to our previously released Stable Code Alpha 3B and represents the first major Stable Code version, offering a new state-of-the-art model specifically designed for code completion with a variety of additional features.

Stable Code 3B Model Performance and Features:

1. Small Size, High Efficiency: Compared to CodeLLaMA 7b, Stable Code 3B is 60% smaller in size while delivering similar advanced performance across programming languages. Based on our Stable LM 3B foundation model pre-trained on 4 trillion natural language data points, Stable Code was further trained on software engineering-specific data (including code). The model's compact size enables it to run privately in real-time on the edge on modern laptops, even those without dedicated GPUs.

2. Runs on Standard Computers: Due to its compact model size, Stable Code 3B can run in real-time on modern laptops, including models without dedicated GPUs.

3. 100k Context Size: Stable Code 3B supports a larger context size, capable of processing sequences of up to 100,000 tokens, thereby providing richer and more precise code completions. Stable Code 3B offers more capabilities and significantly better performance across multiple languages, with additional benefits such as support for Fill-in-the-Middle (FIM) and an extended context size. Stable Code serves as a foundation, trained on sequences of up to 16,384 tokens, but following a similar approach to CodeLlama, it implements rotary embeddings, optionally allowing modification of up to 1,000,000 rotary bases, further extending the model's context length to 100k tokens.

4. Training on 18 Programming Languages: Stable Code was trained on 18 programming languages (selected based on the 2023 StackOverflow developer survey) and demonstrates state-of-the-art performance on the MultiPL-E metrics across multiple programming languages tested (compared to similarly sized models).

Stable Code 3B is included in our new Stability AI membership. Please visit our membership page to take advantage of our core commercial model offerings, including SDXL Turbo and Stable Video Diffusion.