LLMs-from-scratch: Build a Large Language Model from Scratch 56

"LLMs-from-scratch" is an open-source project created by Sebastian Raschka, aimed at helping developers implement a ChatGPT-like large language model from scratch. This project serves as the official code repository accompanying his book Build a Large Language Model (From Scratch).

Project Goals

• Build an LLM from scratch: Help readers gain a deep understanding of the internal mechanisms of LLMs through step-by-step coding.

• Pre-training and fine-tuning: Provide complete code examples for pre-training on unlabeled data and fine-tuning for specific tasks.

• Educational purpose: Help readers understand the development process of large models by building small yet fully functional models.

Main Functional Structure

Chapter 2: Working with Text Data

• Implement a byte pair encoding (BPE) tokenizer from scratch.

• Provide the implementation and summary of data loaders.

Chapter 3: Implementing the Attention Mechanism

• Implement multi-head attention.

Chapter 4: Implementing a GPT Model from Scratch

Provides the complete GPT model code.

Chapter 5: Pre-training on Unlabeled Data

• Provides pre-training code and generator code.

Chapter 6: Fine-tuning for Text Classification

• Provides fine-tuning code for text classification tasks.

Chapter 7: Fine-tuning for Instruction Following

• Provides instruction fine-tuning code and evaluation code.

LLMs-from-scratch is suitable for learners interested in natural language processing and deep learning, especially developers who wish to build large language models from scratch.

Original English Address:https://github.com/rasbt/LLMs-from-scratch

Chinese Resource Address:https://github.com/MLNLP-World/LLMs-from-scratch-CN