LettuceDetect: A Fast, Low-Cost Hallucination Detection Tool for RAG Systems with Precise Localization 185

What is LettuceDetect?

LettuceDetect is a hallucination detection tool for Retrieval-Augmented Generation (RAG) systems. It identifies unsupported portions of an answer by comparing the answer with the context, thereby detecting hallucinated content.

Core Features of LettuceDetect

• Addressing limitations of existing methods: LettuceDetect overcomes the context window limitations of traditional encoder-based methods and the computational inefficiency of large language model (LLM)-based methods.

• Built on ModernBERT: The tool leverages ModernBERT's extended context processing capability (supporting up to 8192 tokens), enabling it to handle long texts.

• High accuracy and efficiency: It performs outstandingly on the RAGTruth dataset, achieving an F1 score of 79.22% on example-level detection, a 14.8% improvement over the previous state-of-the-art model Luna. Additionally, it can process 30 to 60 samples per second on a single GPU.

• Token-level detection: LettuceDetect is a token classification-based model that takes the context, question, and answer as input, and determines token-by-token whether each token in the answer is supported by the context.

• Optimized inference performance: A smaller model size is used to achieve faster inference speed.

• Long context processing: The tool leverages ModernBERT's extended context processing capability (supporting up to 8192 tokens) to handle long texts, enabling more accurate judgment of whether an answer is supported by the context.

Technical Implementation

• Training and evaluation: LettuceDetect is trained and evaluated on the RAGTruth dataset, utilizing ModernBERT for long context processing.

• Model and code licensing: Provided under the MIT license, the model and code can be used freely and have been integrated into Hugging Face, allowing the model to be loaded with a single line of code.

Application Scenarios of LettuceDetect:

• Finance: Detecting misinformation in financial consulting and financial analysis to prevent misleading users.

• Legal: Ensuring accuracy in legal consulting and contract interpretation to reduce legal risks.

• Healthcare: Assisting medical consulting and diagnosis to avoid misdiagnosis caused by hallucinations.

• Customer service: Improving the accuracy of enterprise customer service chatbots to enhance user trust.

• Education: Ensuring reliability of intelligent tutoring and online learning content.

• Enterprise knowledge management: Guaranteeing the accuracy of information output from internal knowledge management systems.

• High-risk industries: Combining multiple detection methods to improve the trustworthiness of system outputs.

GitHub Repository:https://github.com/KRLabsOrg/LettuceDetect