Proxyless LLM Websearch: LLM Web Search Engine Without Proxy Required 93

What is Proxyless LLM Websearch?

Proxyless LLM Websearch is a proxy-free LLM web search engine that supports multi-engine search, intelligent parsing, and asynchronous processing. It also supports URL content parsing and web crawling. Combined with LangGraph, it enables a modular agent pipeline, specifically designed for external knowledge retrieval scenarios in large language models.

Features of Proxyless LLM Websearch

• Proxy-Free: By configuring domestic browsers via Playwright, web search can be carried out without a proxy.

• Multi-Search Engine Support: Supports mainstream search engines such as Bing, Quark, Baidu, and Sogou, enhancing the diversity of information sources.

• Intent Recognition: The system can automatically determine whether to perform a web search or parse a URL based on the user's input.

• Query Decomposition: Based on the user's search intent, the system automatically decomposes a query into multiple sub-tasks and executes them sequentially, thereby improving the relevance and efficiency of the search.

• Agent Architecture: Built on LangGraph with encapsulated "web_search" and "link_parser" components.

• Asynchronous Concurrent Task Processing: Supports asynchronous concurrent task processing, efficiently handling multiple search tasks.

• Content Processing Optimization

• Content Chunking: Splits long web page content into segments.

• Content Reranking: Intelligently reorders content to improve information relevance.

• Content Filtering: Automatically removes irrelevant or duplicate content.

• Multi-Platform Support

• FastAPI Backend Interface: Can be integrated into any system.

• Gradio Web UI: Can be quickly deployed as a visual application.

• Browser Extension Support: Supports Edge, providing an intelligent URL parsing plugin that allows web page parsing and content extraction requests to be initiated directly in the browser.

Installation and Usage

Clone the repository:

git clone https://github.com/itshyao/proxyless-llm-websearch.git
cd proxyless-llm-websearch

Install dependencies:

pip install -r requirements.txt
python -m playwright install

Environment variable configuration:

Environment variables such as OPENai_API_KEY, OPENAI_BASE_URL, MODEL_NAME, EMBEDDING_MODEL_NAME, EMBEDDING_API_KEY, and EMBEDDING_BASE_URL need to be configured.

Run examples:

Run the demo:

python demo.py

Run the FastAPI backend interface:

python api_serve.py

Run the Gradio Web UI:

python gradio_demo.py

Test Results

Based on Alibaba's challenging WebWalkerQA question-answering dataset, and compared with well-known online retrieval interfaces such as Volcengine Ark and Bailian, proxyless-llm-websearch achieved the highest accuracy and the lowest error rate.

GitHub repository:https://github.com/itshyao/proxyless-llm-websearch