sitemcp: An Open-Source Project That Turns Any Website Into an MCP Server 134

What is siteMCP?

sitemcp can crawl website content and cache it, and supports using micromatch to specify the page paths to crawl along with CSS selectors to finely select page content. The crawled content is then used as the data source for the MCP server. In other words, it turns a static website into an interactive knowledge base that works with an MCP client for asking questions and obtaining more precise information.

sitemcp Features

Content Crawling and Caching: It can crawl all or part of the pages of a specified website and cache them. It supports using micromatch to specify the paths to crawl, for example:

sitemcp https://vite.dev -m "/blog/**" -m "/guide/**"

Content Selector: In some pages, the content extracted by mozilla/readability may not be accurate enough. In such cases, you can precisely extract readable content by specifying a CSS selector:

sitemcp https://vite.dev --content-selector ".content"

Working with MCP Clients: sitemcp can work with MCP clients (such as Claude Desktop). Through configuration files, you can launch the sitemcp server in the client. For example:

{
"mcpServers": {
"daisy-ui": {
  "command": "npx",
  "args": [
"-y",
"sitemcp",
"https://daisyui.com",
"-m",
"/components/**",
  ]
}
}
}

Caching Mechanism: By default, crawled pages are cached in ~/.cache/sitemcp. If you do not need caching, you can disable it with the --no-cache flag.

sitemcp Applications

sitemcp essentially turns static websites into an interactive knowledge base that works with an MCP client for asking questions, unlocking richer application scenarios. For example, users can use the crawled website content as a knowledge base and ask questions through an MCP client to obtain more precise information.

sitemcp Usage Tutorial

Installation: sitemcp supports multiple installation methods. You can use the following commands for a one-time installation:

bunx sitemcp
npx sitemcp
pnpx sitemcp

Alternatively, you can install it globally:

bun i -g sitemcp
npm i -g sitemcp
pnpm i -g sitemcp

Usage: When using it, you only need to specify the target website URL:

sitemcp https://daisyui.com

You can also use the --concurrency parameter to specify the number of concurrent requests to improve crawling efficiency:

sitemcp https://daisyui.com --concurrency 10

GitHub address: https://github.com/ryoppippi/sitemcp