search-web

LLMs.txt Explorer▌

by thedaviddias

Explore websites using llms.txt files with LLMs.txt Explorer. Fetch and parse site-specific language model instructions

Provides a bridge for exploring websites with llms.txt files, enabling fetching and parsing of website-specific language model instructions for context-aware interactions in web-based environments.

github stars

★ 75

GitHub npm

Supports both llms.txt and llms-full.txt formatsBuilt-in validation and parsing

best for

/ AI developers building context-aware web interactions
/ Finding websites that provide AI interaction guidelines
/ Validating llms.txt implementations

capabilities

/ Check websites for llms.txt files
/ Parse and validate llms.txt content
/ List known websites with llms.txt files
/ Filter results by file type

what it does

Discovers and fetches llms.txt files from websites, which contain instructions for how AI models should interact with those sites.

about

LLMs.txt Explorer is a community-built MCP server published by thedaviddias that provides AI assistants with tools and capabilities via the Model Context Protocol. Explore websites using llms.txt files with LLMs.txt Explorer. Fetch and parse site-specific language model instructions It is categorized under search web.

how to install

You can install LLMs.txt Explorer in your AI client of choice. Use the install panel on this page to get one-click setup for Cursor, Claude Desktop, VS Code, and other MCP-compatible clients. This server runs locally on your machine via the stdio transport.

license

NOASSERTION

LLMs.txt Explorer is released under the NOASSERTION license.

readme

MCP LLMS.txt Explorer

A Model Context Protocol server for exploring websites with llms.txt files. This server helps you discover and analyze websites that implement the llms.txt standard.

Features

Resources

Check websites for llms.txt and llms-full.txt files
Parse and validate llms.txt file contents
Access structured data about compliant websites

Tools

check_website - Check if a website has llms.txt files
- Takes domain URL as input
- Returns file locations and validation status
list_websites - List known websites with llms.txt files
- Returns structured data about compliant websites
- Supports filtering by file type (llms.txt/llms-full.txt)

Development

Install dependencies:

pnpm install

Build the server:

pnpm run build

For development with auto-rebuild:

pnpm run watch

Installation

Installing via Smithery

To install mcp-llms-txt-explorer for Claude Desktop automatically via Smithery:

npx -y @smithery/cli install @thedaviddias/mcp-llms-txt-explorer --client claude

Installing Manually

To use this server:

# Clone the repository
git clone https://github.com/thedaviddias/mcp-llms-txt-explorer.git
cd mcp-llms-txt-explorer

# Install dependencies
pnpm install

# Build the server
pnpm run build

Configuration with Claude Desktop

To use with Claude Desktop, add the server config:

On MacOS: ~/Library/Application Support/Claude/claude_desktop_config.json On Windows: %APPDATA%/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "llms-txt-explorer": {
      "command": "node",
      "args": ["/path/to/llms-txt-explorer/build/index.js"]
    }
  }
}

For npx usage, you can use:

{
  "mcpServers": {
    "llms-txt-explorer": {
      "command": "npx",
      "args": ["-y", "@thedaviddias/mcp-llms-txt-explorer"]
    }
  }
}

Debugging

Since MCP servers communicate over stdio, debugging can be challenging. We recommend using the MCP Inspector, which is available as a package script:

pnpm run inspector

The Inspector will provide a URL to access debugging tools in your browser.

License

This project is licensed under the MIT License—see the LICENSE file for details.