Explore websites using llms.txt files with LLMs.txt Explorer. Fetch and parse site-specific language model instructions
by thedaviddias·Fetch and extract information from websites automatically
Example: Research competitor pricing, scrape product reviews, monitor news mentions
Automate 5-10 hours/week of manual web research
Track website changes, new content, price updates
Example: Monitor competitor blog for new posts, track stock availability, watch for pricing changes
Stay informed without manual checking, never miss important updates
Extract structured data from multiple websites
Example: Compile product listings from 10 e-commerce sites, aggregate job postings, collect real estate data
Build datasets 100x faster than manual copying
Interact with services that don't offer APIs
Example: Check form submissions, validate website functionality, test user flows
Automate interactions with any website, even without API
Scrapling Fetch
Scrapling Fetch enables secure web scraping with three protection levels to scrape any website and access content blocke
Google Custom Search
Integrate Google Custom Search API for real-time web searches, fact-checking, and research using reliable Google data.
Read Website Fast
Extract web content and convert to clean Markdown. Fast data extraction from web pages with caching, robots.txt support,
YouTube Transcripts
Extract and analyze YouTube transcripts in multiple languages. Use our YouTube transcriptor to easily transcribe for You
HTTP Request
Advanced web scraper lets LLMs bypass anti-bot protection using HTTP requests, ideal for web scraping tools like Octopar
LLMs.txt Explorer is among the better-indexed MCP projects we tried; the explainx.ai summary tracks the official description.
LLMs.txt Explorer is among the better-indexed MCP projects we tried; the explainx.ai summary tracks the official description.
Strong directory entry: LLMs.txt Explorer surfaces stars and publisher context so we could sanity-check maintenance before adopting.
LLMs.txt Explorer is among the better-indexed MCP projects we tried; the explainx.ai summary tracks the official description.
Strong directory entry: LLMs.txt Explorer surfaces stars and publisher context so we could sanity-check maintenance before adopting.
Strong directory entry: LLMs.txt Explorer surfaces stars and publisher context so we could sanity-check maintenance before adopting.
Useful MCP listing: LLMs.txt Explorer is the kind of server we cite when onboarding engineers to host + tool permissions.
LLMs.txt Explorer reduced integration guesswork — categories and install configs on the listing matched the upstream repo.
Useful MCP listing: LLMs.txt Explorer is the kind of server we cite when onboarding engineers to host + tool permissions.
I recommend LLMs.txt Explorer for teams standardizing on MCP; the explainx.ai page compares cleanly with sibling servers.
showing 1-10 of 50
Share your MCP server with the developer community
Select a client to integrate this server.
Claude Desktop configuration
{
"mcpServers": {
"llms-txt-explorer": {
"command": "npx",
"args": [
"-y",
"@thedaviddias/mcp-llms-txt-explorer"
]
}
}
}Discovers and fetches llms.txt files from websites, which contain instructions for how AI models should interact with those sites.
LLMs.txt Explorer is a community-built MCP server published by thedaviddias that provides AI assistants with tools and capabilities via the Model Context Protocol. Explore websites using llms.txt files with LLMs.txt Explorer. Fetch and parse site-specific language model instructions It is categorized under search web.
You can install LLMs.txt Explorer in your AI client of choice. Use the install panel on this page to get one-click setup for Cursor, Claude Desktop, VS Code, and other MCP-compatible clients. This server runs locally on your machine via the stdio transport.
A Model Context Protocol server for exploring websites with llms.txt files. This server helps you discover and analyze websites that implement the llms.txt standard.
check_website - Check if a website has llms.txt files
list_websites - List known websites with llms.txt files
Install dependencies:
pnpm install
Build the server:
pnpm run build
For development with auto-rebuild:
pnpm run watch
To install mcp-llms-txt-explorer for Claude Desktop automatically via Smithery:
npx -y @smithery/cli install @thedaviddias/mcp-llms-txt-explorer --client claude
To use this server:
# Clone the repository
git clone https://github.com/thedaviddias/mcp-llms-txt-explorer.git
cd mcp-llms-txt-explorer
# Install dependencies
pnpm install
# Build the server
pnpm run build
To use with Claude Desktop, add the server config:
On MacOS: ~/Library/Application Support/Claude/claude_desktop_config.json
On Windows: %APPDATA%/Claude/claude_desktop_config.json
{
"mcpServers": {
"llms-txt-explorer": {
"command": "node",
"args": ["/path/to/llms-txt-explorer/build/index.js"]
}
}
}
For npx usage, you can use:
{
"mcpServers": {
"llms-txt-explorer": {
"command": "npx",
"args": ["-y", "@thedaviddias/mcp-llms-txt-explorer"]
}
}
}
Since MCP servers communicate over stdio, debugging can be challenging. We recommend using the MCP Inspector, which is available as a package script:
pnpm run inspector
The Inspector will provide a URL to access debugging tools in your browser.
This project is licensed under the MIT License—see the LICENSE file for details.
Time estimate: 20-40 minutes including configuration and testing
Prerequisites
Steps
Troubleshooting
✓ Do
✗ Don't
💡 Pro Tips
Architecture
MCP server handles HTTP requests, HTML parsing, JavaScript rendering (if headless browser), and returns structured data to Claude.
Protocols
Compatibility
✓ Use when
Use for research automation, content monitoring, data aggregation from multiple sources, and when official APIs don't exist. Best for read-only information gathering.
✗ Avoid when
Avoid for sites with APIs (use API instead), sites that explicitly forbid scraping, when data is copyrighted, or for login-required content without proper authorization.