explainx.aijoin newsletter3.01k
search-web

Scrapling Fetch▌

by cyberchitta

Scrapling Fetch enables secure web scraping with three protection levels to scrape any website and access content blocke

Enables AI to access text content from websites protected by bot detection mechanisms through three protection levels (basic, stealth, max-stealth), retrieving complete pages or specific content patterns without manual copying.

github stars

★ 68

GitHubnpm
Bypasses bot detection mechanismsThree stealth protection levelsOptimized for documentation retrieval

best for

  • / Accessing documentation on protected sites
  • / Retrieving reference materials from bot-protected websites
  • / Low-volume content retrieval for research
  • / AI assistants needing access to blocked web content

capabilities

  • / Fetch complete web pages bypassing bot detection
  • / Extract specific content patterns with regex
  • / Handle pagination automatically
  • / Use three protection levels (basic, stealth, max-stealth)
  • / Retrieve text and HTML content only

what it does

Fetches web page content that's normally blocked by bot detection systems, allowing AI to access protected websites that would otherwise be inaccessible.

about

Scrapling Fetch is a community-built MCP server published by cyberchitta that provides AI assistants with tools and capabilities via the Model Context Protocol. Scrapling Fetch enables secure web scraping with three protection levels to scrape any website and access content blocke It is categorized under search web. This server exposes 2 tools that AI clients can invoke during conversations and coding sessions.

how to install

You can install Scrapling Fetch in your AI client of choice. Use the install panel on this page to get one-click setup for Cursor, Claude Desktop, VS Code, and other MCP-compatible clients. This server runs locally on your machine via the stdio transport.

license

Apache-2.0

Scrapling Fetch is released under the Apache-2.0 license. This is a permissive open-source license, meaning you can freely use, modify, and distribute the software.

readme

scrapling-fetch-mcp

License PyPI version

An MCP server that helps AI assistants access text content from websites that implement bot detection, bridging the gap between what you can see in your browser and what the AI can access.

Intended Use

This tool is optimized for low-volume retrieval of documentation and reference materials (text/HTML only) from websites that implement bot detection. It has not been designed or tested for general-purpose site scraping or data harvesting.

Note: This project was developed in collaboration with Claude Sonnets 3.7 and 4.5, using LLM Context.

Installation

Requirements

  • Python 3.10+
  • uv package manager

Install

# Install scrapling-fetch-mcp
uv tool install scrapling-fetch-mcp

# Install browser binaries (REQUIRED - large downloads)
uvx --from scrapling-fetch-mcp scrapling install

Important: The browser installation downloads hundreds of MB of data and must complete before first use. If the MCP server times out on first use, the browsers may still be installing in the background. Wait a few minutes and try again.

Setup with Claude Desktop

Add this configuration to your Claude Desktop MCP settings:

MacOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "scrapling-fetch": {
      "command": "uvx",
      "args": ["scrapling-fetch-mcp"]
    }
  }
}

After updating the config, restart Claude Desktop.

What It Does

This MCP server provides two tools that Claude can use automatically when you ask it to fetch web content:

  • Page fetching: Retrieves complete web pages with support for pagination
  • Pattern extraction: Finds and extracts specific content using regex patterns

The AI decides which tool to use based on your request. You just ask naturally:

"Can you fetch the docs at https://example.com/api"
"Find all mentions of 'authentication' on that page"
"Get me the installation instructions from their homepage"

Protection Modes

The tools support three levels of bot detection bypass:

  • basic: Fast (1-2s), works for most sites
  • stealth: Moderate (3-8s), handles more protection
  • max-stealth: Maximum (10+s), for heavily protected sites

Claude automatically starts with basic mode and escalates if needed.

Tips for Best Results

  • Just ask naturally - Claude handles the technical details
  • For large pages, Claude can page through content automatically
  • For specific searches, mention what you're looking for and Claude will use pattern matching
  • The metadata returned helps Claude decide whether to page or search

Limitations

  • Designed for text content only (documentation, articles, references)
  • Not for high-volume scraping or data harvesting
  • May not work with sites requiring authentication
  • Performance varies by site complexity and protection level

Built with Scrapling for web scraping with bot detection bypass.

License

Apache 2.0