search-web

AgentQL

tinyfish-io

by tinyfish-io

AgentQL lets you scrape any website and extract structured data to JSON easily—no custom web scraping code needed.

Extracts structured data from web pages based on natural language descriptions, converting website content into JSON format without custom scraping code.

github stars

149

0 commentsdiscussion

Both formats append explainx.ai attribution and the canonical URL for this MCP server listing.

Natural language data descriptionsNo custom scraping code needed

best for

  • / Data analysts gathering web information
  • / Researchers collecting structured datasets
  • / Developers building data pipelines
  • / Anyone needing web scraping without coding

capabilities

  • / Extract structured data from web pages
  • / Convert website content to JSON format
  • / Describe data fields in natural language
  • / Scrape without writing custom code
  • / Parse complex web layouts automatically

what it does

Extracts structured data from any web page using plain English descriptions instead of writing custom scraping code. Returns clean JSON data based on what you ask for.

about

AgentQL is an official MCP server published by tinyfish-io that provides AI assistants with tools and capabilities via the Model Context Protocol. AgentQL lets you scrape any website and extract structured data to JSON easily—no custom web scraping code needed. It is categorized under search web.

how to install

You can install AgentQL in your AI client of choice. Use the install panel on this page to get one-click setup for Cursor, Claude Desktop, VS Code, and other MCP-compatible clients. This server runs locally on your machine via the stdio transport.

license

MIT

AgentQL is released under the MIT license. This is a permissive open-source license, meaning you can freely use, modify, and distribute the software.

readme

AgentQL MCP Server

This is a Model Context Protocol (MCP) server that integrates AgentQL's data extraction capabilities.

Features

Tools

  • extract-web-data - extract structured data from a given 'url', using 'prompt' as a description of actual data and its fields to extract.

Installation

To use AgentQL MCP Server to extract data from web pages, you need to install it via npm, get an API key from our Dev Portal, and configure it in your favorite app that supports MCP.

Install the package

npm install -g agentql-mcp

Configure Claude

  • Open Claude Desktop Settings via +, (don't confuse with Claude Account Settings)
  • Go to Developer sidebar section
  • Click Edit Config and open claude_desktop_config.json file
  • Add agentql server inside mcpServers dictionary in the config file
  • Restart the app
{
  "mcpServers": {
    "agentql": {
      "command": "npx",
      "args": ["-y", "agentql-mcp"],
      "env": {
        "AGENTQL_API_KEY": "YOUR_API_KEY"
      }
    }
  }
}

Read more about MCP configuration in Claude here.

Configure VS Code

For one-click installation, click one of the install buttons below:

Install with NPX in VS Code Install with NPX in VS Code Insiders

Manual Installation

Click the install buttons at the top of this section for the quickest installation method. For manual installation, follow these steps:

Add the following JSON block to your User Settings (JSON) file in VS Code. You can do this by pressing Ctrl + Shift + P and typing Preferences: Open User Settings (JSON).

{
  "mcp": {
    "inputs": [
      {
        "type": "promptString",
        "id": "apiKey",
        "description": "AgentQL API Key",
        "password": true
      }
    ],
    "servers": {
      "agentql": {
        "command": "npx",
        "args": ["-y", "agentql-mcp"],
        "env": {
          "AGENTQL_API_KEY": "${input:apiKey}"
        }
      }
    }
  }
}

Optionally, you can add it to a file called .vscode/mcp.json in your workspace. This will allow you to share the configuration with others.

{
  "inputs": [
    {
      "type": "promptString",
      "id": "apiKey",
      "description": "AgentQL API Key",
      "password": true
    }
  ],
  "servers": {
    "agentql": {
      "command": "npx",
      "args": ["-y", "agentql-mcp"],
      "env": {
        "AGENTQL_API_KEY": "${input:apiKey}"
      }
    }
  }
}

Configure Cursor

  • Open Cursor Settings
  • Go to MCP > MCP Servers
  • Click + Add new MCP Server
  • Enter the following:
    • Name: "agentql" (or your preferred name)
    • Type: "command"
    • Command: env AGENTQL_API_KEY=YOUR_API_KEY npx -y agentql-mcp

Read more about MCP configuration in Cursor here.

Configure Windsurf

  • Open Windsurf: MCP Configuration Panel
  • Click Add custom server+
  • Alternatively you can open ~/.codeium/windsurf/mcp_config.json directly
  • Add agentql server inside mcpServers dictionary in the config file
{
  "mcpServers": {
    "agentql": {
      "command": "npx",
      "args": ["-y", "agentql-mcp"],
      "env": {
        "AGENTQL_API_KEY": "YOUR_API_KEY"
      }
    }
  }
}

Read more about MCP configuration in Windsurf here.

Validate MCP integration

Give your agent a task that will require extracting data from the web. For example:

Extract the list of videos from the page https://www.youtube.com/results?search_query=agentql, every video should have a title, an author name, a number of views and a url to the video. Make sure to exclude ads items. Format this as a markdown table.

[!TIP] In case your agent complains that it can't open urls or load content from the web instead of using AgentQL, try adding "use tools" or "use agentql tool" hint.

Development

Install dependencies:

npm install

Build the server:

npm run build

For development with auto-rebuild:

npm run watch

If you want to try out development version, you can use the following config instead of the default one:

{
  "mcpServers": {
    "agentql": {
      "command": "/path/to/agentql-mcp/dist/index.js",
      "env": {
        "AGENTQL_API_KEY": "YOUR_API_KEY"
      }
    }
  }
}

[!NOTE] Don't forget to remove the default AgentQL MCP server config to not confuse Claude with two similar servers.

Debugging

Since MCP servers communicate over stdio, debugging can be challenging. We recommend using the MCP Inspector, which is available as a package script:

npm run inspector

The Inspector will provide a URL to access debugging tools in your browser.

FAQ

What is the AgentQL MCP server?
AgentQL is a Model Context Protocol (MCP) server profile on explainx.ai. MCP lets AI hosts (e.g. Claude Desktop, Cursor) call tools and resources through a standard interface; this page summarizes categories, install hints, and community ratings.
How do MCP servers relate to agent skills?
Skills are reusable instruction packages (often SKILL.md); MCP servers expose live capabilities. Teams frequently combine both—skills for workflows, MCP for APIs and data. See explainx.ai/skills and explainx.ai/mcp-servers for parallel directories.
How are reviews shown for AgentQL?
This profile displays 40 aggregated ratings (sample rows for discoverability plus signed-in user reviews). Average score is about 4.6 out of 5—verify behavior in your own environment before production use.

Use Cases

Web Research & Information Gathering

Fetch and extract information from websites automatically

Example

Research competitor pricing, scrape product reviews, monitor news mentions

Automate 5-10 hours/week of manual web research

Content Monitoring & Alerts

Track website changes, new content, price updates

Example

Monitor competitor blog for new posts, track stock availability, watch for pricing changes

Stay informed without manual checking, never miss important updates

Data Extraction & Aggregation

Extract structured data from multiple websites

Example

Compile product listings from 10 e-commerce sites, aggregate job postings, collect real estate data

Build datasets 100x faster than manual copying

API-less Integration

Interact with services that don't offer APIs

Example

Check form submissions, validate website functionality, test user flows

Automate interactions with any website, even without API

Implementation Guide

Prerequisites

  • Claude Desktop or Cursor with MCP support
  • Understanding of web scraping ethics and robots.txt
  • Rate limiting awareness to avoid overwhelming target sites
  • Knowledge of legal restrictions on data collection

Time Estimate

20-40 minutes including configuration and testing

Installation Steps

  1. 1.Install web automation MCP server via npm or pip
  2. 2.Configure allowed domains and rate limits in MCP config
  3. 3.Test with simple fetch: 'Get content from example.com'
  4. 4.Progress to extraction: 'Extract all product prices from this page'
  5. 5.Set up monitoring: 'Check this URL daily for changes'
  6. 6.Parse structured data: 'Create CSV from this table'
  7. 7.Respect robots.txt and rate limits always

Troubleshooting

  • 403 Forbidden: Website blocks bots—respect their wishes, use official API instead
  • Rate limit errors: Slow down requests, add delays between fetches
  • Stale data: Target site changed HTML structure—update selectors
  • Timeout errors: Site is slow or blocking—increase timeout, try different user agent
  • JavaScript-rendered content: Use headless browser MCP servers for dynamic sites

Best Practices

✓ Do

  • +Check robots.txt and respect crawl rules
  • +Rate limit requests: 1-2 requests/second maximum
  • +Use official APIs when available instead of scraping
  • +Identify your bot with descriptive user agent
  • +Cache results to minimize repeated requests
  • +Handle errors gracefully with retries and fallbacks
  • +Validate extracted data for accuracy

✗ Don't

  • Don't scrape sites that explicitly forbid it (robots.txt, ToS)
  • Don't overwhelm servers with rapid requests—use rate limiting
  • Don't scrape personal data without consent and legal basis
  • Don't ignore copyright on extracted content
  • Don't assume HTML structure is stable—handle changes
  • Don't use scraped data for commercial purposes without permission

💡 Pro Tips

  • Use CSS selectors or XPath for robust data extraction
  • Set up monitoring alerts for extraction failures (structure changed)
  • Implement exponential backoff for retries on failures
  • Store raw HTML for reprocessing if extraction logic changes
  • Combine with data analysis tools for insights from extracted data
  • Consider using official APIs or RSS feeds as more stable alternatives

Technical Details

Architecture

MCP server handles HTTP requests, HTML parsing, JavaScript rendering (if headless browser), and returns structured data to Claude.

Protocols

  • HTTP/HTTPS
  • WebSocket (for real-time sites)
  • Puppeteer/Playwright (for JavaScript sites)

Compatibility

  • Static HTML sites
  • JavaScript-rendered SPAs (with headless browser)
  • REST APIs
  • GraphQL endpoints

When to Use This

✓ Use When

Use for research automation, content monitoring, data aggregation from multiple sources, and when official APIs don't exist. Best for read-only information gathering.

✗ Avoid When

Avoid for sites with APIs (use API instead), sites that explicitly forbid scraping, when data is copyrighted, or for login-required content without proper authorization.

Integration

  • Scheduled monitoring with change detection
  • Multi-source data aggregation pipelines
  • Fallback to web scraping when API rate limits hit
  • Headless browser for JavaScript-heavy sites

Discussion

Product Hunt–style comments (not star reviews)
  • No comments yet — start the thread.

List & Promote Your MCP Server

Share your MCP server with the developer community

GET_STARTED →
MCP server reviews

Ratings

4.640 reviews
  • Chaitanya Patil· Dec 28, 2024

    We wired AgentQL into a staging workspace; the listing’s GitHub and npm pointers saved time versus hunting across READMEs.

  • Xiao Mehta· Dec 12, 2024

    According to our notes, AgentQL benefits from clear Model Context Protocol framing — fewer ambiguous “AI plugin” claims.

  • Piyush G· Nov 19, 2024

    AgentQL is a well-scoped MCP server in the explainx.ai directory — install snippets and categories matched our Claude Code setup.

  • Arjun Mensah· Nov 3, 2024

    AgentQL is among the better-indexed MCP projects we tried; the explainx.ai summary tracks the official description.

  • Tariq Ndlovu· Oct 22, 2024

    AgentQL is a well-scoped MCP server in the explainx.ai directory — install snippets and categories matched our Claude Code setup.

  • Shikha Mishra· Oct 10, 2024

    AgentQL is among the better-indexed MCP projects we tried; the explainx.ai summary tracks the official description.

  • Rahul Santra· Sep 21, 2024

    Strong directory entry: AgentQL surfaces stars and publisher context so we could sanity-check maintenance before adopting.

  • Amelia Diallo· Sep 17, 2024

    AgentQL is a well-scoped MCP server in the explainx.ai directory — install snippets and categories matched our Claude Code setup.

  • Tariq Lopez· Sep 13, 2024

    I recommend AgentQL for teams standardizing on MCP; the explainx.ai page compares cleanly with sibling servers.

  • Sofia Kim· Sep 9, 2024

    AgentQL has been reliable for tool-calling workflows; the MCP profile page is a good permalink for internal docs.

showing 1-10 of 40

1 / 4