Scrapling is a powerful Python web scraping library with a comprehensive CLI for extracting data from websites directly from the terminal without writing code. The primary use case is the extract command group for quick data extraction.
Works with
AI-first code editor with Composer
Before installing skills in Cursor, ensure your development environment meets these requirements:
node --versionscraplingExecute the skills CLI command in your project's root directory to begin installation:
Fetches scrapling from hyperpuncher/dotagents and configures it for Cursor.
The CLI shows a list of agents. Use arrow keys and space to select Cursor:
Confirm successful installation by checking the skill directory location:
Restart Cursor to activate scrapling. Access via /scrapling in your agent's command palette.
We perform automated surface-level scans (Gen AI Scanner, Socket, Snyk) during installation. These checks detect common vulnerabilities but do not guarantee complete security. Always review skill source code and verify the publisher's reputation before production use.
Skills execute code in your environment. Always review source, verify the publisher, and test in isolation before production.
Submit your Claude Code skill and start earning
Create detailed user stories, acceptance criteria, and feature specs
Example
Generate user stories for 'password reset feature' with acceptance criteria, edge cases, and test scenarios
Reduce spec writing time by 50%, ensure comprehensive coverage
Research competitors, compare features, identify gaps
Example
Analyze 5 competitor products, create feature comparison matrix, suggest differentiation opportunities
Complete competitive research in 2 hours instead of 2 days
Evaluate features using frameworks (RICE, ICE, Kano) and create prioritized backlogs
Example
Score 20 feature ideas using RICE framework, generate prioritized roadmap with rationale
2
total installs
2
this week
1
GitHub stars
0
upvotes
Run in your terminal
2
installs
2
this week
1
stars
Scrapling is a powerful Python web scraping library with a comprehensive CLI for extracting data from websites directly from the terminal without writing code. The primary use case is the extract command group for quick data extraction.
Install with the shell extras using uv:
uv tool install "scrapling[shell]"
Then install fetcher dependencies (browsers, system dependencies, fingerprint manipulation):
scrapling install
Update to the latest version:
uv tool update "scrapling[shell]"
The scrapling extract command group allows you to download and extract content from websites without writing any code. Output format is determined by file extension:
Note: All examples use
--ai-targetedby default. This flag extracts only main body content, strips noise tags (script, style, noscript, svg), removes hidden elements, strips zero-width unicode characters, and removes HTML comments - ideal when output is destined for an AI model.
.md - Convert HTML to Markdown.html - Save raw HTML.txt - Extract clean text content# Basic website download as text
scrapling extract get "https://example.com" page_content.txt --ai-targeted
# Download as markdown
scrapling extract get "https://blog.example.com" article.md --ai-targeted
# Save raw HTML
scrapling extract get "https://example.com" page.html --ai-targeted
| Use Case | Command |
|---|---|
| Simple websites, blogs, news articles | get |
| Modern web apps, dynamic content (JavaScript) | fetch |
| Protected sites, Cloudflare, anti-bot | stealthy-fetch |
| Form submissions, APIs | post, put, delete |
Most common command for downloading website content:
# Basic download
scrapling extract get "https://news.site.com" news.md --ai-targeted
# Download with custom timeout
scrapling extract get "https://example.com" content.txt --timeout 60 --ai-targeted
# Extract specific content using CSS selectors
scrapling extract get "https://blog.example.com" articles.md --css-selector "article" --ai-targeted
# Send request with cookies
scrapling extract get "https://scrapling.requestcatcher.com" content.md \
--cookies "session=abc123; user=john" --ai-targeted
# Add user agent
scrapling extract get "https://api.site.com" data.json \
-H "User-Agent: MyBot 1.0" --ai-targeted
# Add multiple headers
scrapling extract get "https://site.com" page.html \
-H "Accept: text/html" \
-H "Accept-Language: en-US" --ai-targeted
# With query parameters
scrapling extract get "https://api.example.com" data.json \
-p "page=1" -p "limit=10" --ai-targeted
GET options:
-H, --headers TEXT HTTP headers "Key: Value" (multiple allowed)
--cookies TEXT Cookies "name1=value1;name2=value2"
--timeout INTEGER Request timeout in seconds (default: 30)
--proxy TEXT Proxy URL from $PROXY_URL env variable
-s, --css-selector TEXT Extract specific content with CSS selector
-p, --params TEXT Query parameters "key=value" (multiple)
--follow-redirects / --no-follow-redirects (default: True)
--verify / --no-verify SSL verification (default: True)
--impersonate TEXT Browser to impersonate (chrome, firefox)
--stealthy-headers / --no-stealthy-headers (default: True)
--ai-targeted Extract main content and sanitize for AI
# Submit form data
scrapling extract post "https://api.site.com/search" results.html \
--data "query=python&type=tutorial" --ai-targeted
# Send JSON data
scrapling extract post "https://api.site.com" response.json \
--json '{"username": "test", "action": "search"}' --ai-targeted
POST options: (same as GET plus)
-d, --data TEXT Form data "param1=value1¶m2=value2"
-j, --json TEXT JSON data as string
# Send data
scrapling extract put "https://api.example.com" results.html \
--data "update=info" \
--impersonate "firefox" --ai-targeted
# Send JSON data
scrapling extract put "https://api.example.com" response.json \
--json '{"username": "test", "action": "search"}' --ai-targeted
scrapling extract delete "https://api.example.com/resource" response.txt --ai-targeted
# With impersonation
scrapling extract delete "https://api.example.com/" response.txt \
--impersonate "chrome" --ai-targeted
Use browser-based fetching for JavaScript-heavy sites or when HTTP requests fail.
For websites that load content dynamically or have slight protection:
# Wait for JavaScript to load and network activity to finish
scrapling extract fetch "https://example.com" content.md --network-idle --ai-targeted
# Wait for specific element to appear
scrapling extract fetch "https://example.com" data.txt \
--wait-selector ".content-loaded" --ai-targeted
# Visible browser mode for debugging
scrapling extract fetch "https://example.com" page.html \
--no-headless --disable-resources --ai-targeted
# Use installed Chrome browser
scrapling extract fetch "https://example.com" content.md --real-chrome --ai-targeted
# With CSS selector extraction
scrapling extract fetch "https://example.com" articles.md \
--css-selector "article" \
--network-idle --ai-targeted
fetch options:
--headless / --no-headless Run browser headless (default: True)
--disable-resources Drop unnecessary resources for speed boost
--network-idle Wait for network idle
--timeout INTEGER Timeout in milliseconds (default: 30000)
--wait INTEGER Additional wait time in ms (default: 0)
-s, --css-selector TEXT Extract specific content
--wait-selector TEXT Wait for selector before proceeding
--locale TEXT User locale (default: system)
--real-chrome Use installed Chrome browser
--proxy TEXT Proxy URL
-H, --extra-headers TEXT Extra headers (multiple)
--ai-targeted Extract main content and sanitize for AI
For websites with anti-bot protection or Cloudflare:
# Bypass basic protection
scrapling extract stealthy-fetch "https://example.com" content.md --ai-targeted
# Solve Cloudflare challenges
scrapling extract stealthy-fetch "https://nopecha.com/demo/cloudflare" data.txt \
--solve-cloudflare \
--css-selector "#padded_content a" --ai-targeted
# Use proxy for anonymity (set PROXY_URL environment variable)
scrapling extract stealthy-fetch "https://site.com" content.md \
--proxy "$PROXY_URL" --ai-targeted
# Hide canvas fingerprint
scrapling extract stealthy-fetch "https://example.com" content.md \
--hide-canvas \
--block-webrtc --ai-targeted
stealthy-fetch options: (same as fetch plus)
--block-webrtc Block WebRTC entirely
--solve-cloudflare Solve Cloudflare challenges
--allow-webgl / --block-webgl Allow WebGL (default: True)
--hide-canvas Add noise to canvas operations
--ai-targeted Extract main content and sanitize for AI
Extract specific content with the -s or --css-selector flag:
# Extract all articles
scrapling extract get "https://blog.example.com" articles.md -s "article" --ai-targeted
# Extract specific class
scrapling extract get "https://example.com" titles.txt -s ".title" --ai-targeted
# Extract by ID
scrapling extract get "https://example.com" content.md -s "#main-content" --ai-targeted
# Extract links (href attributes)
scrapling extract get "https://example.com" links.txt -s "a::attr(href)" --ai-targeted
# Extract text only
scrapling extract get "https://example.com" titles.txt -s "h1::text" --ai-targeted
# Extract multiple elements with fetch
scrapling extract fetch "https://example.com" products.md \
-s ".product-card" \
--network-idle --ai-targeted
scrapling --help
scrapling extract --help
scrapling extract get --help
scrapling extract post --help
scrapling extract fetch --help
scrapling extract stealthy-fetch --help
Make data-driven prioritization decisions faster
Draft PRDs, status updates, and stakeholder presentations
Example
Create executive summary of Q3 roadmap, monthly progress report, feature launch announcement
Save 3-5 hours/week on communication overhead
Prerequisites
Time Estimate
30-60 minutes to see productivity improvements
Steps
Common Pitfalls
✓ Do
✗ Don't
💡 Pro Tips
✓ Use when
Use for user story writing, competitive research, roadmap prioritization, stakeholder communication, and PRD drafting. Best for reducing repetitive documentation and research work.
✗ Avoid when
Avoid for strategic product vision (requires deep customer empathy), pricing decisions (needs market and financial expertise), or when face-to-face customer discovery is more valuable than speed.
mattpocock/skills
parcadei/continuous-claude-v3
cursor/plugins
ailabs-393/ai-labs-claude-skills
pproenca/dot-skills
mattpocock/skills
Registry listing for scrapling matched our evaluation — installs cleanly and behaves as described in the markdown.
scrapling is among the better-maintained entries we tried; worth keeping pinned for repeat workflows.
Solid pick for teams standardizing on skills: scrapling is focused, and the summary matches what you get after install.
scrapling reduced setup friction for our internal harness; good balance of opinion and flexibility.
scrapling has been reliable in day-to-day use. Documentation quality is above average for community skills.
Keeps context tight: scrapling is the kind of skill you can hand to a new teammate without a long onboarding doc.
scrapling fits our agent workflows well — practical, well scoped, and easy to wire into existing repos.
I recommend scrapling for anyone iterating fast on agent tooling; clear intent and a small, reviewable surface area.
Registry listing for scrapling matched our evaluation — installs cleanly and behaves as described in the markdown.
Keeps context tight: scrapling is the kind of skill you can hand to a new teammate without a long onboarding doc.
showing 1-10 of 38