firecrawl-scraping
Scrape individual web pages and convert them to clean, LLM-ready markdown. Handles JavaScript rendering, anti-bot protection, and dynamic content.
Works with
0
total installs
0
this week
10
GitHub stars
0
upvotes
Install Skill
Run in your terminal
0
installs
0
this week
10
stars
Installation Guide
How to use firecrawl-scraping on Cursor
AI-first code editor with Composer
Prerequisites
Before installing skills in Cursor, ensure your development environment meets these requirements:
- ›Cursor installed and configured on your machine
- ›Node.js 16+ with npm — verify with
node --version - ›Active project directory where you want to add
firecrawl-scraping
Run the install command
Execute the skills CLI command in your project's root directory to begin installation:
Fetches firecrawl-scraping from casper-studios/casper-marketplace and configures it for Cursor.
Select Cursor when prompted
The CLI shows a list of agents. Use arrow keys and space to select Cursor:
Verify installation
Confirm successful installation by checking the skill directory location:
Restart Cursor to activate firecrawl-scraping. Access via /firecrawl-scraping in your agent's command palette.
Security Notice
We perform automated surface-level scans (Gen AI Scanner, Socket, Snyk) during installation. These checks detect common vulnerabilities but do not guarantee complete security. Always review skill source code and verify the publisher's reputation before production use.
Skills execute code in your environment. Always review source, verify the publisher, and test in isolation before production.
Documentation
Firecrawl Scraping
Overview
Scrape individual web pages and convert them to clean, LLM-ready markdown. Handles JavaScript rendering, anti-bot protection, and dynamic content.
Quick Decision Tree
What are you scraping?
│
├── Single page (article, blog, docs)
│ └── references/single-page.md
│ └── Script: scripts/firecrawl_scrape.py
│
└── Entire website (multiple pages, crawling)
└── references/website-crawler.md
└── (Use Apify Website Content Crawler for multi-page)
Environment Setup
# Required in .env
FIRECRAWL_API_KEY=fc-your-api-key-here
Get your API key: https://firecrawl.dev/app/api-keys
Common Usage
Simple Scrape
python scripts/firecrawl_scrape.py "https://example.com/article"
With Options
python scripts/firecrawl_scrape.py "https://wsj.com/article" \
--proxy stealth \
--format markdown summary \
--timeout 60000
Proxy Modes
| Mode | Use Case |
|---|---|
basic |
Standard sites, fastest |
stealth |
Anti-bot protection, premium content (WSJ, NYT) |
auto |
Let Firecrawl decide (recommended) |
Output Formats
markdown- Clean markdown content (default)html- Raw HTMLsummary- AI-generated summaryscreenshot- Page screenshotlinks- All links on page
Cost
~1 credit per page. Stealth proxy may use additional credits.
Security Notes
Credential Handling
- Store
FIRECRAWL_API_KEYin.envfile (never commit to git) - API keys can be regenerated at https://firecrawl.dev/app/api-keys
- Never log or print API keys in script output
- Use environment variables, not hardcoded values
Data Privacy
- Only scrapes publicly accessible web pages
- Scraped content is processed by Firecrawl servers temporarily
- Markdown output stored locally in
.tmp/directory - Screenshots (if requested) are stored locally
- No persistent data retention by Firecrawl after request
Access Scopes
- API key provides full access to scraping features
- No granular permission scopes available
- Monitor usage via Firecrawl dashboard
Compliance Considerations
- Robots.txt: Firecrawl respects robots.txt by default
- Public Content Only: Only scrape publicly accessible pages
- Terms of Service: Respect target site ToS
- Rate Limiting: Built-in rate limiting prevents abuse
- Stealth Proxy: Use stealth mode only when necessary (paywalled news, not auth bypass)
- GDPR: Scraped content may contain PII - handle accordingly
- Copyright: Respect intellectual property rights of scraped content
Troubleshooting
Common Issues
Issue: Credits exhausted
Symptoms: API returns "insufficient credits" or quota exceeded error Cause: Account credits depleted Solution:
- Check credit balance at https://firecrawl.dev/app
- Upgrade plan or purchase additional credits
- Reduce scraping frequency
- Use
basicproxy mode to conserve credits
Issue: Page not rendering correctly
Symptoms: Empty content or partial HTML returned Cause: JavaScript-heavy page not fully loading Solution:
- Enable JavaScript rendering with
--js-renderflag - Increase timeout with
--timeout 60000(60 seconds) - Try
stealthproxy mode for protected sites - Wait for specific elements with
--wait-forselector
Issue: 403 Forbidden error
Symptoms: Script returns 403 status code Cause: Site blocking automated access Solution:
- Enable
stealthproxy mode - Add delay between requests
- Try at different times (some sites rate limit by time)
- Check if site requires login (not supported)
Issue: Empty markdown output
Symptoms: Scrape succeeds but markdown is empty or malformed Cause: Dynamic content loaded after page load, or unusual page structure Solution:
- Increase wait time for JavaScript to execute
- Use
--wait-forto wait for specific content - Try
htmlformat to see raw content - Check if content is in an iframe (not always supported)
Issue: Timeout errors
Symptoms: Request times out before completion Cause: Slow page load or large page content Solution:
- Increase timeout value (up to 120000ms)
- Use
basicproxy for faster response - Target specific page sections if possible
- Check if site is experiencing issues
Resources
- references/single-page.md - Single page scraping details
- references/website-crawler.md - Multi-page website crawling
Integration Patterns
Scrape and Analyze
Skills: firecrawl-scraping → parallel-research Use case: Scrape competitor pages, then analyze content strategy Flow:
- Scrape competitor website pages with Firecrawl
- Convert to clean markdown
- Use parallel-research to analyze positioning, messaging, features
Scrape and Document
Skills: firecrawl-scraping → content-generation Use case: Create summary documents from web research Flow:
- Scrape multiple article pages on a topic
- Combine markdown content
- Generate summary document via content-generation
Scrape and Enrich CRM
Skills: firecrawl-scraping → attio-crm Use case: Enrich company records with website data Flow:
- Scrape company website (about page, team page, product pages)
- Extract key information (funding, team size, products)
- Update company record in Attio CRM with enriched data
List & Monetize Your Skill
Submit your Claude Code skill and start earning
Use Cases
Task Automation & Efficiency
Automate repetitive workflows and reduce manual effort
Example
Generate reports, summarize documents, draft communications
Save 3-5 hours per week on routine tasks
Knowledge Enhancement
Learn new skills, understand complex topics, get expert guidance
Example
Explain concepts, provide examples, suggest learning resources
Accelerate learning and skill development by 2x
Quality Improvement
Enhance output quality through reviews, suggestions, and refinements
Example
Review drafts, suggest improvements, catch errors
Improve work quality by 30-40% with less effort
Implementation Guide
Prerequisites
- ›Claude Desktop or compatible AI client with skill support
- ›Clear understanding of task or problem to solve
- ›Willingness to iterate and refine outputs
Time Estimate
15-45 minutes depending on use case complexity
Steps
- 1Install skill using provided installation command
- 2Test with simple use case relevant to your work
- 3Evaluate output quality and relevance
- 4Iterate on prompts to improve results
- 5Integrate into regular workflow if valuable
Common Pitfalls
- ⚠Expecting perfect results without iteration
- ⚠Not providing enough context in prompts
- ⚠Using skill for tasks outside its intended scope
- ⚠Accepting outputs without review and validation
Best Practices
✓ Do
- +Start with clear, specific prompts
- +Provide relevant context and constraints
- +Review and refine all outputs before using
- +Iterate to improve output quality
- +Document successful prompt patterns
✗ Don't
- −Don't use without understanding skill limitations
- −Don't skip validation of outputs
- −Don't share sensitive information in prompts
- −Don't expect skill to replace human judgment
💡 Pro Tips
- ★Be specific about desired format and style
- ★Ask for multiple options to choose from
- ★Request explanations to understand reasoning
- ★Combine AI efficiency with human expertise
When to Use This
✓ Use when
Use when skill capabilities match your task, clear ROI on time saved, and you can validate outputs. Best for repetitive tasks, learning, and quality improvement.
✗ Avoid when
Avoid when task requires deep expertise you can't validate, involves sensitive decisions, or when learning process is more valuable than speed of completion.
Learning Path
- 1Familiarize yourself with skill capabilities and limitations
- 2Start with low-risk, non-critical tasks
- 3Progress to more complex and valuable use cases
- 4Build expertise through regular use and experimentation
Related Skills
typescript-best-practices
96jwynia/agent-skills
java-springboot
42github/awesome-copilot
google-search-console
41kostja94/marketing-skills
python-expert-best-practices-code-review
34wispbit-ai/skills
fastapi-python
29mindrally/skills
backend-architect
15sickn33/antigravity-awesome-skills
Reviews
- KKwame Choi★★★★★Dec 12, 2024
Useful defaults in firecrawl-scraping — fewer surprises than typical one-off scripts, and it plays nicely with `npx skills` flows.
- KKaira Kim★★★★★Dec 12, 2024
We added firecrawl-scraping from the explainx registry; install was straightforward and the SKILL.md answered most questions upfront.
- KKaira Wang★★★★★Dec 12, 2024
Keeps context tight: firecrawl-scraping is the kind of skill you can hand to a new teammate without a long onboarding doc.
- LLucas Perez★★★★★Dec 8, 2024
I recommend firecrawl-scraping for anyone iterating fast on agent tooling; clear intent and a small, reviewable surface area.
- MMateo Nasser★★★★★Dec 4, 2024
firecrawl-scraping is among the better-maintained entries we tried; worth keeping pinned for repeat workflows.
- HHiroshi Srinivasan★★★★★Nov 23, 2024
Useful defaults in firecrawl-scraping — fewer surprises than typical one-off scripts, and it plays nicely with `npx skills` flows.
- SSakshi Patil★★★★★Nov 15, 2024
Solid pick for teams standardizing on skills: firecrawl-scraping is focused, and the summary matches what you get after install.
- SSakura Shah★★★★★Nov 3, 2024
firecrawl-scraping is among the better-maintained entries we tried; worth keeping pinned for repeat workflows.
- KKaira Li★★★★★Nov 3, 2024
firecrawl-scraping fits our agent workflows well — practical, well scoped, and easy to wire into existing repos.
- LLucas Gonzalez★★★★★Nov 3, 2024
firecrawl-scraping has been reliable in day-to-day use. Documentation quality is above average for community skills.
showing 1-10 of 58
Discussion
Comments — not star reviews- No comments yet — start the thread.