Grokipedia

✓ Do

• Check robots.txt and respect crawl rules
• Rate limit requests: 1-2 requests/second maximum
• Use official APIs when available instead of scraping
• Identify your bot with descriptive user agent
• Cache results to minimize repeated requests
• Handle errors gracefully with retries and fallbacks
• Validate extracted data for accuracy

✗ Don't

• Don't scrape sites that explicitly forbid it (robots.txt, ToS)
• Don't overwhelm servers with rapid requests—use rate limiting
• Don't scrape personal data without consent and legal basis
• Don't ignore copyright on extracted content
• Don't assume HTML structure is stable—handle changes
• Don't use scraped data for commercial purposes without permission

Protocols

• HTTP/HTTPS
• WebSocket (for real-time sites)
• Puppeteer/Playwright (for JavaScript sites)

Compatibility

• Static HTML sites
• JavaScript-rendered SPAs (with headless browser)
• REST APIs
• GraphQL endpoints

✓ Use when

Use for research automation, content monitoring, data aggregation from multiple sources, and when official APIs don't exist. Best for read-only information gathering.

✗ Avoid when

Avoid for sites with APIs (use API instead), sites that explicitly forbid scraping, when data is copyrighted, or for login-required content without proper authorization.