Web Scraping with just-scrape
AI-powered web scraping CLI by ScrapeGraph AI. Get an API key at dashboard.scrapegraphai.com.
Setup
Always install or run the @latest version to ensure you have the most recent features and fixes.
npm install -g just-scrape@latest
pnpm add -g just-scrape@latest
yarn global add just-scrape@latest
bun add -g just-scrape@latest
npx just-scrape@latest --help
bunx just-scrape@latest --help
export SGAI_API_KEY="sgai-..."
API key resolution order: SGAI_API_KEY env var → .env file → ~/.scrapegraphai/config.json → interactive prompt (saves to config).
Command Selection
| Need |
Command |
| Extract structured data from a known URL |
smart-scraper |
| Search the web and extract from results |
search-scraper |
| Convert a page to clean markdown |
markdownify |
| Crawl multiple pages from a site |
crawl |
| Get raw HTML |
scrape |
| Automate browser actions (login, click, fill) |
agentic-scraper |
| Generate a JSON schema from description |
generate-schema |
| Get all URLs from a sitemap |
sitemap |
| Check credit balance |
credits |
| Browse past requests |
history |
| Validate API key |
validate |
Common Flags
All commands support --json for machine-readable output (suppresses banner, spinners, prompts).
Scraping commands share these optional flags:
--stealth — bypass anti-bot detection (+4 credits)--headers <json> — custom HTTP headers as JSON string--schema <json> — enforce output JSON schema
Commands
Smart Scraper
Extract structured data from any URL using AI.
just-scrape smart-scraper <url> -p <prompt>
just-scrape smart-scraper <url> -p <prompt> --schema <json>
just-scrape smart-scraper <url> -p <prompt> --scrolls <n>
just-scrape smart-scraper <url> -p <prompt> --pages <n>
just-scrape smart-scraper <url> -p <prompt> --stealth
just-scrape smart-scraper <url> -p <prompt> --cookies <json> --headers <json>
just-scrape smart-scraper <url> -p <prompt> --plain-text
just-scrape smart-scraper https://store.example.com/shoes -p "Extract all product names, prices, and ratings"
just-scrape smart-scraper https://news.example.com -p "Get headlines and dates" \
--schema '{"type":"object","properties":{"articles":{"type":"array","items":{"type":"object","properties":{"title":{"type":"string"},"date":{"type":"string"}}}}}}' \
--scrolls 5
just-scrape smart-scraper https://app.example.com/dashboard -p "Extract user stats" \
--stealth
Search Scraper
Search the web and extract structured data from results.
just-scrape search-scraper <prompt>
just-scrape search-scraper <prompt> --num-results <n>
just-scrape search-scraper <prompt> --no-extraction
just-scrape search-scraper <prompt> --schema <json>
just-scrape search-scraper <prompt> --stealth --headers <json>
just-scrape search-scraper "Best Python web frameworks in 2025" --num-results 10
just-scrape search-scraper "React vs Vue comparison" --no-extraction --num-results 5
just-scrape search-scraper "Top 5 cloud providers pricing" \
--schema '{"type":"object","properties":{"providers":{"type":"array","items":{"type":"object","properties":{"name":{"type":"string"},"free_tier":{"type":"string"}}}}}}'
Markdownify
Convert any webpage to clean markdown.
just-scrape markdownify <url>
just-scrape markdownify <url> --stealth
just-scrape markdownify <url> --headers <json>
just-scrape markdownify https://blog.example.com/my-article
just-scrape markdownify https://protected.example.com --stealth
just-scrape markdownify https://docs.example.com/api --json | jq -r '.result' > api-docs.md
Crawl
Crawl multiple pages and extract data from each.
just-scrape crawl <url> -p <prompt>
just-scrape crawl <url> -p <prompt> --max-pages <n>
just-scrape crawl <url> -p <prompt> --depth <n>
just-scrape crawl <url> --no-extraction --max-pages <n>
just-scrape crawl <url> -p <prompt> --schema <json>
just-scrape crawl <url> -p <prompt> --rules <json>
just-scrape crawl <url> -p <prompt> --no-sitemap
just-scrape crawl <url> -p <prompt> --stealth
just-scrape crawl https://docs.example.com -p "Extract all code snippets" --max-pages 20 --depth 3
just-scrape crawl https://example.com -p "Extract article titles" \
--rules '{"include_paths":["/blog/*"],"same_domain":true}' --max-pages 50
just-scrape crawl https://example.com --no-extraction --max-pages 10
Scrape
Get raw HTML content from a URL.
just-scrape scrape <url>
just-scrape scrape <url> --stealth
just-scrape scrape <url> --branding
just-scrape scrape <url> --country-code <iso>
just-scrape scrape https://example.com
just-scrape scrape https://store.example.com --stealth --country-code DE
just-scrape scrape https://example.com --branding
Agentic Scraper
Browser automation with AI — login, click, navigate, fill forms. Steps are comma-separated strings.
just-scrape agentic-scraper <url> -s <steps>
just-scrape agentic-scraper <url> -s <steps> --ai-extraction -p <prompt>
just-scrape agentic-scraper <url> -s <steps> --schema <json>
just-scrape agentic-scraper <url> -s <steps> --use-session
just-scrape agentic-scraper https://app.example.com/login \
-s "Fill email with [email protected],Fill password with secret,Click Sign In" \
--ai-extraction -p "Extract all dashboard metrics"
just-scrape agentic-scraper https://example.com/wizard \
-s "Click Next,Select Premium plan,Fill name with John,Click Submit"
just-scrape agentic-scraper https://app.example.com \
-s "Click Settings" --use-session
Generate Schema
Generate a JSON schema from a natural language description.
just-scrape generate-schema <prompt>
just-scrape generate-schema <prompt> --existing-schema <json>
just-scrape generate-schema "E-commerce product with name, price, ratings, and reviews array"
just-scrape generate-schema "Add an availability field" \
--existing-schema '{"type":"object","properties":{"name":{"type":"string"},"price":{"type":"number"}}}'
Sitemap
Get all URLs from a website's sitemap.
just-scrape sitemap <url>
just-scrape sitemap https://example.com --json | jq -r '.urls[]'
History
Browse request history. Interactive by default (arrow keys to navigate, select to view details).
just-scrape history <service>
just-scrape history<
