Guides configuration and auditing of robots.txt for search engine and AI crawler control.
Works with
AI-first code editor with Composer
Before installing skills in Cursor, ensure your development environment meets these requirements:
node --versionrobots-txtExecute the skills CLI command in your project's root directory to begin installation:
Fetches robots-txt from kostja94/marketing-skills and configures it for Cursor.
The CLI shows a list of agents. Use arrow keys and space to select Cursor:
Confirm successful installation by checking the skill directory location:
Restart Cursor to activate robots-txt. Access via /robots-txt in your agent's command palette.
We perform automated surface-level scans (Gen AI Scanner, Socket, Snyk) during installation. These checks detect common vulnerabilities but do not guarantee complete security. Always review skill source code and verify the publisher's reputation before production use.
Skills execute code in your environment. Always review source, verify the publisher, and test in isolation before production.
Submit your Claude Code skill and start earning
Create detailed user stories, acceptance criteria, and feature specs
Example
Generate user stories for 'password reset feature' with acceptance criteria, edge cases, and test scenarios
Reduce spec writing time by 50%, ensure comprehensive coverage
Research competitors, compare features, identify gaps
Example
Analyze 5 competitor products, create feature comparison matrix, suggest differentiation opportunities
Complete competitive research in 2 hours instead of 2 days
Evaluate features using frameworks (RICE, ICE, Kano) and create prioritized backlogs
Example
Score 20 feature ideas using RICE framework, generate prioritized roadmap with rationale
0
total installs
0
this week
309
GitHub stars
0
upvotes
Run in your terminal
0
installs
0
this week
309
stars
Guides configuration and auditing of robots.txt for search engine and AI crawler control.
When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output.
Check for project context first: If .claude/project-context.md or .cursor/project-context.md exists, read it for site URL and indexing goals.
Identify:
https://example.com)| Point | Note |
|---|---|
| Purpose | Controls crawler access; does NOT prevent indexing (disallowed URLs may still appear in search without snippet) |
| Advisory | Rules are advisory; malicious crawlers may ignore |
| Public | robots.txt is publicly readable; use noindex or auth for sensitive content. See indexing |
| Tool | Controls | Prevents indexing? |
|---|---|---|
| robots.txt | Crawl (path-level) | No—blocked URLs may still appear in SERP |
| noindex (meta / X-Robots-Tag) | Index (page-level) | Yes. See indexing |
| nofollow | Link equity only | No—does not control indexing |
| Use | Tool | Example |
|---|---|---|
| Path-level (whole directory) | robots.txt | Disallow: /admin/, Disallow: /api/, Disallow: /staging/ |
| Page-level (specific pages) | noindex meta / X-Robots-Tag | Login, signup, thank-you, 404, legal. See indexing for full list |
| Critical | Do NOT block in robots.txt | Pages that use noindex—crawlers must access the page to read the directive |
Paths to block in robots.txt: /admin/, /api/, /staging/, temp files. Paths to use noindex (allow crawl): /login/, /signup/, /thank-you/, etc.—see indexing.
| Item | Requirement |
|---|---|
| Path | Site root: https://example.com/robots.txt |
| Encoding | UTF-8 plain text |
| Standard | RFC 9309 (Robots Exclusion Protocol) |
| Directive | Purpose | Example |
|---|---|---|
User-agent: |
Target crawler | User-agent: Googlebot, User-agent: * |
Disallow: |
Block path prefix | Disallow: /admin/ |
Allow: |
Allow path (can override Disallow) | Allow: /public/ |
Sitemap: |
Declare sitemap absolute URL | Sitemap: https://example.com/sitemap.xml |
Clean-param: |
Strip query params (Yandex) | See below |
| Do not block | Reason |
|---|---|
| CSS, JS, images | Google needs them to render pages; blocking breaks indexing |
/_next/ (Next.js) |
Breaks CSS/JS loading; static assets in GSC "Crawled - not indexed" is expected. See indexing |
| Pages that use noindex | Crawlers must access the page to read the noindex directive; blocking in robots.txt prevents that |
Only block: paths that don't need crawling: /admin/, /api/, /staging/, temp files.
robots.txt is effective for all measured AI crawlers (Vercel/MERJ study, 2024). Set rules per user-agent; check each vendor's docs for current tokens.
| User-agent | Purpose | Typical |
|---|---|---|
| OAI-SearchBot | ChatGPT search | Allow |
| GPTBot | OpenAI training | Disallow |
| Claude-SearchBot | Claude search | Allow |
| ClaudeBot | Anthropic training | Disallow |
| PerplexityBot | Perplexity search | Allow |
| Google-Extended | Gemini training | Disallow |
| CCBot | Common Crawl (LLM training) | Disallow |
| Bytespider | ByteDance | Disallow |
| Meta-ExternalAgent | Meta | Disallow |
| AppleBot | Apple (Siri, Spotlight); renders JS | Allow for indexing |
Allow vs Disallow: Allow search/indexing bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot); Disallow training-only bots (GPTBot, ClaudeBot, CCBot) if you don't want content used for model training. See site-crawlability for AI crawler optimization (SSR, URL management).
Clean-param: utm_source&utm_medium&utm_campaign&utm_term&utm_content&ref&fbclid&gclid
Make data-driven prioritization decisions faster
Draft PRDs, status updates, and stakeholder presentations
Example
Create executive summary of Q3 roadmap, monthly progress report, feature launch announcement
Save 3-5 hours/week on communication overhead
Prerequisites
Time Estimate
30-60 minutes to see productivity improvements
Steps
Common Pitfalls
✓ Do
✗ Don't
💡 Pro Tips
✓ Use when
Use for user story writing, competitive research, roadmap prioritization, stakeholder communication, and PRD drafting. Best for reducing repetitive documentation and research work.
✗ Avoid when
Avoid for strategic product vision (requires deep customer empathy), pricing decisions (needs market and financial expertise), or when face-to-face customer discovery is more valuable than speed.
kostja94/marketing-skills
mattpocock/skills
parcadei/continuous-claude-v3
cursor/plugins
ailabs-393/ai-labs-claude-skills
ailabs-393/ai-labs-claude-skills
I recommend robots-txt for anyone iterating fast on agent tooling; clear intent and a small, reviewable surface area.
Useful defaults in robots-txt — fewer surprises than typical one-off scripts, and it plays nicely with `npx skills` flows.
Solid pick for teams standardizing on skills: robots-txt is focused, and the summary matches what you get after install.
Keeps context tight: robots-txt is the kind of skill you can hand to a new teammate without a long onboarding doc.
robots-txt reduced setup friction for our internal harness; good balance of opinion and flexibility.
robots-txt is among the better-maintained entries we tried; worth keeping pinned for repeat workflows.
We added robots-txt from the explainx registry; install was straightforward and the SKILL.md answered most questions upfront.
Keeps context tight: robots-txt is the kind of skill you can hand to a new teammate without a long onboarding doc.
robots-txt fits our agent workflows well — practical, well scoped, and easy to wire into existing repos.
robots-txt is among the better-maintained entries we tried; worth keeping pinned for repeat workflows.
showing 1-10 of 40