Robots.txt Generator
Generate a robots.txt file.
About this tool
Set the user-agent, list the paths to disallow and allow, add one or more sitemap URLs, and copy the file. Tick Block everything for a staging site. With empty path boxes the file allows everything.
How to use
How it works
Each path becomes a Disallow or Allow line under your User-agent. Paths without a leading slash get one. Block everything writes Disallow: /. If you give no rules at all, the file contains an empty Disallow, which means no restrictions. Sitemap lines come last.
Worked example
Hide a cart and admin area
- Input
- Disallow: cart, /admin; sitemap https://example.com/sitemap.xml
- Output
- User-agent: * Disallow: /cart Disallow: /admin Sitemap: https://example.com/sitemap.xml
- Walkthrough
- cart becomes /cart; the sitemap line is separated by a blank line.
Limitations
- One user-agent group per run; generate again for others.
- robots.txt is a request, not security.
- Crawl-delay is ignored by Google.
Questions
How do I block all crawlers?
Does it still add /admin?
Can I add several sitemaps?
Is anything uploaded?
Features
- Custom user-agent
- Allow and disallow lists
- Block-all switch
- Multiple sitemaps
- Runs in your browser
Related resources
Guides and videos from around the web—opens on YouTube or the publisher’s site.
Intro to Google Search Console
Google Search Central
Official GSC intro next to SEO preview and audit helpers.
Watch on YouTubeHow to avoid duplicate content
Google Search Central
Canonical and uniqueness issues meta tools help you catch.
Watch on YouTube
- Sitemaps overview
Google Search Central
When sitemap and URL tools matter for discovery.
- The Open Graph protocol
ogp.me
OG tags used by social/SEO preview tools.