Robots.txt Generator

Generate a robots.txt file.

FreeNo accountOn your device

Runs in your browser. How to verify

Loading tool…

About this tool

Set the user-agent, list the paths to disallow and allow, add one or more sitemap URLs, and copy the file. Tick Block everything for a staging site. With empty path boxes the file allows everything.

How to use

1
Step 1
Enter the user-agent (* for all crawlers).
2
Step 2
List disallow and allow paths, one per line, or tick Block everything.
3
Step 3
Add sitemap URLs and an optional crawl-delay.
4
Step 4
Press Generate, copy it and upload it as /robots.txt.

How it works

Each path becomes a Disallow or Allow line under your User-agent. Paths without a leading slash get one. Block everything writes Disallow: /. If you give no rules at all, the file contains an empty Disallow, which means no restrictions. Sitemap lines come last.

Worked example

Hide a cart and admin area

Input
Disallow: cart, /admin; sitemap https://example.com/sitemap.xml
Output
User-agent: * Disallow: /cart Disallow: /admin Sitemap: https://example.com/sitemap.xml
Walkthrough
cart becomes /cart; the sitemap line is separated by a blank line.

Limitations

  • One user-agent group per run; generate again for others.
  • robots.txt is a request, not security.
  • Crawl-delay is ignored by Google.

Questions

How do I block all crawlers?
Tick Block everything. The file will say Disallow: /.
Does it still add /admin?
No. Only the paths you enter are written.
Can I add several sitemaps?
Yes, one URL per line.
Is anything uploaded?
No. The text is built in your browser.

Features

  • Custom user-agent
  • Allow and disallow lists
  • Block-all switch
  • Multiple sitemaps
  • Runs in your browser

Related resources

Guides and videos from around the web—opens on YouTube or the publisher’s site.

  • Intro to Google Search Console

    Google Search Central

    Official GSC intro next to SEO preview and audit helpers.

    Watch on YouTube
  • How to avoid duplicate content

    Google Search Central

    Canonical and uniqueness issues meta tools help you catch.

    Watch on YouTube