Robots.txt generator

How it works

The file is built from directives, one per line. User-agent names the crawler the following rules apply to, and an asterisk means all of them. Disallow lists paths that must not be fetched, and an empty Disallow means nothing is blocked, which is how a file allows everything. A line starting with a bang in the paths box becomes an Allow line instead, so you can block a folder and reopen one file inside it. Blank lines separate groups and are ignored. The optional Sitemap line takes a full URL and points crawlers at the sitemap; it is not part of any group. Paths must start with a slash. The output is plain text you can copy or download as robots.txt.

Worked example

With "allow everything" selected and the sitemap https://YOURDOMAIN.com/sitemap.xml, the tool writes three lines: User-agent: * then Disallow: with nothing after it, then a blank line and Sitemap: https://YOURDOMAIN.com/sitemap.xml. The empty Disallow is deliberate: it means no path is blocked, which is the usual way to say everything is allowed.

Questions and answers