Robots.txt generator
Build a correct robots.txt with rules, presets and a sitemap line, right in your browser.
Files are processed in your browser and never uploaded.
How to use Robots.txt generator
Add your text
Type or paste your text into the input box. Everything is handled by your own browser, so nothing is sent to a server.
Set the options
Choose User-agent (* = all), Rule type, Disallow Allow, Crawl-delay (optional) and Sitemap URL (optional). The options are visible from the start, not hidden until you paste something.
Read it and keep a copy
The output updates as you type, so there is nothing to submit and nothing to wait for. Copy robots.txt, Don puts the result on your clipboard and Download robots.txt saves it as a file.
What this tool does
User-agent allow and disallow rules
Add as many user-agent blocks as you need, each with its own Allow and Disallow rules, which is how robots.txt is actually structured.
Common presets (WordPress, block admin, etc.)
Presets cover allow everything, block everything, block admin and cart, allow Google only, and a sensible WordPress starting point.
Optional crawl-delay and sitemap
Crawl-delay and a sitemap line are optional, and the sitemap is worth adding because it is how most crawlers find your URL list.
Live output, copy or download
The file is written out as you edit, so you can read exactly what you are about to publish, then copy or download it.
Works offline, no sign-up
Everything is generated in your browser, so an unpublished site structure is never uploaded.
Frequently asked questions
Is anything uploaded?
No. The robots.txt is built in your browser from your choices, with nothing uploaded. It works offline once loaded.
Does robots.txt keep pages private?
No. It only asks well-behaved crawlers what to crawl; it does not stop access or hide content, and the file itself is public. For truly private pages use authentication or the noindex meta tag.
Where do I put the file?
Save it as robots.txt in the root of your site, so it is reachable at your-domain.com/robots.txt. It must be at the root to be recognised.
Is Crawl-delay respected?
Some crawlers honour Crawl-delay, but Google ignores it and uses its own crawl rate settings instead. It is included as optional for the crawlers that do use it.
Should I list my sitemap here?
Yes. Adding a Sitemap line pointing to your sitemap.xml helps search engines discover all your pages, and it is independent of the allow and disallow rules.
Last updated: