Frequently asked questions
Is anything uploaded?
No. The robots.txt is built in your browser from your choices, with nothing uploaded. It works offline once loaded.
Does robots.txt keep pages private?
No. It only asks well-behaved crawlers what to crawl; it does not stop access or hide content, and the file itself is public. For truly private pages use authentication or the noindex meta tag.
Where do I put the file?
Save it as robots.txt in the root of your site, so it is reachable at your-domain.com/robots.txt. It must be at the root to be recognised.
Is Crawl-delay respected?
Some crawlers honour Crawl-delay, but Google ignores it and uses its own crawl rate settings instead. It is included as optional for the crawlers that do use it.
Should I list my sitemap here?
Yes. Adding a Sitemap line pointing to your sitemap.xml helps search engines discover all your pages, and it is independent of the allow and disallow rules.