Robots.txt Generator
Build a robots.txt file
Open, runs in your browserFree tools
Sitemaps, meta tags and checks for site owners.
Build a robots.txt file
Open, runs in your browserBuild campaign tracking links
Open, runs in your browserA robots.txt file asks crawlers to stay out of certain paths; a sitemap lists the pages you want found. Both are plain text files at fixed addresses on your domain, and both are easy to get slightly wrong: one stray slash in a Disallow line can block a whole site. The generators here build the file from your choices and explain each line they write.
Robots rules are requests, not locks. Well-behaved search engines follow them, but a blocked page can still appear in results if other sites link to it, and private content needs a login or a noindex rule instead.
Search engines offer their own testing tools in their webmaster consoles. Check a new file there after uploading it, and keep a copy of the old one so you can roll back.