Free tool · SEO and webmaster
Robots.txt generator
Add user-agent groups with allow and disallow paths and your sitemap lines, and the robots.txt is written as you type. Copy it or download it, then test any path against it to see which rule decides. Nothing leaves your browser.
- Free
- No sign-up
- Runs in your browser
- 1 group
- 3 rules
- 1 sitemap
- 120 bytes
Save it as robots.txt (plain text, UTF-8) in the top-level folder of the site, so it is served at https://your-domain/robots.txt. It applies only to the host, protocol and port it is served from. It is a request to well-behaved crawlers, not access control: anyone can read the file, and a URL that is disallowed can still show up in search results without a snippet.
Test a path
BlockedGooglebot fetching /private/report.pdf: Disallow: /private/ is the longest matching rule (9 octets)
- Group used
- User-agent: * (no group for Googlebot)
- Deciding rule
- Disallow: /private/
- Path compared
- /private/report.pdf
| Rule | Match | Octets |
|---|---|---|
Disallow: /private/ | yes, decides | 9 |
Disallow: /tmp/ | no | 5 |
Allow: /private/public/ | no | 16 |
Longest match: of the rules that match, the one with the most octets decides. If an allow and a disallow are equally long, allow wins (RFC 9309 section 2.2.2). The path is compared as typed after the host, so include any query string you want tested.
How to use the Robots.txt Generator
- In Group 1, list the User-agents (* for every crawler), the Disallow paths to keep crawlers out of and any Allow exceptions. Use Add another group for crawlers that need their own rules.
- Add your Sitemap URLs, one per line, as full URLs. Set a Crawl-delay only if a crawler you care about documents that it reads it.
- Under Test a path, type a Crawler name and a Path or URL to test and read whether it is allowed, with the group and rule that decided.
- Press Copy or Download robots.txt and upload it to the top-level folder of your site.
What is a robots.txt file and where does it go?
A robots.txt file tells crawlers which parts of a site they should not fetch. Its format is the Robots Exclusion Protocol, standardized as RFC 9309. The file must be named robots.txt in lowercase, UTF-8 encoded plain text, and sit at the top level of the site, so it is served at https://your-domain/robots.txt. Google notes that it applies only to the host, protocol and port it is served from, so a subdomain needs its own file.
Google generally caches the file for up to 24 hours, so a change can take a day to be noticed. RFC 9309 requires crawlers to read at least the first 500 KiB, and Google ignores anything past that, so keep the file short. The tool warns you if the output goes over that size.
A robots.txt file is a request, not a lock. Well-behaved crawlers follow it, but anyone can read the file and see the paths you list, so do not use it to hide private pages. Google also says a disallowed URL can still appear in search results without a snippet, because blocking a crawl is not the same as blocking indexing.
How do I allow or block a bot or a folder, and how are the rules matched?
The file is made of groups. A group starts with one or more User-agent lines and is followed by Disallow and Allow lines. A crawler looks for the group that names it, ignoring capital letters, and falls back to the * group when none does. RFC 9309 combines several groups that name the same crawler, and Google uses one group per crawler, so the specific group replaces the * group rather than adding to it. A crawler name is made of letters, hyphens and underscores only.
Each path starts with / and is matched from the first character of the URL path. Paths are case sensitive. A * stands for any run of characters, and a $ at the end anchors the match to the end of the URL, so /*.pdf$ blocks every URL that ends in .pdf. When several rules match, the one with the most characters wins, and if an allow and a disallow are equally long, the allow wins. A URL that no rule matches is allowed. An empty Disallow or Allow is ignored.
The default example shows how this works. With Disallow: /private/ (9 characters) and Allow: /private/public/ (16 characters), the URL /private/public/page.html is allowed because the allow rule is longer, while /private/report.pdf is blocked. Googlebot has no group of its own here, so the test falls back to the * group. Be careful with Disallow: /: it asks a crawler to stay out of the whole site, and the tool warns you when you write it.
How do I add a sitemap line, block AI crawlers, and test my robots.txt?
Type your sitemap addresses under Sitemap URLs, one per line. Each must be a fully qualified URL with the protocol and host, and you can list as many as you need. Sitemap lines are not tied to a user-agent group, so the tool writes them after the groups. RFC 9309 mentions a sitemap record only as an example of an extra record that crawlers may read, and Google reads it.
To block a specific crawler, add a group for it with the name from the crawler operator's own documentation, then a Disallow path. This page does not list AI crawler names: they change, and the right name is the one the operator publishes. Crawl-delay is not part of RFC 9309 and Google does not support it, so the tool only writes it if you type a number, and it adds a note saying so.
To test, enter a crawler name and a path or a full URL under Test a path. The tool reads the file it just generated, finds the group for that crawler, lists the rules in it that match, and shows which one decided and why. The path is compared as you type it after the host, so include the query string if you want it tested. The tester does not fetch your live robots.txt, so it shows what this file would do once it is uploaded.
Use it with
- Conversion Rate Calculator: conversion rate and uplift.
- JSON Compare: diff two JSON documents.
- Unix Timestamp Converter: epoch to date and back.
Questions
What is a robots.txt file and where does it go?
It is a plain text file named robots.txt, in UTF-8, that tells crawlers which paths to avoid. Put it in the top-level folder of the site so it is served at https://your-domain/robots.txt. It applies only to the host, protocol and port it is served from.
How do I allow or block a bot or folder?
Start a group with a User-agent line (* for every crawler, or a crawler name) and add Disallow lines for the paths to block and Allow lines for exceptions. Paths start with /, are case sensitive, and the longest matching rule wins.
How do I add a sitemap line?
Add a line such as Sitemap: https://www.example.com/sitemap.xml with the full URL. You can add several, and they are not tied to any user-agent group.
How do I block AI crawlers?
Add a group with the crawler's name as the operator documents it, then Disallow: / or the paths you want to block. This tool does not list AI crawler names, because the name must come from each operator's own documentation.
How do I test my robots.txt?
Use Test a path: enter a crawler name and a path or URL. The tool shows the group used, the matching rules and the rule that decided, using RFC 9309 longest-match matching. It tests the file you generated, not a live file.
Does robots.txt keep a page out of Google?
Not reliably. Blocking a crawl stops Google reading the page, but the URL can still appear in results without a snippet. Google gives separate guidance on blocking indexing.
Sources
How this page was made: the tool's results are computed in your browser by the code on this page, using the sources listed above; the explanations were drafted with AI help and checked against the tool's own output.