Robots.txt Generator and Tester

How to use the Robots.txt Generator

1. Choose a rule for all crawlers: allow everything, block specific paths, or block the whole site (useful for a staging copy).
2. If you are blocking paths, enter one per line, such as /admin/ or /*?sessionid=. Add exceptions in the second box if part of a blocked folder should stay crawlable.
3. Tick any crawlers you want to keep out entirely, such as AI training crawlers, and add any other user-agent names.
4. Add the full address of each sitemap so crawlers can find them.
5. Review the generated file on the right. You can edit it directly, then test any path with the tester to confirm the result.
6. Download the file, name it exactly robots.txt, and upload it to the root of your domain so it is available at https://your-domain/robots.txt.

Example: an online shop

A shop wants search engines to crawl products and categories, but not the basket, checkout or endlessly sortable filter URLs. It also wants to opt out of one AI training crawler:

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?sort=
Allow: /checkout/help/

User-agent: GPTBot
Disallow: /

Sitemap: https://www.example-shop.com/sitemap.xml

Testing /checkout/help/returns for Googlebot returns Allowed, because the longer Allow: /checkout/help/ rule beats the shorter Disallow: /checkout/ rule. Testing /shoes?sort=price returns Blocked by /*?sort=.

What robots.txt does — and what it does not

A robots.txt file is a plain-text set of instructions at the root of a website that tells well-behaved crawlers which URLs they may request. It is mainly used to keep crawlers away from low-value or duplicate URLs, such as internal search results and filter combinations, so they spend their time on pages that matter.

Two misunderstandings cause real problems:

• Blocking is not the same as hiding. A URL blocked in robots.txt can still appear in search results, without a description, if other sites link to it. To keep a page out of search results, allow crawling and add a noindex robots meta tag instead.
• It is not security. The file is public and compliance is voluntary. Never rely on it to protect private content; use authentication.

How the tester decides

The tester follows the Robots Exclusion Protocol as standardised in RFC 9309, which Google and Bing follow:

1. It finds the group whose User-agent line matches the crawler name most specifically. A group for Googlebot takes precedence over the * group; the crawler follows only one group.
2. Within that group, it compares the path with every Allow and Disallow rule, treating * as any sequence of characters and $ as the end of the URL.
3. The matching rule with the longest path wins. If an allow and a disallow rule are equally long, allow wins.
4. If no rule matches, the path is allowed.

Paths are case-sensitive, so /Admin/ and /admin/ are different rules.

About blocking AI crawlers

Several AI companies publish the user-agent names of the crawlers they use to collect training data, and state that those crawlers respect robots.txt. The generator includes a selection of these names. Blocking them only affects crawlers that choose to comply, and some products use separate user-agents for training, search and fetching pages on a user’s request, so check each company’s current documentation if this matters to you.

Limitations to keep in mind

• Crawlers are not obliged to follow robots.txt. Reputable search engines do; malicious bots generally do not.
• The tester implements the standard matching rules used by major search engines. Smaller crawlers may interpret wildcards or conflicting rules differently.
• Crawl-delay is ignored by Google. To reduce Googlebot’s crawl rate, your server can temporarily respond with HTTP 503 or 429 status codes.
• The list of crawler names reflects published names at the time of writing. Companies add and rename crawlers, so review the list periodically.

How your data is handled

The robots.txt file is generated and tested in your browser. Nothing you enter is sent to a server, and the tool does not fetch your live robots.txt file.

Frequently asked questions