OneStopWebTools

SEO & Marketing

How to Create a Robots.txt File (with Examples)

A robots.txt file tells crawlers which parts of your site they may visit. Getting it right is simple, but a single wrong line can hide your whole site from search engines. This guide explains the syntax with safe examples.

Updated · 2 min read

What robots.txt does — and does not do

The file sits at the root of your domain (https://example.com/robots.txt) and contains rules for user-agents, which are the names crawlers identify themselves with. Reputable crawlers such as Googlebot and Bingbot read it before crawling.

Two points are often misunderstood:

  • It controls crawling, not indexing. A blocked page can still appear in search results (usually without a description) if other sites link to it. To keep a page out of results, allow crawling and use a noindex robots meta tag instead.
  • It is not security. The file is public and malicious bots ignore it. Protect private areas with a password.

Common examples

Allow everything (most small sites):

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Block an admin area and internal search results:

User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /admin/public-file.pdf

Sitemap: https://example.com/sitemap.xml

Block a staging copy entirely:

User-agent: *
Disallow: /

Never upload the last example to your live site — it asks every crawler to stay away from every page.

Blocking AI training crawlers

Several AI companies publish the names of their crawlers so site owners can opt out, for example GPTBot, ClaudeBot and CCBot. To block one, add a group for it:

User-agent: GPTBot
Disallow: /

Crawler names change, so check each company’s current documentation from time to time. Blocking these crawlers does not affect Googlebot or Bingbot.

Generate and test your file

  1. Open the Robots.txt Generator and choose a rule for all crawlers: allow everything, block specific paths or block the whole site.
  2. Enter paths to block, one per line, and any exceptions that should stay crawlable.
  3. Tick crawlers to block completely and add your sitemap address.
  4. Use the built-in tester: enter a path such as /blog/my-post/ and confirm it is allowed, then test a path you meant to block.
  5. Download the file, name it exactly robots.txt and upload it to your site’s root folder.

Don’t have a sitemap yet? Create one with the XML sitemap guide.

Matching rules in brief

  • Paths are case-sensitive: /Admin/ and /admin/ are different.
  • * matches any sequence of characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks PDF files.
  • When an Allow and a Disallow rule both match, Google follows the longer, more specific rule.
  • A crawler follows only the most specific group that names it; if it has its own group, the * group no longer applies to it.

Frequently asked questions

Do I need a robots.txt file?

Not strictly. If there is no file, crawlers assume everything is allowed. Having one is still useful to point crawlers to your sitemap.

Does Google follow Crawl-delay?

No. Google ignores Crawl-delay. Bing and some other crawlers respect it.

How quickly do changes take effect?

Google generally caches robots.txt for up to about a day, so changes are usually picked up within 24 hours.

All guides