Where does robots.txt go?

At your domain root, so it loads at https://yoursite.com/robots.txt. Crawlers check that exact path before fetching anything else — a robots.txt anywhere else is ignored.

Can I block AI crawlers but keep Google?

Yes. Each crawler has its own user-agent — GPTBot, ClaudeBot, Googlebot, and so on — so you can disallow the AI bots while leaving search bots fully allowed, or any mix you like.

Does Disallow guarantee a page stays private?

No. robots.txt is a request, not a lock. Well-behaved crawlers honour it, but it does not stop access or guarantee a URL won't appear in results. Use real auth for anything sensitive.

// free tool

Free robots.txt generator

robots.txt tells crawlers which parts of your site they can access. You can allow or block AI crawlers like GPTBot and ClaudeBot exactly the same way as Googlebot — one directive per agent. Configure your rules below and download the file.

Preview

User-agent: *
Allow: /
Disallow: /admin

User-agent: Googlebot
Allow: /
Disallow: /admin

User-agent: Bingbot
Allow: /
Disallow: /admin

User-agent: GPTBot
Allow: /
Disallow: /admin

User-agent: ClaudeBot
Allow: /
Disallow: /admin

User-agent: PerplexityBot
Allow: /
Disallow: /admin

User-agent: Google-Extended
Allow: /
Disallow: /admin

User-agent: CCBot
Allow: /
Disallow: /admin

// place robots.txt at your domain root: example.com/robots.txt

// next step

Your robots.txt controls who gets in — but once they’re in, can they actually read you? Rankport audits how legible your site is to Google and the AI engines, page by page.

Run a free Rankport audit →

What is a robots.txt file?

robots.txt is a plain-text file at your domain root that gives crawlers their rules of engagement: which paths they may fetch and which to leave alone. Every bot — search engines and the newer AI crawlers alike — checks it first, so it’s your front door for controlling automated access.

How to use this generator

Choose which crawlers to allow or block, add any paths you want kept out of bounds, and the tool writes the matching directives. Download the result, save it as robots.txt, and place it at your web root so it serves at /robots.txt.

FAQ

Where does robots.txt go?: At your domain root, so it loads at https://yoursite.com/robots.txt. Crawlers check that exact path before fetching anything else — a robots.txt anywhere else is ignored.
Can I block AI crawlers but keep Google?: Yes. Each crawler has its own user-agent — GPTBot, ClaudeBot, Googlebot, and so on — so you can disallow the AI bots while leaving search bots fully allowed, or any mix you like.
Does Disallow guarantee a page stays private?: No. robots.txt is a request, not a lock. Well-behaved crawlers honour it, but it does not stop access or guarantee a URL won't appear in results. Use real auth for anything sensitive.