Robots.txt Generator

robots.txt is a plain text file, placed at the root of a domain (yourdomain.com/robots.txt), that tells well-behaved web crawlers which parts of a site they may or may not request. It follows the Robots Exclusion Protocol (REP) - a small set of directives (User-agent, Disallow, Allow, Sitemap, Crawl-delay) that search engines and other bots read before crawling anything else on the site. It controls crawling, not search visibility directly: keeping that distinction straight is the single most useful thing to understand before editing this file.

This generator is for anyone who needs a correct robots.txt without memorizing the syntax rules - site owners publishing their first file, developers migrating a site and needing to protect a staging environment, WordPress or Blogger users pasting custom rules into a settings box, and SEOs auditing or rebuilding an existing file. It builds the file, checks it for the mistakes that most commonly break a site's crawlability, and hands you the finished text.

The tool is split into two simple parts - your steps on the left, the live result on the right. There's nothing to install and nothing is sent anywhere until you choose to copy or download.

1. 1
   Choose your site type (left side)

   Pick WordPress, Blog, E-commerce, or Custom (plus Portfolio, Documentation, and Landing Page under "More site types"). Each pre-fills a sensible starting rule set for that kind of site - Custom starts from a blank group.
2. 2
   Configure your rules (left side)

   Adjust the pre-filled Allow/Disallow rules, add more groups or a Sitemap URL, and open Advanced settings only if you actually need Crawl-delay or Host - most sites don't.
3. 3
   Watch the live result (right side)

   Every change updates the generated file and its verification results immediately - no separate "generate" step. A Valid banner means no issues; otherwise it shows how many issues were found, plus the full list.
4. 4
   Copy or download

   Use Copy robots.txt to paste it into your CMS's custom robots.txt field, or Download robots.txt to get a file ready to upload via FTP to your site's root directory.

Each row in the Verification Results panel is color-coded by severity:

• Green (Passed) - a confirmation that a check found nothing wrong, such as "the entire site is not blocked." No action needed.
• Amber (Warning) - a best-practice recommendation or something worth double-checking, like a missing Sitemap or a Crawl-delay value that Google won't honor. Not a syntax error, but usually worth addressing.
• Red (Error) - a genuine mistake: an accidental full-site block, a path missing its leading slash, or a malformed Sitemap URL. Fix these before publishing.

Only indirectly, and it's easy to overstate. Three distinctions matter:

• Not a direct ranking factor. Google has never described robots.txt as something that influences ranking position. It's a crawl-access control, not a relevance or quality signal.
• Indirect effect through crawl budget. On very large sites, disallowing low-value paths (faceted search filters, internal search results, duplicate parameter URLs) can help crawlers spend more of their limited crawl budget on pages that matter - a real but second-order effect, and mostly irrelevant on small to mid-sized sites.
• Large, direct downside if misused. A single Disallow: / under User-agent: * can stop crawling of an entire site, and that does tank visibility - not because robots.txt is a ranking factor, but because pages that can't be crawled generally can't stay indexed. This is why the verification panel treats a full-site block as an error, not a suggestion.

robots.txt is also where you control AI training and answer-engine crawlers - relevant if you care about AEO/GEO (how your content gets cited by ChatGPT, Claude, Gemini, or Perplexity) as much as classic search. This generator's User-agent dropdown includes the current AI crawler names, but they're worth understanding before you block anything:

• Training vs. retrieval bots are separate. OpenAI's GPTBot crawls to train models; ChatGPT-User and OAI-SearchBot fetch pages live in response to a user's question. Blocking GPTBot stops training-data collection but has no effect on live browsing done by the other two - they're independent User-agent entries with independent rules.
• Google-Extended is not Googlebot. It controls whether your content can be used for Gemini and AI Overviews training/grounding, separately from Googlebot, which controls classic Search indexing. Disallowing Google-Extended does not remove you from Google Search results.
• Blocking has a real tradeoff. Disallowing AI crawlers keeps your content out of model training and reduces load from aggressive scrapers like Bytespider or CCBot, but it can also mean your pages are never cited or quoted by an AI answer - decide per crawler rather than blocking the whole category by default.

Say you're launching an online store and want to keep checkout and account pages out of crawl results while making sure product pages get discovered.

• WordPress sites - disallow /wp-admin/ while allowing /wp-admin/admin-ajax.php (many plugins depend on it staying crawlable), and declare the XML sitemap your SEO plugin generates.
• Blogger/Blogspot sites - paste the generated text into Settings > Crawlers and indexing > Custom robots.txt, since Blogger hosts the file for you rather than accepting an upload.
• Staging and development servers - a Disallow: / block under User-agent: * is one of the only cases where blocking the whole site is exactly correct, since a staging copy indexed alongside the live site creates duplicate-content problems.
• Large catalog or listing sites - disallowing faceted-navigation parameters (?sort=, ?filter=, ?color=) keeps crawlers from spending budget on thousands of near-duplicate filtered URLs.
• Sites managing AI crawler access - a dedicated group for GPTBot, ClaudeBot, or Google-Extended, separate from the general User-agent: * group, keeps that policy decision isolated and easy to change later.

• Blocking the entire site by accident. A bare Disallow: / under User-agent: * left over from a staging environment is the single most damaging robots.txt mistake, and it's easy to miss because the file still "looks fine."
• Assuming Disallow removes a page from search results. It only blocks crawling. A disallowed URL that's already indexed, or that's linked to from elsewhere, can still appear in results - use a noindex tag for that instead, and remember the page must stay crawlable for a search engine to see the noindex tag at all.
• Blocking CSS and JavaScript folders. Older advice suggested disallowing /assets/ or /wp-includes/ wholesale; modern Google rendering needs to fetch your stylesheets and scripts to understand the page layout, so blocking them can hurt how a page is understood.
• Leaving off the leading slash. Disallow: admin/ (no leading /) is not a valid path match and is effectively ignored - it needs to be Disallow: /admin/.
• Getting case sensitivity wrong. Paths are case-sensitive - Disallow: /Private/ does not match /private/. Directive names themselves (Disallow, Allow) are conventionally capitalized but are not case-sensitive.
• Treating robots.txt as a security tool. The file is publicly readable and only respected voluntarily - listing a private path in Disallow can even advertise its existence to anyone reading the file.
• Publishing multiple robots.txt files. Each protocol and subdomain needs its own - http://example.com, https://example.com, and https://shop.example.com each read a separate robots.txt at their own root; one file doesn't cover all of them.

• Keep it as short as the job requires. A handful of well-considered rules is easier to audit and less likely to have a costly mistake than a long file copied from several sources over time.
• Order rules by specificity, not by habit. Google resolves conflicts by picking the longest matching path, not the first or last rule listed - write rules assuming that, rather than relying on ordering to "win."
• Always confirm in Search Console after publishing. The robots.txt Report shows precisely what Google fetched and parsed, which catches server-side issues (a caching layer serving a stale file, a redirect on the URL) that a local generator can't see.
• Don't rely on Crawl-delay for Google traffic. If Googlebot's request volume is a real problem, use the crawl rate settings in Search Console instead - the directive itself is silently ignored.
• Re-check robots.txt after any migration or replatform. A rule that made sense on the old CMS (blocking a URL structure that no longer exists) can silently keep blocking a live path that replaced it.

• Meta robots tag - a per-page <meta name="robots" content="noindex"> tag; unlike robots.txt, it controls indexing directly, but only works on pages crawlers are allowed to reach. Build one with the Meta Tags Generator.
• X-Robots-Tag HTTP header - the same indexing directives as the meta robots tag, sent as a response header instead - the only way to apply noindex to non-HTML files like PDFs.
• Canonical tag - tells search engines which URL is the authoritative version among near-duplicates; a different problem than crawl access, and not a substitute for it.
• XML sitemap - the list of URLs you want crawled and indexed, complementing robots.txt (which mainly says what not to crawl). Declare it here with the Sitemap directive and submit it in Search Console.
• Crawl budget - the practical limit on how many URLs a search engine will crawl on your site in a given period; mainly a concern on very large sites, where trimming low-value paths in robots.txt can help.

• robots.txt is a voluntary standard - it cannot force any crawler to comply, and offers no protection against a bot written specifically to ignore it.
• The file must live at the domain root - /robots.txt, not /folder/robots.txt - or crawlers won't find it and will assume unrestricted access.
• It must be a UTF-8 plain text file served with a 200 status; if it 404s, crawlers generally treat that as "no restrictions," while a 5xx server error can cause crawlers to pause crawling the site entirely.
• robots.txt itself needs to be reachable and un-blocked - a crawler that can't fetch /robots.txt at all may hold off crawling the rest of the site as a precaution.
• This tool checks syntax and common logical mistakes, not your live server configuration - always confirm the final result with Search Console or a direct request to yourdomain.com/robots.txt after publishing.

This generator runs entirely client-side: every group, rule, and directive you enter is processed by JavaScript in your own browser and never transmitted to AudEsto's servers or any third party. Refreshing or closing the tab clears everything, since nothing is stored remotely - copy or download the file before you navigate away if you want to keep it.

Ready to build your file? Scroll up and choose your site type to get started - it only takes a few clicks.