Robots.txt generator and tester

Build a robots.txt without syntax mistakes

A robots.txt file sits at the root of a site (https://www.example.com/robots.txt) and tells crawlers which parts they may visit. This robots.txt generator writes the file from groups of rules, flags common mistakes and includes a robots.txt tester: type an address and a crawler name, and it answers “Allowed” or “Blocked” and shows the rule that decided. Nothing is uploaded; everything runs in your browser.

How to use it

1. Start from a preset: “Allow everything”, “Block everything”, “Block an admin folder” or “Block the listed AI crawlers”. Or paste your current file into “Import an existing robots.txt” and click “Import”.
2. In each group, fill in “User-agent”: * for every crawler, or a crawler's technical name. The “Add an example…” list offers common names, which you should check against each crawler's own documentation.
3. Add rules with “+ Add a rule”: pick Disallow or Allow and type a path starting with /. Optionally set a “Crawl-delay”, and at the bottom enter the full address of your sitemap.
4. Review “Things to check”, try a few addresses in the “URL tester”, then click “Download robots.txt” or “Copy”.

Syntax and rule precedence

A group starts with one or more User-agent lines followed by its rules. A crawler obeys a single group: the one carrying its name (case does not matter), otherwise the * group. If its name appears in two groups, their rules are merged. Inside that group, RFC 9309 ignores the order of the lines: the matching rule with the longest path wins, and when two are the same length, Allow wins.

Example for an online shop:

• User-agent: *
• Disallow: /checkout/
• Allow: /checkout/help
• Disallow: /*.zip$

Without $, a rule matches the start of the address: Disallow: /news also blocks /newsletter. Encoded characters are compared after normalisation: /%61dmin equals /admin, but %2F is not the same as /.

Sitemaps, crawling and de-indexing

The Sitemap line gives the full address of your XML sitemap; it does not belong to any group and can be repeated. Keep in mind that Disallow stops crawling, not indexing: a blocked page can still be listed, without a snippet, if other sites link to it. To remove it, leave it crawlable and add a noindex robots meta tag. The file is not a security measure either: it is public, and badly behaved bots ignore it.

AI crawlers and common mistakes

Some artificial intelligence services crawl the web with their own bots under a published technical name. You can allow or block them like any other crawler; the dedicated preset creates a group with the names from the example list and Disallow: /. Those names change over time, so review your file now and then. Frequent mistakes include a Disallow: / left over from a staging site, a path without a leading /, a line with no colon and a file that is not at the root. The generator flags the first three.

Frequently asked questions

Is Crawl-delay honoured?

Not everywhere. The directive is not part of the standard: some crawlers apply it, others ignore it and offer a crawl rate setting in their webmaster tools instead.

Does an empty “Disallow:” block anything?

No. A Disallow: rule with no path matches no address. It is the classic way to say “everything is allowed”, and it is what the “Allow everything” preset writes.

Do I need a robots.txt for each subdomain?

Yes. The file only applies to the host, protocol and port it is served from, so shop.example.com and www.example.com each need their own.