robots.txt Generator
Build a robots.txt with user-agent groups, allow and disallow rules, sitemaps and presets, plus warnings for the rules that surprise people.
Rule groups
Group 1
Use "*" for every crawler, or list specific agents separated by commas (e.g. "Googlebot, Bingbot").
One path per line. Paths are root-relative; a missing leading slash is added automatically.
One path per line. Use "/" to block the whole site for these agents.
Not honoured by Googlebot — Bing and Yandex do respect it. A warning is added to the output when set.
Must be an absolute URL; a bare hostname gets https:// prepended.
Generated robots.txt
noindex meta tag to keep a page out of an index, and robots.txt to save crawl budget.How it works
- Add one or more rule groups: pick which crawlers (user-agents) they apply to, then list the paths to allow or disallow.
- Optionally add a sitemap URL and a header comment; the robots.txt updates as you type, in your browser.
- Copy the generated file or download it as robots.txt and upload it to the root of your site.
Frequently Asked Questions
Does robots.txt remove pages from Google?
No, and this is the most common misunderstanding. It asks crawlers not to FETCH a URL. A page already indexed, or one linked from elsewhere, can still appear in results — and blocking it actually prevents Google from seeing a noindex tag. To remove a page, allow crawling and use a noindex meta tag.
Is Crawl-delay respected?
Not by Googlebot, which ignores it silently. Bing and Yandex do honour it. The tool emits it if you ask but says plainly that Google needs the crawl rate setting in Search Console instead.
What does an empty Disallow mean?
"Disallow:" with no value means allow everything — it is a meaningful directive, not a blank line, so it is preserved exactly rather than being tidied away.
Is the output stable?
Yes. Groups and rules keep the order you entered them, so regenerating an unchanged configuration produces a byte-identical file and does not show up as a spurious diff.