🤖

robots.txt Generator

Build a valid robots.txt with allow, disallow, crawl-delay and sitemap rules.

Your files stay on this device

Files are processed in your browser, never uploaded. The site downloads code and assets to run the tools. No accounts or trackers.

Use Download to save this tool as one HTML file for offline use.

Privacy →

User-agent

Crawl-delay (seconds)

Allow paths (one per line)

Disallow paths (one per line)

Sitemap URL

output

— output appears here —

Try a worked example

Example

Keep the user agent as User-agent: *, disallow /admin/ and /cart/ on separate lines and add the sitemap https://example.com/sitemap.xml. The file lists both Disallow rules and ends with a Sitemap line.

Keep in mind

robots.txt asks crawlers not to fetch pages; it does not hide them. A blocked URL can still be indexed if other sites link to it, and anyone can read the file, so never list secret paths. To keep a page out of search results, allow crawling and use noindex.

About this tool

Generates a standards-compliant robots.txt file from your crawl rules. The file is built locally in your browser.

Frequently asked questions

Where does robots.txt go?

At the root of your domain, for example https://example.com/robots.txt, so crawlers can find it.

What does Disallow do?

It asks well-behaved crawlers not to fetch the listed paths; one path per line, starting with a slash.