Robots.txt Generator

The robots.txt file tells search engine crawlers which URLs they can access on your site. Common uses include:

Configuration

User Agent Name

Use * for all bots, or specify: Googlebot, Bingbot, etc.

Disallow Paths (one per line)

Paths that bots should NOT crawl

Allow Paths (one per line)

Paths that bots CAN crawl (overrides disallow)

Crawl Delay (seconds)

Delay between requests in seconds (optional)

Sitemap URLs (one per line)

Full URLs to your sitemap files