robots.txt Generator
Build a robots.txt file using presets for common platforms and scenarios. Add custom rules, include your sitemap, then test URLs before downloading.
Click to add common rule sets. You can combine multiple presets.
Crawl-delay is respected by some crawlers (Bingbot, Yandex) but not Googlebot. Use it to reduce server load from aggressive crawlers, not for SEO.
Generated robots.txt
0 linesTest a URL against your rules
robots.txt quick guide
User-agent: *
Applies rules to all crawlers. Use specific bot names to target individual crawlers.
Disallow: /path/
Tells bots not to crawl this path. Empty Disallow means allow everything.
Allow: /path/
Overrides a Disallow for a specific path. Useful for allowing a subfolder within a blocked directory.
⚠ Not a security tool
robots.txt only works for compliant crawlers. It does not hide content from users or bad bots.