Robots.txt Generator
Generate robots.txt file
Web & SEO
Instant results
Rule #1
/admin//private/
User-agent: * Disallow: /admin/ Disallow: /private/
Important Notes:
- Place robots.txt in your website root (e.g., example.com/robots.txt)
- robots.txt is public - don't hide sensitive URLs in it
- Use Allow to create exceptions within Disallowed paths
- Not all bots respect robots.txt - use authentication for sensitive content
About Robots.txt Generator
Generate robots.txt files to control how search engine crawlers access your website. Set rules for different user agents, specify allowed and disallowed paths, and include your sitemap URL.