Robots.txt Generator

Create robots.txt files for search engines

Settings
Disallow Paths:
robots.txt Output

Related SEO Tools

What is robots.txt?

Robots.txt is a text file that tells search engine crawlers which pages to crawl or ignore. Placed in your website's root directory, it uses simple directives like User-agent, Disallow, and Allow to control bot access. It's the first file crawlers check when visiting your site.

Frequently Asked Questions

Where do I put robots.txt?
Place it in your website's root directory (e.g., https://yourdomain.com/robots.txt). It must be publicly accessible. Search engines automatically look for it when crawling your site.
What's the difference between Disallow and Allow?
Disallowing blocks crawlers from accessing specific paths. Allowing overrides disallow rules for specific paths within disallowed directories. Use carefully to avoid blocking important content.
Can robots.txt hide pages from search results?
No, it only prevents crawling. Pages might still appear if linked elsewhere. Use noindex meta tags for pages you want completely out of search results.