Technical SEO
Robots.txt Generator
Assemble robots.txt directives without hand-writing every line. Remember that robots.txt controls crawling, not indexing or privacy. Test carefully before upload.
Use the tool
User-agent: * Disallow: /admin Disallow: /cart Sitemap: https://example.com/sitemap.xml
Features
- Allow and disallow path rules
- User-agent selection
- Sitemap URL lines
- Preview and copy of the final file
How to use it
- Choose the user-agent scope (often * for all bots).
- Add disallow or allow paths as needed.
- Include your XML sitemap URL.
- Copy or download the file and place it at the site root.
FAQ
Does robots.txt stop a page from ranking?
Not by itself. It can block crawling. To keep a page out of results you typically need noindex once the page can be crawled, or other access controls.