Robots.txt Generator
Generate a standards-compliant robots.txt file that tells search engines exactly what to crawl.
How it works
The tool assembles User-agent and Allow/Disallow directives, then appends your Sitemap: line. The output follows the Robots Exclusion Protocol so Google, Bing and other crawlers understand it instantly.
Worked example
User-agent: * / Allow: / / Disallow: /admin / Disallow: /private / Sitemap: https://example.com/sitemap.xml — a safe starting point for most sites.
Recommendations
- Block admin, staging and private folders to avoid indexing sensitive pages
- Always reference your sitemap.xml so crawlers find new content faster
- Keep robots.txt small and readable — huge files are ignored by Google
Tips & advice
- Use Disallow: / with no path to block an entire bot, but never block Googlebot globally
- Test your file with Google Search Console → robots.txt Tester
- Add separate User-agent rules for specialized bots like GPTBot if you want to opt out of AI training
Article Schema Generator
Add Article, NewsArticle or BlogPosting structured data to help Google understand and display your content better.
Product Schema Generator
Add Product structured data with prices, availability and ratings to qualify for rich results and Google Shopping.
LocalBusiness Schema Generator
Add LocalBusiness structured data with your address, opening hours and contact details to improve local SEO.
Open Graph Generator
Generate Open Graph meta tags so your links look stunning when shared on social media and messaging apps.
Robots.txt Generator
Generate a standards-compliant robots.txt file that tells search engines exactly what to crawl.
Generate now — FreeRobots.txt Generator — step by step
Pick User-agent: * for all crawlers or target specific bots like Googlebot, Bingbot or GPTBot for finer control.
Enter the folders or files you want to block (for example /admin, /cart). Everything else is allowed by default.
Paste your sitemap location (https://yourdomain.com/sitemap.xml). This line helps search engines discover all your pages.
Click Generate, copy the output into a file named robots.txt and upload it to your website root. Verify it in Google Search Console.
SEO Tools FAQs
robots.txt is a text file placed in your site root that tells search engine crawlers which URLs they may or may not access. It is the first file crawlers check.
No. robots.txt blocks crawling, not indexing. To prevent indexing use a noindex meta tag or X-Robots-Tag header. If Google cannot crawl a page it usually cannot index it either.
Upload it to the root of your domain so it is available at https://yourdomain.com/robots.txt. Most hosts let you place it in the public_html or www folder.
Yes. Add a User-agent: GPTBot or User-agent: CCBot block if you want to opt out of AI model training while keeping Google crawling.
More Free Tools
Explore the whole collection — no signup, 100% free & private.