Robots.txt Generator
Generate clean, SEO-optimized robots.txt files in real-time. Control how search engines index and crawl your website.
Settings
Add Crawl Rules
Active Rules
| User-Agent | Type | Path | Action |
|---|
Generated robots.txt
About This Tool
Robots.txt Generator
Generate a properly formatted robots.txt file for your website with rules for all major search engine bots.
Why Use This Tool?
- Block search engines from indexing admin panels, login pages, or private directories
- Allow specific bots like Googlebot while blocking harmful scrapers
- Set crawl-delay to prevent bots from overloading your server
- Generate robots.txt for new websites quickly without manual syntax
- Used by webmasters and SEO professionals during site setup
Overview
A robots.txt file tells search engine crawlers which pages they can and cannot access on your website. It is one of the first files crawlers look for when visiting your site. Getting it wrong can accidentally block important pages from Google, or fail to protect sensitive directories from being indexed. Our Robots.txt Generator creates a correctly formatted robots.txt file from a simple form — no need to memorise the syntax. Configure rules for specific bots, set crawl delays, block directories, and add your sitemap URL.
How to Use
-
1
Choose Bot Rules
Select whether to apply rules to all bots (*) or specific crawlers like Googlebot or Bingbot.
-
2
Set Allow/Disallow Paths
Add paths you want to allow or block (e.g., Disallow: /admin/, Disallow: /private/).
-
3
Set Crawl Delay
Optionally add a crawl delay for bots that respect it (e.g., SEMrush, Ahrefs).
-
4
Add Sitemap URL
Include your sitemap URL so search engines can find and index all your pages.
-
5
Copy the Output
Copy the generated robots.txt content and save it as robots.txt in your website's root directory.