XML Sitemap Generator from URL List
A sitemap generator from a URL list is a tool that wraps each URL in the loc element of the sitemaps.org XML…
Coming 9 OctFree robots.txt generator
Build a valid robots.txt file with allow, disallow, sitemap and AI crawler rules in one minute.
Result
0Groups
0Rules
0Bytes
The free robots.txt generator runs in your browser, needs no sign-up and never sends what you type anywhere.
Quick answer
This robots.txt generator builds a ready to upload file from your user agents, allow and disallow paths, and sitemap URL. Rule of thumb: block only crawl waste such as admin, cart and internal search pages, never block CSS or JavaScript, and use noindex rather than robots.txt to keep a page out of Google.
What is a robots.txt generator?
A robots.txt generator is a tool that writes the plain text file crawlers read at the root of your site before they crawl. It groups your rules under each User-agent line, adds Allow and Disallow paths, and appends Sitemap lines; Google follows the most specific matching rule, and on a tie the less restrictive one wins.
How do you use the Robots.txt Generator?
- Choose whether crawlers may access the whole site by default or should be blocked.
- List the paths to disallow, one per line, such as /wp-admin/ or /cart/, and any exceptions to allow.
- Add your full sitemap URL and tick the AI crawlers you want to opt out of, if any.
- Copy or download robots.txt and upload it to the root of your domain, for example https://example.com/robots.txt.
- Open the live file in a browser and check it in the robots.txt report in Google Search Console.
Does robots.txt stop a page from being indexed?
No, robots.txt controls crawling, not indexing, so a blocked URL can still appear in Google if other pages link to it. To keep a page out of results, allow crawling and add a noindex meta tag or X-Robots-Tag header, or password protect it. Google states this in its introduction to robots.txt.
Does Google support crawl-delay in robots.txt?
No, Googlebot ignores the crawl-delay line. Google only supports user-agent, allow, disallow and sitemap. Some other crawlers such as Bingbot do read crawl-delay, so it can still be useful for them. If Googlebot is overloading your server, return 503 or 429 status codes temporarily.
Where should robots.txt be placed?
Robots.txt must sit at the root of the host it controls, such as https://example.com/robots.txt. A file in a subfolder is ignored. Each subdomain and each protocol needs its own file, so blog.example.com needs a separate robots.txt.
Quick reference
| Rule | Example | Effect |
|---|---|---|
| User-agent: * | User-agent: * | Rules below apply to all crawlers without their own group |
| Disallow | Disallow: /cart/ | Blocks crawling of /cart/ and everything under it |
| Allow | Allow: /wp-admin/admin-ajax.php | Reopens a path inside a blocked folder |
| Wildcard * | Disallow: /*?s= | Blocks any URL containing ?s= such as internal search |
| End anchor $ | Disallow: /*.pdf$ | Blocks URLs that end in .pdf |
| Sitemap | Sitemap: https://example.com/sitemap.xml | Tells crawlers where your sitemap is; must be a full URL |
Worked example
Example
Example: a WordPress shop blocks /wp-admin/, /cart/, /checkout/ and /*?s= but allows /wp-admin/admin-ajax.php, and lists https://example.com/sitemap_index.xml. The generator outputs 1 group with 5 rules and 1 sitemap line, about 170 bytes, far below Google's 500 KiB limit.
Frequently asked questions
What should a basic robots.txt contain?
A basic file needs a User-agent line, any Disallow rules for crawl waste, and a Sitemap line with your full sitemap URL. If you do not want to block anything, User-agent: * with an empty Disallow, or no robots.txt at all, both mean everything may be crawled.
Are robots.txt rules case sensitive?
Yes. Paths in Allow and Disallow rules are case sensitive, so Disallow: /File.html does not block /file.html. The User-agent name and directive names are not case sensitive. Copy paths exactly as they appear in your URLs.
Should I block AI crawlers in robots.txt?
It is a business choice. Blocking tokens such as GPTBot, CCBot or Google-Extended asks those systems not to use your pages for training, and well behaved bots respect it. Google-Extended does not affect Google Search rankings or crawling by Googlebot.
Can I block CSS and JavaScript files?
You should not. Google renders pages like a browser, so blocking CSS or JavaScript can stop it from seeing your layout and content, which can hurt how pages are understood and ranked. Only block resources that are truly unimportant, such as tracking scripts.
How big can a robots.txt file be?
Google reads up to 500 KiB of a robots.txt file and ignores anything after that. Most sites need under 2 KB. If your file is large, group URLs into folders or use wildcards instead of listing single pages.
How long until Google sees robots.txt changes?
Google generally caches robots.txt for up to 24 hours, so changes usually apply within a day. You can request a recrawl of the file from the robots.txt report in Search Console. This generator runs in your browser and sends nothing to a server.
Sources
Facts checked against these official pages. Read how I check facts in my editorial policy.
Related free tools
All 104 toolsMeta Robots Tag Generator
A meta robots tag generator is a tool that writes the meta name="robots" element and the equivalent…
Coming 12 OctSecurity.txt Generator
A security.txt generator is a tool that writes the machine readable file defined in RFC 9116, listing fields…
Coming 14 OctWant a full audit instead?
Send your website and I will check it for you, free.