Free tool · runs in your browser
robots.txt and sitemap.xml generator
robots.txt tells crawlers which areas to visit and sitemap.xml gives them a list of your important pages. This tool builds both and lets you decide separately about AI crawlers such as GPTBot, ClaudeBot and Google-Extended. The files are built in your browser.
robots.txt file
Search and answer crawlers such as OAI-SearchBot and PerplexityBot are not affected by this option.
User-agent: *
Allow: /
sitemap.xml file
Relative addresses (such as /about) are completed with the site address.
0 addresses in the sitemap
Enter addresses to build the sitemap.
How to use it
- Enter your site address, the paths crawlers should skip, and your decision about AI crawlers.
- For the sitemap, write one page address per line.
- Download the files and place them at the site root (for example example.com/robots.txt).
- Submit the sitemap in Search Console.
Frequently asked questions
Does robots.txt remove a page from results?
Not necessarily. robots.txt blocks crawling, not indexing. To keep a page out of results use noindex and leave the page open to crawlers.
Should I block AI crawlers?
It depends on your goal. If you want to appear in ChatGPT and Claude answers, allow them. If you do not want your content used to train models, block them. This tool builds either.
How many pages should the sitemap list?
All pages you want in Google that are publicly reachable. One file supports up to 50,000 URLs.
More free tools
- Free
Free website SEO and health check
Enter your address and see your basic SEO, response speed and the issues that matter in seconds.
Use the tool - Free
Meta tag and Open Graph generator
Write your title and description, preview the Google result and copy ready-made HTML.
Use the tool - Free
FAQ schema (JSON-LD) generator
Enter your page's questions and answers and build FAQPage JSON-LD.
Use the tool
Next step
Leave the fixing to us
A tool shows the problem; doing the fix properly is our team's job.