Free Robots.txt Generator for Websites
Configure allow/disallow rules, add your sitemap, and optionally block abusive AI scrapers - then copy a production-ready robots.txt for your site root.
Crawling
Generate robots.txt
Allow public pages, disallow private paths, include your sitemap - and optionally block abusive scrapers without hiding from SEO bots.
robots.txt preview
# Generated by ScrollLaunch robots.txt generator # https://www.scrolllaunch.com/tools/robots-txt-generator User-agent: * Allow: / Disallow: /api/ Disallow: /auth/ Disallow: /dashboard/ # Abusive scrapers (optional) - keep SEO bots allowed User-agent: Bytespider Disallow: / User-agent: PetalBot Disallow: / Sitemap: https://www.example.com/sitemap.xml
Tip: after you add a Sitemap: line, verify it with the sitemap checker or build an index with the sitemap index generator. Keep AhrefsBot / SemrushBot allowed if you care about DR - see the Ahrefs DR check.
How to use
How to use Robots.txt
- 1
Set your sitemap URL
Paste the absolute URL to your XML sitemap (for example https://example.com/sitemap.xml).
- 2
Choose disallow paths
List private prefixes such as /api/ or /dashboard/. Leave blank to allow everything public.
- 3
Copy robots.txt
Copy the generated file and publish it at your site root as /robots.txt.
Guide
Robots.txt mistakes that hurt SEO
Do not Disallow: / your whole site unless you truly want to stay out of search. Prefer blocking only private paths like /api/, /admin/, or /auth/.
Always include a Sitemap: line pointing at your XML sitemap so search engines discover URLs faster. Keep robots.txt at https://yoursite.com/robots.txt.
Blocking SEO measurement bots (AhrefsBot, SemrushBot, etc.) can hide your backlinks from the indexes founders use to judge DR - ScrollLaunch keeps those bots allowed on purpose.
Explore more on free ScrollLaunch tools, browse DR-ranked directories, or launch with a dofollow backlink.
Questions
Robots.txt FAQ
Where do I put robots.txt?
Should I block AhrefsBot?
Does robots.txt hide pages from Google?
How do I check the sitemap I referenced?
Sitemap Finder & Checker
Find and validate sitemaps on any website - robots.txt discovery, validity, and URL totals.
Sitemap Index Generator
Build a sitemapindex XML file from a list of child sitemap URLs.
robots.txt Tester
Free robots.txt tester - fetch live rules, list sitemaps, and test if a path is allowed.
Sitemap Link Counter
Free sitemap link counter - count URLs in XML sitemaps and sitemap indexes, including gzipped files.




