Skip to main content
Scrolllaunch LogoScrollLaunch

Free Robots.txt Generator for Websites

Configure allow/disallow rules, add your sitemap, and optionally block abusive AI scrapers - then copy a production-ready robots.txt for your site root.

Crawling

Generate robots.txt

Allow public pages, disallow private paths, include your sitemap - and optionally block abusive scrapers without hiding from SEO bots.

robots.txt preview

# Generated by ScrollLaunch robots.txt generator
# https://www.scrolllaunch.com/tools/robots-txt-generator

User-agent: *
Allow: /
Disallow: /api/
Disallow: /auth/
Disallow: /dashboard/

# Abusive scrapers (optional) - keep SEO bots allowed
User-agent: Bytespider
Disallow: /

User-agent: PetalBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Tip: after you add a Sitemap: line, verify it with the sitemap checker or build an index with the sitemap index generator. Keep AhrefsBot / SemrushBot allowed if you care about DR - see the Ahrefs DR check.

How to use

How to use Robots.txt

  1. 1

    Set your sitemap URL

    Paste the absolute URL to your XML sitemap (for example https://example.com/sitemap.xml).

  2. 2

    Choose disallow paths

    List private prefixes such as /api/ or /dashboard/. Leave blank to allow everything public.

  3. 3

    Copy robots.txt

    Copy the generated file and publish it at your site root as /robots.txt.

Guide

Robots.txt mistakes that hurt SEO

Do not Disallow: / your whole site unless you truly want to stay out of search. Prefer blocking only private paths like /api/, /admin/, or /auth/.

Always include a Sitemap: line pointing at your XML sitemap so search engines discover URLs faster. Keep robots.txt at https://yoursite.com/robots.txt.

Blocking SEO measurement bots (AhrefsBot, SemrushBot, etc.) can hide your backlinks from the indexes founders use to judge DR - ScrollLaunch keeps those bots allowed on purpose.

Explore more on free ScrollLaunch tools, browse DR-ranked directories, or launch with a dofollow backlink.

Questions

Robots.txt FAQ

Where do I put robots.txt?

At the site root: https://yoursite.com/robots.txt. It must be reachable over HTTPS without a login.

Should I block AhrefsBot?

Usually no - if you care about Domain Rating and backlink indexes, keep SEO crawlers allowed. Use our Ahrefs DR check to see how those indexes see your site.

Does robots.txt hide pages from Google?

Disallow stops crawling, not always indexing of URLs discovered elsewhere. Use noindex (meta or header) when a page must stay out of search results.

How do I check the sitemap I referenced?

Paste your domain into the sitemap checker - it reads Sitemap: lines from robots.txt. For multi-file setups, build an index with the sitemap index generator.

Sitemap Finder & Checker

Find and validate sitemaps on any website - robots.txt discovery, validity, and URL totals.

Sitemap Index Generator

Build a sitemapindex XML file from a list of child sitemap URLs.

robots.txt Tester

Free robots.txt tester - fetch live rules, list sitemaps, and test if a path is allowed.

Sitemap Link Counter

Free sitemap link counter - count URLs in XML sitemaps and sitemap indexes, including gzipped files.