# Free Robots.txt Generator for Websites

> Configure allow/disallow rules, add your sitemap, and optionally block abusive AI scrapers - then copy a production-ready robots.txt for your site root.

```yaml
url: "https://www.scrolllaunch.com/tools/robots-txt-generator"
slug: robots-txt-generator
category: onpage
keywords: "robots.txt generator, robots txt generator free, create robots.txt, robots.txt sitemap, block ai crawlers robots.txt, website robots.txt example"
```

## How to use

1. **Set your sitemap URL** - Paste the absolute URL to your XML sitemap (for example https://example.com/sitemap.xml).
2. **Choose disallow paths** - List private prefixes such as /api/ or /dashboard/. Leave blank to allow everything public.
3. **Copy robots.txt** - Copy the generated file and publish it at your site root as /robots.txt.

## Robots.txt mistakes that hurt SEO

Do not Disallow: / your whole site unless you truly want to stay out of search. Prefer blocking only private paths like /api/, /admin/, or /auth/.

Always include a Sitemap: line pointing at your XML sitemap so search engines discover URLs faster. Keep robots.txt at https://yoursite.com/robots.txt.

Blocking SEO measurement bots (AhrefsBot, SemrushBot, etc.) can hide your backlinks from the indexes founders use to judge DR - ScrollLaunch keeps those bots allowed on purpose.

## What you get

- Sitemap URL included
- Custom disallow paths
- Optional AI-scraper blocks
- One-click copy of robots.txt

## Related tools

- **[Sitemap Finder & Checker](https://www.scrolllaunch.com/tools/sitemap-checker)** - Find and validate sitemaps on any website - robots.txt discovery, validity, and URL totals.
- **[Sitemap Index Generator](https://www.scrolllaunch.com/tools/sitemap-index-generator)** - Build a sitemapindex XML file from a list of child sitemap URLs.
- **[robots.txt Tester](https://www.scrolllaunch.com/tools/robots-txt-tester)** - Free robots.txt tester - fetch live rules, list sitemaps, and test if a path is allowed.
- **[Sitemap Link Counter](https://www.scrolllaunch.com/tools/sitemap-link-counter)** - Free sitemap link counter - count URLs in XML sitemaps and sitemap indexes, including gzipped files.

## Related

- [All tools](https://www.scrolllaunch.com/tools)
- [Pricing](https://www.scrolllaunch.com/pricing) - launch free; dofollow while the badge stays on your site.
- [Directories](https://www.scrolllaunch.com/directories) - curated startup directories ranked by DR.