Skip to main content
Scrolllaunch LogoScrollLaunch

Free robots.txt Tester - Fetch & Path Check

Paste a domain to fetch /robots.txt, parse User-agent groups and Sitemap lines, and test whether a path is allowed for a given bot - no signup.

Technical SEO

Test robots.txt allow & disallow

Fetch a site's robots.txt, list Sitemap lines and user-agent groups, and check whether a path is allowed for a given crawler.

Fetches /robots.txt for the host and evaluates Allow/Disallow for a path + user-agent. Need a clean file? Use the robots.txt generator.

How to use

How to use robots.txt Test

  1. 1

    Enter a domain

    Paste example.com - we request https://example.com/robots.txt.

  2. 2

    Optional path test

    Add a path like /dashboard and a user-agent (default *) to see allow vs disallow.

  3. 3

    Review sitemaps

    Confirm Sitemap: URLs resolve, then count URLs with the sitemap link counter.

Guide

Test robots.txt before you block yourself

A bad Disallow can hide /products, /blog, or your whole site from Google. Always test critical paths after edits.

Keep Sitemap: lines accurate and allow SEO measurement bots (AhrefsBot, SemrushBot, etc.) if you care about backlink indexes.

Generate a clean file with the robots.txt generator, then verify it here on production.

Explore more on free ScrollLaunch tools, browse DR-ranked directories, or launch with a dofollow backlink.

Questions

robots.txt Test FAQ

What does a robots.txt tester do?

It downloads /robots.txt, parses User-agent / Allow / Disallow / Sitemap rules, and evaluates whether a path is crawlable for the agent you specify.

How is path matching decided?

We use a simplified Google-style longest-prefix match across Allow and Disallow rules for the matching User-agent group (falling back to *).

Can I generate a robots.txt instead?

Yes - use the robots.txt generator to draft a clean file with a Sitemap line, then re-test it here.

Does blocking a path in robots.txt hide it from Google?

It asks well-behaved crawlers not to fetch the URL. It is not a security control and does not remove already-indexed pages - use noindex or removal tools for that. Pair with a clean sitemap.

Are private or localhost URLs supported?

No. Only public http(s) hosts are allowed.

Robots.txt Generator

Build a clean robots.txt with sitemap URL, disallow paths, and optional AI-scraper blocks.

Sitemap Link Counter

Free sitemap link counter - count URLs in XML sitemaps and sitemap indexes, including gzipped files.

HTTP Header Checker

Free HTTP header checker - inspect status, security headers, caching, and X-Robots-Tag.

llms.txt Generator

Generate llms.txt and a draft ai.txt so answer engines can understand your site.