Skip to main content

What is robots.txt?

robots.txt tells crawlers which paths they may fetch. It does not remove a URL from the index by itself.

SEO measurement bots may crawl public pages. Dashboard and auth pages are deindexed with a meta robots tag, not a robots.txt block.

On ScrollLaunch

More terms

Frequently Asked Questions

What does robots.txt mean?

robots.txt tells crawlers which paths they may fetch. It does not remove a URL from the index by itself.

How does ScrollLaunch use robots.txt?

SEO measurement bots may crawl public pages. Dashboard and auth pages are deindexed with a meta robots tag, not a robots.txt block.

Browse the full glossary or the FAQ.