What is robots.txt?
robots.txt tells crawlers which paths they may fetch. It does not remove a URL from the index by itself.
SEO measurement bots may crawl public pages. Dashboard and auth pages are deindexed with a meta robots tag, not a robots.txt block.
On ScrollLaunch
More terms
Frequently Asked Questions
What does robots.txt mean?
robots.txt tells crawlers which paths they may fetch. It does not remove a URL from the index by itself.
How does ScrollLaunch use robots.txt?
SEO measurement bots may crawl public pages. Dashboard and auth pages are deindexed with a meta robots tag, not a robots.txt block.


