# What is robots.txt?

robots.txt tells crawlers which paths they may fetch. It does not remove a URL from the index by itself.

SEO measurement bots may crawl public pages. Dashboard and auth pages are deindexed with a meta robots tag, not a robots.txt block.

| Field | Value |
| --- | --- |
| term | robots.txt |
| canonical | https://www.scrolllaunch.com/glossary/robots-txt |
| kind | defined term |

## On ScrollLaunch

- [robots.txt tester](https://www.scrolllaunch.com/tools/robots-txt-tester)