loading
loading
A public file that gives crawler instructions about which paths they may access.
robots.txt can block or allow crawler paths and point to sitemaps. It is advisory, public, and not a security control.
Plainly
Think of robots.txt as the checklist that keeps the app open for real people. A public file that gives crawler instructions about which paths they may access.
In practice
Use it when a change has to survive deploys, users, incidents, analytics, or billing reality. In practice, define the owner, input, output, and failure mode before you rely on it.