Are you blocking the engines?

Browsers block cross-origin requests to other sites, so a purely client-side tool cannot fetch your robots.txt for you. Pasting also means nothing is sent anywhere — the entire check runs in your browser.

Open yourdomain.com/robots.txt in a browser and copy everything. Use the live domain, not the file in your repository. This distinction matters more than it sounds.

Several CDNs and hosts now prepend their own managed rules to whatever your origin serves. We found exactly this on our own domain the week we launched — the file we wrote allowed every AI crawler, and the file the internet saw disallowed four of them.

Blocking a crawler prevents that engine citing you. The one arguable exception is a bulk training-corpus scraper with no search or citation surface, since there is no visibility upside. Everything with a user-facing answer product should be allowed.