robots.txt file checker
Crawlers look at /robots.txt before they look at the HTML. A forgotten Disallow: / for * is how a launch stays invisible. A 404 here is common and not fatal — but then you also have nowhere to declare the sitemap.
What we measure
A separate fetch of {origin}/robots.txt after the page audit. HTML served in that slot (a SPA fallback) is treated as missing. We record status, Sitemap: URLs, user-agents seen, and whether * disallows /.
We do not evaluate every path against every bot. If you need “is /blog allowed for Googlebot?”, that is a full robots parser — this tool answers whether the file exists and what it declares.
Do not confuse it with robots meta
The robots meta (and X-Robots-Tag) lives on the page and can noindex a single URL. robots.txt is the host-wide door. Staging leftovers usually leak through the file, not the meta. Use the other tool for noindex.
What the file can and cannot do
robots.txt controls crawling, not indexing. A URL blocked by Disallow can still appear in results if other pages link to it, shown without a snippet; to keep a page out of the index it must be crawlable and carry a noindex meta or header. The file lives at the root of the host, applies to that host only (www and apex are two files), and is read before the crawler fetches anything else. It is also where the Sitemap: line tells crawlers where the sitemap lives, which is why a 404 here costs more than it seems.
The checker fetches {origin}/robots.txt after the page audit, records the status, the user-agents and Sitemap lines it finds, and whether the * group disallows /. HTML in that slot — a single-page app answering every path with the shell — is reported as missing, because that is what a crawler concludes too.
Common mistakes
A staging file (Disallow: /) shipped to production with the rest of the deploy. Blocking the CSS and JS folders, which stops the engine from rendering the page as a visitor sees it. Using Disallow to hide a page that is already indexed — it stays, now without a description. Forgetting the Sitemap line, or pointing it at a sitemap on a different host, which is ignored.
What the audit of https://example.com/ returned on 5 September 2026
As found
GET https://example.com/robots.txt → 404
<!-- robots_txt_missing (info): allowed by default, but no Sitemap line and no place to shape the crawl -->Fixed
User-agent: *
Allow: /
Disallow: /api/
Sitemap: https://example.com/sitemap.xml
<!-- explicit, small, and it names the sitemap -->A missing file is not an error: everything is allowed. The checker reports it as information because a site with a sitemap and no robots.txt has nowhere to declare it, and because the staging leftover that blocks everything looks exactly like the correct file until someone reads it.
Questions
Will a 404 robots.txt hurt my ranking?
No. Crawlers treat a missing file as “allow everything”. The cost is indirect: no Sitemap line, and no way to keep crawlers out of search results pages, filters and other URL noise.
Does Disallow remove a page from Google?
No. It stops the crawl, and a page that is linked can still be indexed from the links alone. Use a noindex meta tag on a crawlable page to remove it.
Does this tool check whether a specific path is allowed for a specific bot?
No. It reports whether the file exists, which agents it mentions, the sitemap lines and whether * disallows /. Path-by-path evaluation needs a full parser and the bot's exact name.
See the full audit in your workspace