Free Robots.txt Tester
Fetch, parse, and validate any site's robots.txt, then test whether a specific URL is crawlable for Googlebot or any user-agent. See your rules, sitemaps, and warnings at a glance — no signup, instant results.
How crawlers read robots.txt
When a crawler visits your domain it requests /robots.txt before anything else. It then selects the single most specific user-agent group that matches its name — a group naming 'Googlebot' wins over the catch-all '*' group for Google. Within that group, rules are evaluated by longest match: the most specific Allow or Disallow path that matches the URL wins, and when an Allow and a Disallow are the same length, Allow takes precedence.
This precedence is why small mistakes cause big problems. A stray 'Disallow: /' blocks the entire site, and because it's short, a longer 'Allow: /blog/' can still carve out exceptions. Wildcards (*) match any sequence of characters and a trailing $ anchors the end of the URL, so 'Disallow: /*.pdf$' blocks PDF files specifically. The path tester above applies these exact rules so you can confirm behavior before shipping a change.
robots.txt and security: what it does not do
A common and dangerous misconception is that robots.txt hides sensitive paths. The opposite is true: the file is public, and listing 'Disallow: /admin/' or 'Disallow: /backup/' hands attackers a map of exactly where to look. Crawl directives are honored only by well-behaved bots; a malicious scanner ignores them entirely.
Anything that must stay private needs a real access control — authentication, network restrictions, or removing the resource from the public origin — not a Disallow line. If you're auditing a site, treat every path mentioned in robots.txt as a lead worth checking. Running a full Vuln0x scan will probe those endpoints for missing auth, exposure, and misconfiguration rather than trusting that a crawl directive protected them.
Frequently asked questions
- What is a robots.txt file?
- robots.txt is a plain-text file at the root of a domain (yoursite.com/robots.txt) that tells web crawlers which paths they may or may not request. It follows the Robots Exclusion Protocol, standardized as RFC 9309. Each group starts with one or more User-agent lines followed by Allow and Disallow rules. It's a crawling directive, not a security control — it does not stop anyone from visiting a URL directly.
- How does this robots.txt tester work?
- Enter your site URL and the tool fetches yoursite.com/robots.txt server-side, parses every user-agent group and rule, and lists your sitemaps. Optionally enter a path and a user-agent (like Googlebot) and it evaluates, using longest-match precedence, whether that URL would be allowed or blocked — the same way search engines decide.
- Does robots.txt keep a page out of Google?
- Not reliably. Disallow stops compliant crawlers from fetching a page, but a blocked URL can still be indexed without its content if other sites link to it. To keep a page out of the index, allow crawling and use a noindex meta tag or X-Robots-Tag header, or require authentication. Blocking in robots.txt actually prevents Google from seeing the noindex tag.
- Why is my important page blocked by robots.txt?
- The usual causes are an overly broad Disallow rule (for example Disallow: / left over from staging), a rule that matches more than intended because of missing wildcards or anchors, or a more specific user-agent group overriding the * group. Use the path test above with the relevant crawler to see exactly which rule matches, then narrow or add an Allow rule.
- Should robots.txt reference my sitemap?
- Yes. Adding a Sitemap: https://yoursite.com/sitemap.xml line helps crawlers discover all your URLs, especially ones with few internal links. The tester warns you when no Sitemap directive is present.
Go beyond a single check
This tool covers one surface. Vuln0x runs 40+ scanners — headers, TLS, injection, auth, and more — and grades your site in under 60 seconds. Free, no credit card.