Robots.txt tester

One file, eight URLs, two crawlers

This is the sample the Load sample button fills in. It looks harmless, and most of it is. The trap is the second group.

User-agent: *
Disallow: /cart
Disallow: /*?sort=
Allow: /blog/
Disallow: /blog/drafts/

User-agent: Googlebot
Disallow: /search
Allow: /search/help$

Sitemap: https://example.com/sitemap.xml

Here is what this page reports for each URL, with the line that decides it:

The first row is the one that bites people. Someone adds a Googlebot group to block internal search, and the cart, checkout, and every other * rule quietly stop applying to Google. If you want both, copy the shared rules into the Googlebot group.

How a verdict gets decided

1. Pick one group. The crawler looks for a User-agent line with its own name, case-insensitive. Google's specialised crawlers such as Googlebot-Image and Googlebot-News fall back to a googlebot group if there is no group for them. Everyone else falls back to *. If several groups name the same crawler, their rules are merged.
2. Trim the URL. Only the path and query string are compared. https://example.com/a/b?x=1#top becomes /a/b?x=1. Non-ASCII characters are percent-encoded on both sides, so /café in a rule matches /caf%C3%A9 in a URL.
3. Find every rule that matches. Rules match from the start of the path. * stands for any run of characters, and a trailing $ pins the rule to the end. Matching is case-sensitive, so /Admin does not cover /admin.
4. Longest rule wins. Length is counted in characters of the rule as written, wildcards included. On an exact tie, Allow wins.
5. No match means allowed. So does an empty Disallow: line, and so does /robots.txt itself.

Patterns that do not mean what they look like

The trailing slash row catches folder blocks: Disallow: /private/ leaves /private crawlable. If that bare URL exists and redirects, it usually does not matter. If it serves content, block it too.

What this tester will not catch

• The file Google actually has. You test what you paste or what ToolPetalBot fetched just now. Google caches robots.txt for up to a day, some servers answer Googlebot differently, and a 5xx or redirect loop on the live file changes behaviour. Search Console's robots.txt report shows the version Google fetched.
• Indexing. Allowed means crawlable, nothing more. A page can be allowed and still carry noindex, or be blocked and still show up as a bare URL.
• Non-Google quirks. Some crawlers ignore wildcards, read rules top to bottom, or treat Crawl-delay as binding. The verdicts here use Google's documented logic.
• Subdomains and protocols. Each host has its own file. shop.example.com does not read example.com/robots.txt, and the http and https versions can differ.
• Bad actors. robots.txt is a polite request. Scrapers that ignore it are not stopped by anything on this page.

The tester does flag lines Google skips: rules before any User-agent, unknown directives, Crawl-delay, and the long-dead Noindex: directive. It also warns when a file passes Google's 500 KiB read limit.

Test robots.txt rules in code

All three libraries follow the Google and RFC 9309 rules: * and $ wildcards, and the longest matching rule wins. Each prints false, true, false for the three URLs.

JavaScript: robots-parser

import robotsParser from "robots-parser"; // npm i robots-parser

const txt = `User-agent: *
Disallow: /admin/
Allow: /admin/public/
Disallow: /*.pdf$`;

const robots = robotsParser("https://example.com/robots.txt", txt);
console.log(robots.isAllowed("https://example.com/admin/users", "Googlebot"));      // false
console.log(robots.isAllowed("https://example.com/admin/public/a", "Googlebot"));   // true
console.log(robots.isAllowed("https://example.com/files/report.pdf", "Googlebot")); // false
console.log(robots.getMatchingLineNumber("https://example.com/admin/users", "Googlebot")); // 2

Python: Protego

from protego import Protego  # pip install protego (supports * and $ like Google)

txt = """User-agent: *
Disallow: /admin/
Allow: /admin/public/
Disallow: /*.pdf$"""

rp = Protego.parse(txt)
print(rp.can_fetch("https://example.com/admin/users", "Googlebot"))       # False
print(rp.can_fetch("https://example.com/admin/public/a", "Googlebot"))    # True
print(rp.can_fetch("https://example.com/files/report.pdf", "Googlebot"))  # False

Go: grobotstxt, a port of Google's parser

package main

import (
	"fmt"

	"github.com/jimsmart/grobotstxt" // Go port of Google's robots.txt parser
)

func main() {
	txt := "User-agent: *\nDisallow: /admin/\nAllow: /admin/public/\nDisallow: /*.pdf$\n"

	fmt.Println(grobotstxt.AgentAllowed(txt, "Googlebot", "https://example.com/admin/users"))       // false
	fmt.Println(grobotstxt.AgentAllowed(txt, "Googlebot", "https://example.com/admin/public/a"))    // true
	fmt.Println(grobotstxt.AgentAllowed(txt, "Googlebot", "https://example.com/files/report.pdf"))  // false
}

Python's built-in urllib.robotparser is not a good stand-in here. It has no * or $ wildcards and applies the first matching rule instead of the most specific one, so with this file it reports report.pdf as allowed and /admin/public/a as blocked.

Each snippet was run on October 14, 2026 with robots-parser 3.0.1, Protego 0.7.0 and grobotstxt v1.0.3.

Robots Exclusion Protocol sources

• RFC 9309: Robots Exclusion Protocol
• RFC 9309, section 2.2.2: allow and disallow lines: the most specific (longest) match wins
• Google: how Google interprets robots.txt
• google/robotstxt: Google's open-source C++ parser
• robots-parser (JavaScript)
• Protego (Python)
• grobotstxt (Go)

FAQ