The official store for JNews, a WordPress theme for news, blog and magazine websites. Build a successful publishing website with JNews.
SEO Tools
Robots.txt Validator & Rule Tester
Inspect robots.txt syntax and test crawler rules with line-by-line explanations. Files, paths, and reports stay in your browser.
Your robots.txt
Paste rules or open a local UTF-8 file. Checks and test results update automatically.
Tool limits: 500 KiB UTF-8, 10,000 source lines, 3,000 valid rules, and 4,096 characters per line. Unsupported or malformed lines are reported and ignored. Nothing is fetched or uploaded.
Test crawler access rules
Supply an origin to reject URLs from other hosts, protocols, or ports. Leave blank to test paths without verifying site scope. URLs are inputs, never requests.
Up to 100 targets and 64 KiB total. Queries participate in matching; fragments are ignored. URLs use browser normalization, including dot segments. Use %23 for a literal #.
Model: Google-style REP matching. The most specific case-insensitive product-token match selects groups; repeated selected groups merge, and wildcard groups are fallback only. Custom tokens use this same model, which may differ from the crawler itself.
Groups
Valid rules
Issues
Review notes
Allowed / blocked describes supplied rules only. It does not confirm live access, indexing, rankings, or AI visibility.
Rule test results
Validation findings
Parsed crawler groups
Group boundaries follow Allow/Disallow records, not blank lines. Sitemap and unsupported directives do not split a group. Showing up to 100 groups.
How to interpret the results
How does a rule win?
Paths are case-sensitive and match from the beginning. An asterisk matches zero or more characters; a final dollar sign anchors the end, including the query string. The longest normalized rule pattern wins, with Allow winning equal-priority conflicts. Percent-encoded unreserved ASCII is decoded; escaped reserved characters remain escaped, and Unicode uses UTF-8 percent encoding. Results show matching rules and source lines.
What is outside this simulation?
Only supported User-agent, Allow, Disallow, and Sitemap records are inspected. Crawl-delay, Host, Noindex, and other extensions are reported but not applied. Nonstandard paths are ignored and reported; real crawlers may recover them differently. This tool does not reproduce every crawler implementation, full HTTP User-Agent alias, HTTP status handling, caching, redirects, or server configuration.
Does blocked mean noindex or secure?
No. Robots.txt requests crawl restrictions; it is not authentication and does not reliably prevent a URL appearing in search. This tool cannot confirm whether a crawler obeys the file or whether content appears in AI answers. Protect private content with actual access controls.
Where do my files and results go?
Your inputs stay in this browser tab. Local files use strict UTF-8 decoding. Processing, copying, and report downloads happen on your device, without a processing endpoint, external requests, or browser storage. Results clear when inputs change. Large combinations may hit the documented work limit and require fewer test paths.
References: Google robots.txt specification · RFC 9309.