Robots.txt Validator
SEO & Crawling Rules
Validate and test your robots.txt file to ensure search engines can properly crawl your website. Prevent SEO disasters from misconfigured robot rules.
On This Page
What is Robots.txt?
The robots.txt file is a text file placed in your website's root directory that tells search engine crawlers (like Googlebot) which pages or sections of your site should or shouldn't be crawled and indexed.
This file follows the Robots Exclusion Standard (REP) and is one of the first things search engine bots check when visiting your website. It's located at https://example.com/robots.txt
Example Robots.txt File
User-agent: * Disallow: /admin/ Disallow: /private/ Allow: /public/ User-agent: Googlebot Crawl-delay: 10 Sitemap: https://example.com/sitemap.xml
Why Validation Matters
A single syntax error in your robots.txt file can have devastating consequences for your SEO:
Complete De-indexing
Accidentally blocking all pages with Disallow: / can remove your entire site from search engines.
Lost Revenue
Blocking important pages from crawling means losing organic traffic and potential customers.
Crawl Budget Waste
Not blocking unimportant pages wastes search engine crawl budget on low-value content.
Security Disclosure
Mentioning sensitive URLs in robots.txt can expose them to attackers (robots.txt is publicly accessible).
How to Use the Validator
Enter Your Domain or Content
You can validate in two ways:
- • URL Method: Enter your domain (e.g., example.com) to fetch and validate your live robots.txt
- • Direct Content: Paste your robots.txt content directly to test before deploying
Review Validation Results
The validator checks for:
- ✓ Syntax errors (invalid directives, typos)
- ✓ Dangerous rules (blocking entire site)
- ✓ Deprecated directives
- ✓ Sitemap presence and format
- ✓ User-agent specificity
- ✓ Path pattern correctness
Fix Issues and Re-validate
Address any errors or warnings, then validate again to ensure compliance.
Try Robots.txt ValidatorRobots.txt Syntax Guide
User-agent
Specifies which crawler the rules apply to. Use * for all crawlers.
User-agent: * # All crawlers User-agent: Googlebot # Only Google's crawler User-agent: Bingbot # Only Bing's crawler
Disallow
Tells crawlers not to access specific paths or files.
Disallow: /admin/ # Block entire admin directory Disallow: /temp.php # Block specific file Disallow: /*.pdf$ # Block all PDF files Disallow: / # Block everything (DANGEROUS!)
Allow
Overrides Disallow rules for specific paths (mainly used by Google).
User-agent: * Disallow: /admin/ Allow: /admin/public/ # Allow this subdirectory
Crawl-delay
Sets delay between requests in seconds (not supported by Google).
User-agent: * Crawl-delay: 10 # Wait 10 seconds between requests
Sitemap
Points crawlers to your XML sitemap for better indexing.
Sitemap: https://example.com/sitemap.xml Sitemap: https://example.com/sitemap-images.xml
Common Mistakes to Avoid
Typos in Directives
Wrong: Useragent: or Dissallow:
Correct: User-agent: and Disallow:
Accidentally Blocking Everything
Wrong:
User-agent: * Disallow: /
This blocks your entire website from all search engines!
Missing Trailing Slash
Disallow: /admin blocks /admin, /admin/, /admin.php, /administrator
Disallow: /admin/ blocks only the /admin/ directory
Wrong File Location
Wrong: /subdirectory/robots.txt or /Robots.txt
Correct: /robots.txt (root directory, lowercase)
Using robots.txt for Security
DON'T list sensitive URLs in robots.txt - it's publicly accessible!
Use proper authentication and access controls instead.
Best Practices
Always Include a Sitemap
Add your sitemap URL to help search engines discover all your pages efficiently.
Test Before Deploying
Always validate your robots.txt with our tool before uploading to your server. Use Google Search Console's robots.txt tester as well.
Keep It Simple
Only block what's necessary. Over-complicated robots.txt files are error-prone and hard to maintain.
Monitor Search Console
Regularly check Google Search Console for crawl errors and blocked resources that shouldn't be blocked.
Use Comments
Add comments (using #) to document why certain rules exist, making maintenance easier.