The Robots.txt Mistakes That Are Killing Your SEO
Last month, a developer reached out to me in panic. Their site had dropped from 10,000 visitors per day to virtually zero overnight. No Google penalty. No algorithm update. Just a single character in their robots.txt file.
The $50,000 Typo
Here's what happened: The developer meant to block a staging directory. They wrote this:
WRONG
User-agent: *
Disallow: /This blocks EVERYTHING. Google can't crawl a single page.
What they meant to write:
CORRECT
User-agent: *
Disallow: /staging/This only blocks the staging directory. Everything else is crawlable.
The site was deindexed for three weeks before they noticed. By the time we fixed it, they'd lost an estimated $50,000 in revenue. All because of a missing "/staging" after the slash.
💡 Lesson Learned:
Always use our Robots.txt Validator before deploying changes. It catches these mistakes instantly.
Mistake #1: Blocking Your CSS and JavaScript
I see this one constantly. Developers think: "Google doesn't need to crawl our assets," so they write:
User-agent: *
Disallow: /css/
Disallow: /js/
Disallow: /assets/Why this hurts you: Google renders JavaScript to understand your content. If it can't access your JS and CSS, it can't see your site properly. You might rank lower or not at all.
Real Example: E-commerce Site
An online store blocked /assets/ to save crawl budget. Their product images loaded via JavaScript. Google couldn't render the pages properly and their rankings tanked.
Fix: We removed the CSS/JS blocks. Rankings recovered within two weeks.
✅ What to do instead:
Don't block CSS, JavaScript, or images. Google needs them to properly render and index your pages. If you're worried about crawl budget, focus on blocking truly irrelevant pages like admin panels and search result pages.
Mistake #2: The Wildcard Confusion
This one's subtle but deadly. A client wanted to block all their old blog URLs that had "?page=" parameters:
❌ What they wrote:
Disallow: /*?page=Looks reasonable, right? Wrong. The asterisk (*) means "match anything," so this accidentally blocked pages like:
- /about-us?page=company
- /products/awesome-widget?page=details
- /contact?page=form
Basically, any URL with "?page=" anywhere got blocked, not just the old blog pagination.
✅ The fix:
# Block only old blog pagination
Disallow: /blog/*?page=
# Or use canonical tags instead
# (Better solution - don't block, just canonicalize)Mistake #3: Forgetting About Subdirectories
Here's a head-scratcher I encountered: A site wanted to allow Google to crawl their blog but block their documentation. They wrote:
User-agent: *
Allow: /blog
Disallow: /docsLooks fine? Not quite. This blocks "/docs" but doesn't block "/documentation", "/docs-archive", or any other variation. Also, "Allow" rules are tricky—they need to be more specific than "Disallow" rules to work.
⚠️ Pro tip:
Robots.txt matches the BEGINNING of URLs. "/docs" blocks "/docs/anything" but NOT "/documentation" or "/v2/docs/". Be specific with your paths.
Mistake #4: No Sitemap Reference
This one won't kill your SEO, but it's a missed opportunity. Most developers forget to add their sitemap to robots.txt:
✅ Always include this:
User-agent: *
Disallow: /admin/
Disallow: /private/
Sitemap: https://yourdomain.com/sitemap.xmlWhy? It tells crawlers exactly where to find your sitemap. Sure, Google can usually find it anyway, but why make them work for it?
Mistake #5: The Case Sensitivity Trap
Here's a sneaky one that got me early in my career. A client's robots.txt blocked "/Admin/" with a capital A:
Disallow: /Admin/Problem? Their actual admin panel was at "/admin/" (lowercase). The robots.txt rule did nothing. Google was happily indexing their admin login page.
⚠️ Remember:
Robots.txt is case-sensitive. "/Admin/" ≠ "/admin/". Always match the exact capitalization of your URLs.
How to Test Your Robots.txt (Without Breaking Production)
Step 1: Use Our Validator
Before deploying any changes, paste your robots.txt into our Robots.txt Validator. It'll catch syntax errors and common mistakes instantly.
Step 2: Test Specific URLs
Our validator also lets you test if a specific URL is blocked. Enter your important pages and make sure they're allowed:
- Homepage: Should be allowed
- Product/service pages: Should be allowed
- Blog posts: Should be allowed
- Admin panel: Should be blocked
Step 3: Check Google Search Console
After deploying, go to Google Search Console → Crawl → robots.txt Tester. Google will show you exactly what they see and let you test individual URLs.
The Robots.txt Template That Actually Works
After helping hundreds of sites with their robots.txt, here's the template I recommend for 90% of websites:
# Allow all search engines to crawl everything
User-agent: *
Disallow:
# Block admin and private areas
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /login/
Disallow: /dashboard/
# Block search result pages (if you have them)
Disallow: /*?s=
Disallow: /search?
# Block tracking and API endpoints
Disallow: /api/
Disallow: /track/
# Tell crawlers where to find your sitemap
Sitemap: https://yourdomain.com/sitemap.xml
# Optional: Block bad bots (be careful with this)
User-agent: AhrefsBot
Crawl-delay: 10
User-agent: MJ12bot
Disallow: /Adjust this for your specific needs, but this covers the basics without accidentally blocking important content.
Common Questions (That I Get Asked All The Time)
Q: Can robots.txt remove pages from Google?
A: No! Common misconception. Blocking a URL in robots.txt stops crawling, but if the page is already indexed (or linked from elsewhere), it might still show up in search results. Use noindex meta tags or X-Robots-Tag headers to actually remove pages.
Q: Should I block duplicate content?
A: Usually no. Use canonical tags instead. Robots.txt is for stuff you NEVER want crawled (like admin panels), not for managing duplicates.
Q: How long does it take for changes to take effect?
A: Google rechecks robots.txt every 24 hours or so. Changes aren't instant, but they're usually reflected within a day.
Don't Let a Typo Cost You Thousands
Look, robots.txt is powerful. It's also dangerous. I've seen too many sites lose traffic because of simple mistakes that could've been caught with two minutes of testing.
Before you deploy that robots.txt change, take 60 seconds and run it through our validator. Your future self (and your traffic graphs) will thank you.