The robots.txt file is one of the most powerful yet often overlooked tools for SEO. When used correctly, it can improve your site’s crawl efficiency, prevent indexing of sensitive pages, and enhance overall SEO performance. However, a poorly configured robots.txt can block search engines from accessing important content, which can negatively impact rankings.
In this guide, we’ll cover:
- What robots.txt is and how it works
- How to create and edit robots.txt
- Best practices for SEO
- Advanced robots.txt hacks
What is Robots.txt and How Does It Work?
The robots.txt file is a simple text file located in your website’s root directory. It tells search engine crawlers (like Googlebot) which pages or sections they can or cannot access.
Example of a basic robots.txt file:
User-agent: *
Disallow: /private/
This blocks all crawlers from accessing the /private/ directory.
Key Components of Robots.txt:
- User-agent: Specifies the search engine bot (e.g., Googlebot, Bingbot, * for all).
- Disallow: Blocks bots from specific URLs.
- Allow: (Only for Googlebot) Overrides a Disallow rule for a specific path.
- Sitemap: Links to your XML sitemap for better crawling.
How to Create and Edit Robots.txt
Step 1: Locate or Create Robots.txt
- Check if your website has a robots.txt file
- Visit https://yourwebsite.com/robots.txt in your browser.
- If you see a file, you already have one.
- If you don’t have a robots.txt file:
- Open a plain text editor (e.g., Notepad, VS Code).
- Save a new file as robots.txt.
- Upload it to your website’s root directory via FTP, cPanel, or a file manager.
Step 2: Edit Robots.txt for SEO
Here are some common use cases and best practices:
Allow Full Access (Not Recommended)
If you want all bots to access everything:
User-agent: *
Disallow:
Why avoid this?
It allows search engines to crawl everything, including private pages.
Block Admin & Sensitive Pages
User-agent: *
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Best for: E-commerce sites, WordPress, or membership sites to prevent unnecessary crawling.
Prevent Indexing of Duplicate Content
User-agent: *
Disallow: /tag/
Disallow: /category/
Disallow: /search/
Best for: Blogs, forums, and e-commerce sites to avoid duplicate content issues.
Allow Crawling of Important Pages in Disallowed Directories
For example, WordPress blocks /wp-admin/ but allows access to AJAX:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Why? This ensures JavaScript and dynamic content function properly.
Prevent Crawling of Certain File Types
To block PDFs, images, or specific file types:
User-agent: *
Disallow: /*.pdf$
Disallow: /*.jpg$
Disallow: /*.png$
Useful for: Protecting copyrighted content or limiting unnecessary crawling.
Set Crawl Delay (For Server Load Management)
User-agent: *
Crawl-delay: 10
Best for: Slowing down crawling on sites with limited server resources. (⚠ Note: Google doesn’t support Crawl-delay; use Google Search Console instead.)
Include Your Sitemap for Better Indexing
Sitemap: https://yourwebsite.com/sitemap.xml
Why? This helps search engines find your content faster.
Advanced Robots.txt Hacks
1. Different Rules for Different Crawlers
If you want to block Bing but allow Google:
User-agent: Googlebot
Disallow:
User-agent: Bingbot
Disallow: /
Why? Prioritizes Google crawling over less relevant search engines.
2. Blocking Bad Bots (e.g., Scrapers, Spammers)
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Disallow: /
Best for: Protecting content from competitors’ scrapers.
Prevent Crawling but Allow Indexing with Meta Robots
Sometimes you want pages to be indexed but not crawled. Instead of robots.txt, use:
<meta name=”robots” content=”noindex, follow”>
Robots.txt blocks crawling, but Google may still index blocked pages if they have backlinks.
Testing and Validating Robots.txt
1. Use Google Search Console
- Go to Google Search Console → Robots.txt Tester
- Check for errors and fix them.
2. Use a Robots.txt Testing Tool
- Google Robots.txt Tester
- TechnicalSEO Robots.txt Tester
3. Check Blocked Pages
site:yourwebsite.com inurl:blocked-url
If indexed, add a noindex tag or adjust robots.txt.
Common Robots.txt Mistakes to Avoid
Blocking All Search Engines
❌ User-agent: *
❌ Disallow: /
Blocking Important Pages (By Mistake)
❌ Disallow: /blog/ (Blocks entire blog!)
Blocking JS & CSS Files
❌ Disallow: /wp-content/themes/
Fix: Allow access so Google renders the site properly.
Forgetting to Add Sitemap
❌ No Sitemap: directive in robots.txt.
Final Thoughts
The robots.txt file is a crucial SEO tool when used correctly. Optimising it can improve crawl efficiency, enhance indexing, and protect sensitive content. However, mistakes can cripple your rankings, so test your file carefully!
✅ DO:
✔ Block low-value pages
✔ Allow important content
✔ Submit a sitemap
✔ Use Google Search Console
❌ DON’T:
✖ Block all crawlers
✖ Disallow critical resources
✖ Forget to test
Would you like a custom robots.txt audit for your website? Let me know, and I’ll analyse it for better SEO performance!
