Your WordPress robots.txt Is Blocking Google — Fix It Now in 2026

Open your browser right now and go to yourdomain.com/robots.txt.

What you see there — or do not see — is directly controlling what Google can and cannot crawl on your WordPress site. A single wrong line in that file can block your entire site from Google. A missing line means Googlebot is crawling pages that should never be indexed. And most WordPress site owners have never looked at theirs.

This guide covers everything about WordPress robots.txt: what it does, how WordPress generates it, what a correct robots.txt looks like, the most dangerous mistakes to avoid, and how to optimise yours to protect important pages while directing Googlebot exactly where you want it to go.

Your WordPress robots.txt file viewed in browser at yourdomain.com/robots.txt — this file controls what Googlebot can and cannot crawl. A misconfigured WordPress robots.txt is one of the most damaging and most overlooked SEO issues on WordPress sites.

What Is WordPress robots.txt — And Why Does It Matter?

The robots.txt file is a plain text file that sits at the root of your domain. Every time Googlebot visits your site, it reads robots.txt first — before crawling a single page. The file tells crawlers which URLs they are allowed to visit and which they should skip.

On a WordPress site, robots.txt serves several critical purposes. It prevents Googlebot from wasting crawl budget on pages that should never rank — admin pages, login pages, internal search results, tag archives. It points Googlebot directly to your sitemap. And it can block specific bots — like the Mediapartners-Google AdSense bot — from areas of your site that should not be evaluated for ad serving.

The reason WordPress robots.txt matters so much is that WordPress does not ship with a perfectly optimised robots.txt by default. The default WordPress robots.txt is functional but minimal — it does not block the many low-value URL patterns that WordPress generates automatically, and it does not include a sitemap reference. Every hour you spend optimising other parts of your SEO without checking robots.txt might be partially undermined by Googlebot crawling pages it should be skipping.

How WordPress Generates robots.txt

WordPress handles robots.txt in a way that surprises many site owners: by default, WordPress generates a virtual robots.txt dynamically — there is no actual robots.txt file in your root directory. WordPress intercepts requests to /robots.txt and generates the output on the fly from its internal settings.

This means if you look in cPanel File Manager → public_html and do not see a robots.txt file — that is normal for a default WordPress install. The file is being served virtually.

However, once you use an SEO plugin like RankMath or Yoast, the robots.txt management shifts to the plugin. Both RankMath and Yoast give you a built-in editor to modify your WordPress robots.txt without touching any server files directly. This is the recommended approach — it keeps robots.txt edits inside WordPress and prevents the file from being accidentally overwritten during updates.

If a physical robots.txt file exists in your public_html folder, it takes precedence over the virtual WordPress version. This can cause conflicts if both exist — the physical file may not match what your SEO plugin thinks the robots.txt contains.

The Default WordPress robots.txt — What It Contains

WordPress’s default virtual robots.txt looks like this:

This blocks all crawlers from the WordPress admin area, with one exception: admin-ajax.php, which needs to be accessible because it handles AJAX requests that plugins use for frontend functionality.

That is it. That is the default. It does not block tag archives. It does not block search result pages. It does not block the login page directly. It does not include a sitemap reference. For a simple personal blog, this might be acceptable. For any site actively trying to rank and control its crawl budget, it is not enough.

The Correct WordPress robots.txt — Complete Template

This is the WordPress robots.txt template that covers the most important optimisations for a standard WordPress blog or content site:

Replace yourdomain.com with your actual domain. If you use Yoast instead of RankMath, your sitemap URL may be slightly different — check by visiting yourdomain.com/sitemap_index.xml to confirm it loads.

How to Edit WordPress robots.txt — 3 Methods

Method 1 — RankMath (Recommended)

1. Go to WordPress Dashboard → RankMath → General Settings
2. Click the “Edit robots.txt” button (found in the General tab)
3. The current robots.txt content appears in an editable text box
4. Replace the content with the template above (adjusted for your domain)
5. Click Save Changes
6. Verify by visiting yourdomain.com/robots.txt in your browser

RankMath → General Settings → Edit robots.txt — the built-in robots.txt editor lets you manage your WordPress robots.txt directly inside the dashboard without touching any server files. This is the recommended approach for any WordPress site using RankMath.

Method 2 — Yoast SEO

1. Go to SEO → Tools → File Editor
2. The robots.txt file content is shown in an editable section
3. Make your changes → click Save changes to robots.txt
4. Verify at yourdomain.com/robots.txt

Method 3 — cPanel File Manager (Direct Edit)

1. Login to Hostinger → hPanel → File Manager
2. Navigate to public_html
3. If a robots.txt file exists — right-click → Edit
4. If no file exists — right-click → New File → name it robots.txt → Edit
5. Paste the template above → Save Changes
6. Verify at yourdomain.com/robots.txt

cPanel / Hostinger File Manager → public_html — if a physical robots.txt file exists here, it takes precedence over WordPress’s virtual version. Check for conflicts between this physical file and what your SEO plugin is generating.

How to Test Your WordPress robots.txt

After editing your WordPress robots.txt, always test it before assuming it is working correctly. A single syntax error can block Googlebot from your entire site.

Test 1 — Browser Check

Visit yourdomain.com/robots.txt in your browser. The file content should load as plain text. Verify that every rule you added is visible and correctly formatted — no extra spaces before Disallow:, correct URL patterns, sitemap line present at the bottom.

Test 2 — Google Search Console robots.txt Tester

1. Open Google Search Console → click Settings (gear icon, bottom of left sidebar)
2. Under “Crawling” — find “robots.txt” and click “Open Report”
3. The tester shows your current robots.txt content and lets you test specific URLs
4. Enter a URL you want to test — for example /?s=test — and click Test
5. Result should show “Blocked” for that URL
6. Test a page you DO want indexed — like a blog post URL — result should show “Allowed”

Google Search Console → Settings → robots.txt tester — enter any URL to see whether your current WordPress robots.txt allows or blocks Googlebot from crawling it. Always test after any robots.txt edit to confirm rules are working as intended.

WordPress robots.txt for WooCommerce

WooCommerce sites need additional robots.txt rules because WooCommerce generates many URL patterns that should not be indexed — cart pages, checkout pages, account pages, and product filter combinations that create thousands of near-duplicate URLs.

The Disallow: /shop/?orderby= and Disallow: /shop/?filter_ lines are particularly important — they prevent Googlebot from crawling thousands of product filter and sorting URL combinations that create crawl budget waste without ranking value.

The Most Dangerous WordPress robots.txt Mistakes

WordPress robots.txt and the “Discourage Search Engines” Setting

WordPress has a built-in setting that adds Disallow: / to your robots.txt — blocking your entire site from search engines. It exists for development environments where you do not want Google indexing a site that is not ready.

The problem is that many WordPress sites are launched with this setting accidentally left on. The site goes live. Traffic never comes. The owner spends months wondering why nothing ranks — and the answer has been sitting in Settings → Reading the entire time.

1. Go to WordPress Dashboard → Settings → Reading
2. Find “Search engine visibility”
3. The checkbox should say “Discourage search engines from indexing this site”
4. This checkbox must be UNCHECKED on any live site
5. If it was checked — uncheck it → Save Changes → immediately verify your robots.txt no longer contains Disallow: /

robots.txt vs noindex — What Is the Difference?

This is one of the most important distinctions in technical SEO, and it is commonly misunderstood. robots.txt and noindex both control what Google sees — but they work in completely different ways.

The critical rule: never use robots.txt to block a page you want to noindex. If Googlebot cannot crawl the page, it cannot see the noindex meta tag, and may still show the URL in search results based on links from other pages. For pages you want deindexed — use noindex and allow crawling.

How to Monitor Your robots.txt Going Forward

1. Check after every plugin update: Security plugins, SEO plugins, and caching plugins can all modify your robots.txt. After any major plugin update, visit yourdomain.com/robots.txt and verify the content is still correct.
2. Check after any site migration: Domain changes, HTTP to HTTPS moves, and server migrations can reset or overwrite your WordPress robots.txt. Post-migration robots.txt verification is non-negotiable.
3. Monitor GSC “Blocked by robots.txt” count: GSC → Pages → “Blocked by robots.txt” shows URLs Googlebot tried to crawl but was blocked by your file. Check this monthly — unexpected pages appearing here signal a misconfigured rule.
4. Use GSC robots.txt Tester quarterly: Run a quick test on your 5 most important page types to confirm they are still being allowed correctly after any changes to your setup.

Google Search Console → Pages → “Blocked by robots.txt” — after optimising your WordPress robots.txt, this count should only include genuinely low-value pages you intentionally blocked. Any important content pages appearing here signal a misconfigured Disallow rule that needs immediate fixing.

Complete WordPress robots.txt Checklist

• Visit yourdomain.com/robots.txt right now — confirm it loads and does not contain Disallow: /
• Settings → Reading — “Discourage search engines” is UNCHECKED on your live site
• wp-admin/ is blocked — default WordPress rule, must remain
• admin-ajax.php is allowed — required for plugin AJAX functionality
• /?s= is blocked — prevents unlimited internal search result URLs from being crawled
• Sitemap: line is present at the bottom pointing to your correct sitemap URL
• No /wp-content/ block — blocking this breaks CSS/JS rendering for Googlebot
• No /*.xml wildcard block — this blocks your sitemap
• No /uploads/ block — this prevents image indexing
• Only ONE robots.txt in effect — either physical file OR plugin-generated, not both
• Test in GSC robots.txt Tester — verify allowed and blocked URLs return correct results
• WooCommerce sites: /cart/ /checkout/ /my-account/ blocked
• Monitor monthly — check after every plugin update and migration

The Bottom Line

Your WordPress robots.txt file is one of the first things Googlebot reads when it visits your site — and for most WordPress sites, it has never been deliberately optimised. The default rules are minimal. The common mistakes range from accidentally blocking your entire site to leaving internal search results open for unlimited crawling.

A well-configured WordPress robots.txt takes 15 minutes to set up and requires a quick check after every major plugin update or migration. The payoff is direct: Googlebot stops wasting crawl time on admin pages, login forms, and internal search results, and spends more of each visit crawling the content you actually want ranked.

Start by visiting yourdomain.com/robots.txt right now. Verify what is there. Update it with the template in this guide. Test it in Google Search Console’s robots.txt tester. Then add a monthly calendar reminder to check it after any significant site changes. That combination — a correct robots.txt plus consistent monitoring — is all it takes to eliminate this as an SEO liability permanently.

Related Articles

Frequently Asked Questions