WordPress Robots.txt Not Working? 11 Fixes That Work

A laptop on a clean desk showing a plain website settings screen beside a notebook — wordpress robots.txt not working fix guide
A robots.txt file is a small text file with big power — one wrong line can hide a whole site from search engines.

You edited the file, saved it, and checked the site — but search engines still behave as if nothing changed. Pages that should be crawled stay hidden, or pages that should be blocked keep appearing. When wordpress robots.txt not working leaves you guessing, the cause is rarely the file itself. It is usually a second copy of the file, a WordPress setting that overrides your edit, or a cached copy that visitors and crawlers keep receiving.

The good news is that every one of those causes is easy to find once you know where to look. A robots.txt file is a plain text file that tells search engine crawlers which parts of a site they may visit. WordPress can create this file automatically, a plugin can manage it, or a physical file on the server can take charge. This guide works through those layers in order, from the simplest check to the deeper fixes. Before you change anything, make a backup of the current file, because a wrong rule can hide the whole site from search results.

Table of Contents

  • Why This Error Happens
  • Quick Checks
  • Diagnosing the Problem
  • Fix 1: Check the Live Robots.txt File in Your Browser
  • Fix 2: Make a Backup Before You Change Anything
  • Fix 3: Check the Search Engine Visibility Setting in WordPress
  • Fix 4: Check for a Physical Robots.txt File on the Server
  • Fix 5: Check Your SEO Plugin Robots.txt Settings
  • Fix 6: Fix Common Syntax Mistakes in the Rules
  • Fix 7: Remove a Rule That Blocks the Whole Site
  • Fix 8: Clear Every Cache After Editing the File
  • Fix 9: Check for Plugin and Security Conflicts
  • Fix 10: Check the Sitemap Reference and Test the File
  • Fix 11: Restore a Clean Default File and Re-check
  • What to Avoid
  • Frequently Asked Questions
  • Conclusion
  • Helpful Official Resources

Why This Error Happens

A robots.txt file only works when the file that search engines actually fetch is the file you edited. In WordPress, several layers can stand between your edit and the live file. These are the reasons the live file does not match what you expect, and why the problem survives a simple edit.

1. WordPress Is Serving a Virtual File You Did Not Edit

When no physical robots.txt file exists on the server, WordPress generates a virtual robots.txt on the fly. The file does not exist as a real file you can open in a file manager. It is built from WordPress settings each time it is requested. If you edited a file in one place while WordPress serves its virtual version, your changes never reach search engines. This is the most common cause of wordpress robots.txt not working, because everything looks correct in the editor while the live file stays unchanged.

2. The Search Engine Visibility Setting Is Overriding Your Rules

WordPress has a setting called Discourage search engines from indexing this site under Settings, then Reading. When this box is ticked, WordPress adds a rule that blocks all crawlers from the whole site. The setting is meant for sites under construction, but it is easy to leave ticked by accident after launch. It overrides the spirit of any careful rules you write, because the block-all rule sits in the file that crawlers actually read.

3. A Physical File on the Server Takes Priority Over the Virtual File

If a real robots.txt file sits in the root folder of the site, the server sends that file, and WordPress never gets a chance to generate its virtual version. Edits made inside an SEO plugin editor then change the virtual file only, while the physical file keeps serving old rules. The reverse also happens: an old physical file left behind by a previous setup keeps blocking pages long after the site owner forgot it existed.

4. A Plugin Is Writing Its Own Rules Into the File

SEO plugins such as Rank Math and Yoast SEO include a robots.txt editor, and that is a good thing. The trouble starts when two plugins both manage the file, or when a plugin setting writes rules automatically. A security plugin can also add blocking rules of its own. The result is a file that changes back after you edit it, or rules that appear in the live file that you never wrote. The file is working exactly as the plugins told it to — the problem is that two managers are giving it different orders.

5. A Cache Is Serving an Old Copy of the File

Caching plugins, server caches, and content delivery networks all store copies of files to serve them faster. A robots.txt file can sit in a cache for hours or days after you edit it. You see the new file when you open the editor, but search engines and browsers keep receiving the cached copy. The edit is correct and saved, yet nothing changes, which makes this cause feel like the file itself is broken when it is only stale.

6. A Syntax Mistake Makes Crawlers Ignore the Rule

Robots.txt rules follow a strict, simple format, and small mistakes change their meaning completely. Paths are case-sensitive, so blocking /Images/ does not block /images/. A missing slash, a misplaced wildcard, or a rule placed under the wrong user-agent line can make a crawler skip the rule entirely. The file loads fine and looks right at a glance, but the rule you care about never applies. Syntax mistakes are the quietest cause, because nothing shows an error.

Quick Checks

Two minutes before the deep fixes.

  • Open your site address with /robots.txt added at the end — does the live file match what you expect?
  • Check Settings, then Reading, in the dashboard — is the search engine visibility box ticked?
  • Think about when the problem started — did wordpress robots.txt not working begin after a plugin install, a migration, or a settings change?
  • Check whether an SEO plugin is installed — does it have its own robots.txt editor?
  • Clear the browser cache or open the file in a private browser window — does the live file look different?
  • Note the exact rule that is misbehaving — which page should be blocked or allowed, and what happens instead?

Diagnosing the Problem

SymptomCauseGo to Fix
Live file differs from the edited fileVirtual file or physical file mismatchFix 1
Whole site blocked from search enginesVisibility setting is tickedFix 3
Edits keep reverting after savingPhysical file overrides the editorFix 4
Rules appear that you never wroteSEO plugin writing its own rulesFix 5
One rule is ignored, others workSyntax mistake in that ruleFix 6
All pages blocked, including the homepageBlock-all rule in the fileFix 7
Edit saved but live file stays oldCache serving a stale copyFix 8
File changes back after a plugin updatePlugin or security conflictFix 9

Fix 1: Check the Live Robots.txt File in Your Browser

Start with the live file, because it is the only version that matters to search engines. The editor shows what you saved. The browser shows what crawlers receive. Comparing the two tells you immediately whether the problem is the file itself or one of the layers in front of it.

A person checking a website address in a browser on a laptop at a bright desk
The live file is the truth — open it in the browser and compare it with what the editor shows.
  1. Open your browser and type your site address followed by /robots.txt, then press Enter.
  2. Read the file that appears and compare it with the rules you expect to see.
  3. Copy the whole live file into a plain text note so you have a record of the starting point.
  4. Open a private or incognito browser window and load the same address again.
  5. Compare the two views — if they differ, a cache is serving one of them.
  6. Check the address bar carefully — make sure you are viewing the live site, not a staging copy.
  7. Note every rule that surprises you, including any rule you did not write, before moving on.

The live file is the single source of truth for this whole guide. Every later fix ends with a return to this address to confirm the change took effect. If the live file already matches your rules, the problem may not be the file at all — search engines can take time to revisit a site, and Fix 10 covers how to test and prompt a fresh check. If the live file does not match, the remaining fixes will find the layer that is standing in the way. Do not skip this step on a guess. A two-minute look at the real file saves an hour of editing a copy that nobody reads.

Fix 2: Make a Backup Before You Change Anything

A robots.txt file is short, but a mistake in it can hide the whole site from search engines. A backup takes one minute and turns any bad edit into a harmless experiment. Never edit the file without a way to restore the version that was there before.

  1. Open the live file in the browser as in Fix 1 and select all of the text.
  2. Copy the text into a plain text file on your computer and save it with the date in the name.
  3. If your SEO plugin has a robots.txt editor, open it and copy the text shown there too.
  4. If your hosting control panel has a file manager, check whether a physical file exists and download a copy.
  5. Take a full site backup through your hosting panel or backup plugin if one is available.
  6. Write down the current search engine visibility setting so you can restore it if needed.
  7. Keep the backup somewhere safe until the problem is fixed and the live file is confirmed correct.

A backup also protects you from the most common accident in this whole area: editing the file, forgetting what it said before, and being unable to undo the change. The backup habit matters more here than in most WordPress tasks, because the damage from a wrong rule is silent — no error appears, the site keeps working for visitors, and only search traffic quietly disappears. With a backup in hand, every fix in this guide is safe to try. If a change makes things worse, restore the saved copy and start again from a known state.

Fix 3: Check the Search Engine Visibility Setting in WordPress

The visibility setting is the fastest cause to check and the most embarrassing to miss. One ticked box tells WordPress to block every crawler from the whole site, and it does so through the virtual robots.txt file. Many site owners tick it during development and forget to untick it at launch.

  1. Log in to the WordPress dashboard and open Settings, then Reading.
  2. Find the section called Search engine visibility near the bottom of the page.
  3. If the box labelled Discourage search engines from indexing this site is ticked, untick it.
  4. Scroll down and click Save Changes.
  5. Open the live robots.txt file in the browser again, in a private window.
  6. Check whether the block-all rule has disappeared from the live file.
  7. If the rule is still there, clear the cache as in Fix 8 and check the live file once more.

Unticking the box does not instantly restore search traffic. Search engines revisit the file on their own schedule, and it can take days or weeks for crawling and indexing to recover fully. That delay is normal, and it does not mean the fix failed. What matters now is that the live file no longer tells crawlers to stay away. If the site is still under construction and you want it hidden on purpose, leave the box ticked and accept that the robots.txt file will block crawlers — that is the setting doing its job, not a fault. For a live site that should appear in search results, the box must stay unticked.

Fix 4: Check for a Physical Robots.txt File on the Server

When a physical file exists in the root folder of the site, it wins. The server sends it directly, WordPress never generates its virtual version, and edits made in a plugin editor change a file that nobody reads. Finding and removing or editing the physical file resolves the mismatch at a stroke.

A clean desk with a laptop showing a plain file manager window and a notebook beside it
A physical file in the root folder wins over the virtual file — check the server before editing anything else.
  1. Open your hosting control panel and find the File Manager, or connect with an FTP app.
  2. Open the root folder of the site — the folder that contains wp-config.php and the wp-content folder.
  3. Look for a file named exactly robots.txt in that root folder.
  4. If it exists, download a copy to your computer as a backup before touching it.
  5. Open the file and compare its contents with the live file from Fix 1.
  6. Decide which copy should be the master: edit the physical file directly, or delete it so WordPress and your SEO plugin manage the virtual file.
  7. After your decision, reload the live file in a private browser window and confirm it matches.

Do not keep both a physical file and a plugin-managed file and try to edit both. Pick one master copy and stick with it. For most sites, letting the SEO plugin manage the virtual file is the simplest choice, because the editor lives in the dashboard and changes are easy to track. A physical file makes sense when a developer manages the server directly. Either choice works, but two masters guarantee confusion. If you delete the physical file, WordPress starts serving its virtual version immediately, so check the live file right away to see what the virtual version contains.

Fix 5: Check Your SEO Plugin Robots.txt Settings

SEO plugins give robots.txt a friendly editor, but they also add their own rules, and two plugins managing the same file will fight. Check which plugin owns the file, what rules it is writing, and whether a second plugin is also involved.

  1. Open the dashboard and check which SEO plugin is active — Rank Math, Yoast SEO, or another.
  2. Open the plugin settings and find its robots.txt editor or file editor section.
  3. Read every rule in the editor and compare it with the live file from Fix 1.
  4. Remove any rule you do not recognise, and note which plugin added it if the editor says so.
  5. Check whether a second SEO plugin or a second robots.txt tool is also active.
  6. If two tools manage the file, deactivate one of them so only one editor writes the rules.
  7. Save the file in the remaining editor, then reload the live file and confirm it matches.

Plugin editors sometimes add helpful default rules, such as a sitemap reference at the bottom of the file. Those defaults are usually fine and should be left alone. The danger is duplication: two editors writing the same file, or an old plugin that was deactivated but left its rules behind in a physical file. If the live file still does not match the editor after this fix, return to Fix 4 and confirm no physical file is overriding the virtual one. URL and address problems often travel with robots.txt confusion, and our guide to WordPress permalink structure causing 404 covers the address side when pages return errors instead of loading.

Fix 6: Fix Common Syntax Mistakes in the Rules

Robots.txt syntax is simple, but it is strict. A rule that looks right can fail silently because of a capital letter, a missing slash, or a line in the wrong group. Read each rule slowly, one line at a time, and check it against the format that crawlers expect.

  1. Open the live file and read it line by line, starting from the top.
  2. Check that every group starts with a User-agent line, followed by its Disallow or Allow lines.
  3. Check that every path starts with a forward slash, such as /wp-admin/.
  4. Check capitalisation — paths are case-sensitive, so /Images/ and /images/ are different folders.
  5. Check wildcard use — the star means any characters, and the dollar sign means the end of the address.
  6. Check that no rule has a space at the start of the line or a missing colon after the field name.
  7. Fix one rule at a time, save, and reload the live file to confirm each change.

The most common syntax mistake is a Disallow line with no path, or with a path that does not match the real folder name. A blank Disallow means nothing is blocked, which is the opposite of what most people intend when they write it. Another common mistake is placing a rule before any User-agent line, which leaves the rule belonging to no group, so crawlers ignore it. Keep rules short and specific. One clear rule per line, grouped under the correct user-agent, beats a clever block of wildcards that nobody can read. If a rule still misbehaves after a syntax check, test it with the method in Fix 10 before rewriting the whole file.

Fix 7: Remove a Rule That Blocks the Whole Site

The most damaging rule in any robots.txt file is the shortest one: a Disallow that blocks the root. It tells every crawler to leave the entire site alone. It hides the homepage, the posts, and the pages in one line, and it often arrives through the visibility setting or a copied example file.

  1. Open the live file and look for a line that reads Disallow: / under a user-agent group.
  2. Check which user-agent the rule belongs to — a star means it applies to all crawlers.
  3. If the rule is present and the site should be visible, remove that line from the editor.
  4. Replace it with specific rules that block only the folders you mean to block, such as /wp-admin/.
  5. Add an Allow line for the homepage path if your plugin examples use one, and keep it simple.
  6. Save the file and reload the live version in a private browser window.
  7. Confirm the block-all line is gone from the live file before doing anything else.

A block-all rule is correct in exactly one situation: a site that must stay out of search results, such as a staging copy or a site under construction. For every live site, it is a disaster hiding in plain sight. Visitors see a normal site, the dashboard works normally, and nothing shows an error — only search traffic fades. After removing the rule, expect recovery to take time. Crawlers must refetch the file, revisit the pages, and rebuild their picture of the site. If the rule keeps coming back after you remove it, the visibility setting in Fix 3 or a plugin in Fix 5 is rewriting it, so revisit those fixes before editing the file again.

Fix 8: Clear Every Cache After Editing the File

An edit that is saved but not visible is almost always a cache problem. WordPress sites often have several caches stacked on top of each other, and clearing only one leaves the stale file in place. Work through them all, from the closest cache to the furthest.

A laptop on a desk with a plain settings screen and a cup of tea, in bright daylight
Several caches can stack up — clear each one in turn, then check the live file again.
  1. Open the caching plugin settings in the dashboard and use its clear or purge option for the whole cache.
  2. If the hosting panel has a server cache or object cache option, clear that too.
  3. If a content delivery network serves the site, purge its cache for the robots.txt address.
  4. Clear the browser cache, or simply check the live file in a private browser window.
  5. Wait a few minutes, then reload the live file and compare it with the editor.
  6. Repeat the check after an hour if the file still looks old — some caches clear slowly.
  7. Once the live file matches the editor, stop clearing and move on to testing in Fix 10.

Caching is the cause that makes people doubt their own edits. The file is correct, the save worked, and yet the browser shows the old version. The lesson is to check the live file in a private window after every edit, and to clear caches before concluding that an edit failed. Note that a cache can also hide a good fix: after removing a block-all rule, the cached copy may keep blocking crawlers until the cache clears. For cache trouble that goes beyond this one file, our guide to WordPress cache not clearing covers the deeper cache fixes.

Fix 9: Check for Plugin and Security Conflicts

Some plugins rewrite the robots.txt file on their own schedule. Security plugins add blocking rules, migration plugins leave old files behind, and SEO tools compete for control. When the file changes back after an edit, a plugin is usually the author of the change.

  1. Make a note of every plugin that could touch the file: SEO tools, security plugins, and migration tools.
  2. Deactivate the most likely suspect first, then check the live file again.
  3. If the live file changes, you have found the plugin that was writing the rules.
  4. Open that plugin’s settings and look for a robots.txt or crawler-blocking option.
  5. Turn off the option, or adjust its rules, then reactivate the plugin and check the file again.
  6. If no single plugin is responsible, deactivate plugins one group at a time and test after each group.
  7. Keep a note of the plugin that caused the trouble so the same conflict is not reintroduced later.

Never deactivate all plugins at once on a live site without a plan, and never leave a security plugin off for longer than the test needs. Work one suspect at a time, test after each change, and reactivate as you go. Security plugins deserve care: their blocking rules often protect real parts of the site, such as the login page, and removing them carelessly can open the site to abuse. The goal is not to remove protection. It is to find which tool is writing rules you did not approve, and to bring its settings in line with the file you actually want.

Fix 10: Check the Sitemap Reference and Test the File

A robots.txt file often ends with a line pointing crawlers to the sitemap. A wrong sitemap address sends crawlers to a page that does not exist, and a broken file fails silently. Testing the file with an official tool turns guesswork into a clear pass or fail for each rule.

  1. Open the live file and look for a Sitemap line, usually at the bottom.
  2. Copy the sitemap address from that line and open it in the browser.
  3. If the sitemap address shows an error, fix the address or regenerate the sitemap in your SEO plugin.
  4. Open Google Search Console and find the robots.txt report for the site.
  5. Check whether Search Console reports the file as fetched successfully and shows the current rules.
  6. Test a specific page address against the file to see whether it is blocked or allowed.
  7. Fix any rule the test flags, save, clear the cache, and test again until the report is clean.

Testing matters because robots.txt gives no error messages of its own. A rule either works or it does not, and only a test shows which. The Sitemap line is easy to overlook, but a wrong address wastes the crawl effort the file is meant to guide. Keep the sitemap reference pointing at the live sitemap file, and keep the sitemap itself fresh. For sitemap problems that survive this fix, our guides to WordPress sitemap not found and WordPress sitemap not updating cover the sitemap side in full.

Fix 11: Restore a Clean Default File and Re-check

When the file has been edited by several hands and several plugins, the cleanest fix is to start over. A short default file with only the essential rules removes every hidden mistake at once. Add custom rules back one at a time, testing after each one.

A notebook with a short checklist beside a laptop on a tidy desk in daylight
A short, clean file beats a long, tangled one — rebuild the rules one at a time and test each addition.
  1. Save a backup of the current file as in Fix 2, even if the current file is broken.
  2. Write a short default file: a User-agent star group, a Disallow for /wp-admin/, and an Allow for the admin-ajax file if your setup needs it.
  3. Add the Sitemap line pointing to the live sitemap address.
  4. Save the file through your chosen master editor — one editor only, as decided in Fix 4.
  5. Clear every cache as in Fix 8, then reload the live file in a private window.
  6. Test the file with the method in Fix 10 and confirm the default rules behave as expected.
  7. Add custom rules back one at a time, testing after each addition, until the file does everything you need.

Starting over feels drastic, but it is the fastest route when the file carries years of edits from different tools. A short file is easy to read, easy to test, and easy to trust. If wordpress robots.txt not working survives a clean default file, a cleared cache, and a passing test, the fault lies outside the file — in server settings, a redirect, or a hosting rule that serves a different file. In that case, contact the hosting provider, share the live file address and the test results, and ask them to check for a server-level rule. Those two pieces of evidence tell the host exactly where to look.

What to Avoid

  • Do not edit the file without a backup — a wrong rule can hide the whole site, and you need a way back.
  • Do not keep both a physical file and a plugin-managed file — pick one master copy and edit only that one.
  • Do not copy a long example file from the internet without reading every line — example rules often block folders your site needs.
  • Do not use a block-all rule on a live site — Disallow: / tells every crawler to stay away from everything.
  • Do not expect the file to remove pages from search results — blocking crawling is not the same as removing a page from the index.
  • Do not install a second SEO plugin to fix the file — two managers writing one file is a common cause of this problem.
  • Do not trust the editor view alone — always confirm the change in the live file in a private browser window.

Frequently Asked Questions

Why does the live file differ from the file in my SEO plugin editor?

Because a physical file on the server takes priority over the virtual file the plugin manages, or a cache is serving an old copy. Check the server root folder for a physical file as in Fix 4, pick one master copy, and clear every cache as in Fix 8. The live file in a private browser window is the version that counts.

How long does it take for search engines to notice a fixed file?

Crawlers refetch the file on their own schedule, which can range from hours to days. Recovery of crawling and indexing after a block-all rule can take longer, sometimes weeks for a large site. A clean test result in Fix 10 is the proof the file is correct — the waiting that follows is normal, not a sign of failure.

Should I block the wp-admin folder in the file?

Yes, blocking the admin folder is standard practice and part of the usual default file. It keeps crawlers out of the login and dashboard areas, which should never appear in search results. The common exception is the admin-ajax file, which some parts of the site need crawlers to reach. Keep the default rule and add the exception only if your setup needs it.

Can a robots.txt file remove a page from search results?

No. The file tells crawlers not to visit a page, but a page that is already indexed can stay in the results even when crawling is blocked. To remove a page from results, use a noindex setting for that page through your SEO plugin, and let crawlers visit the page so they can read the instruction. Blocking and removing are different jobs, and the file only does the first one.

Is it safe to delete the physical file from the server?

Yes, if WordPress or an SEO plugin will manage the virtual file instead. Download a backup copy first, then delete the physical file and check the live address immediately to see what the virtual file contains. Never delete the file without a backup, and never leave the site with no file at all unless you have confirmed the virtual version is being served.

Why does the file keep changing back after I edit it?

Because a plugin or the visibility setting is rewriting it. A security plugin, a second SEO tool, or the Discourage search engines box can all restore rules after an edit. Work through Fix 3, Fix 5, and Fix 9 to find the tool that is writing the file, adjust its settings, and then edit the file once more. Editing without fixing the writer only repeats the cycle.

Conclusion

When wordpress robots.txt not working leaves the live file out of step with your edits, the answer is almost always a second copy, a setting, or a cache — not a broken file. Check the live file first, back it up, untick the visibility box, settle on one master copy, fix the syntax, clear every cache, and test the result. Finish with a fresh look at the live file in a private window — a file that matches your rules, line for line, is the only proof that counts. Set it, test it, and let crawlers back in.

Helpful Official Resources

Leave a Reply

Your email address will not be published. Required fields are marked *