How to Block Search Engines from Indexing WordPress

It seems every second client we have has a WordPress site or blog. We do a ton of custom development and design on WordPress – everything from building plugins for companies to developing a video workflow application using Amazon cloud services. WordPress isn’t always the right solution, but it’s quite flexible and we’re pretty good at it.

Many times, we stage sites so that our customers can preview and critique the work before we put it live. Sometimes we even import the client’s current content so that we can work on a real site with live content. We don’t want Google to get confused as to which site is the real site, so we discourage the search engines from indexing the site using a standard technique.

How to Block Search Engines In WordPress

Keep in mind that block may be too strong a term. There are ways to block the search engine crawler from actually accessing your site… but what we’re doing here is really just asking them not to index the site in their search results.

To do this within WordPress is quite simple. In the Settings > Reading menu, you can simply check a box:

How to Block Search Engines Using Robots.txt

Additionally, if you have access to the root web directory your site is in, you can also modify your robots.txt file to:

User-agent: * 
Disallow: /

The robots.txt modification will actually work for any website. Again, if you’re using WordPress, the Rank Math SEO Plugin enables the ability to update your Robots.txt file directly through their interface… which is a bit easier than trying to FTP into your site and editing the file yourself.

If you’re developing an unfinished application, staging software at a different domain or subdomain, or developing a duplicate site for some reason – it’s good to block search engines from indexing your site and taking search engine users to the wrong location!

Disclosure: Martech Zone is a customer and an affiliate of Rank Math.