How To Block WordPress From Search Engines

Learn how to block WordPress from search engines by configuring your WordPress settings and plugins in this tutorial

How To Block WordPress From Search EnginesThis tutorial is part of our WordPress configuration tutorials, where we show you how to set up and configure your WordPress site or blog with no coding skills required.

***

How To Block Your WordPress Site From Search Engines

WordPress combines an easy-to-use content publishing and management system with a powerful business marketing tool that when expertly configured, allows search engines to easily find your site and quickly index your posts and pages. Most of our tutorials assume that you actually want search engines to index your site.

There may be instances, however, when you may want to block search engines from indexing your site.

This tutorial covers some of the possible reasons why you would want to block your site from being indexed by search engines and shows you how to prevent search engines from indexing your site using various methods, including configuring WordPress settings and using different WordPress features and plugins.

Why Block Your WordPress Site From Search Engines?

Some of the reasons why you may want to block search engines from indexing your WordPress site can include:

  • Personal Use. You may want a personal blog to post news or updates about yourself to friends or family only (you could use Facebook for this).
  • Restrict Information To Trade Or Staff. You may want a private site to post confidential or restricted information about your business to clients and/or staff only, such as pricing, company news, updates, downloads or internal training.
  • Inform Members Of A Local Community. You may want a site for posting administrative information pertaining only to members of a local community such as a business network, local community organization, church, network marketing downline, or special interest group.
  • Hide Subscriber-Only Content – You may want to send subscribers to a site containing exclusive information that you don’t want listed in search engines and don’t want to set up a membership site.
  • Testing & Experimental Purposes. You may want to set up a WordPress site for internal testing or experimental purposes, such as installing and testing different themes and plugins, or creating draft content.
  • Demo Purposes. You may want to use a WordPress site for demo purposes, such as educational or training seminars and workshops, sales presentations of your products, etc.

Blocking Your WordPress Site From Search Engines – Step-By-Step Tutorial

There are various methods you can use to prevent search engines from indexing your site.

These include:

  • Configuring WordPress settings
  • Using built-in WordPress features
  • Using WordPress plugins

Blocking WordPress From Search Engines – Configuring WordPress Settings

The first method you can use to help prevent search engines from crawling and indexing your site is to configure your WordPress Reading Settings

You can discourage search engines from indexing your site

(You can discourage search engines from indexing your site)

After configuring your WordPress Reading Settings to discourage search engines from indexing your site, WordPress will also stop automatically notifying the update services listed in your WordPress Writing Settings

You can block WordPress from automatically notifying update services.

(You can block WordPress from automatically notifying update services)

To learn more about configuring your WordPress Writing Settings, see the tutorial below:

To learn more about configuring your WordPress Reading Settings, see the tutorial below:

Important

Configuring the above settings will allow visitors continued access to your site and will instruct search engines not to index your site, but it cannot force-block access to your site. It is up to search engines to honor your request. Malware and spam bots will typically ignore it.

Search Engines Discouraged

(Search Engines Discouraged)

robots.txt File

The next step in the process of blocking search engines from accessing your site is to configure a file in your server called robots.txt

robots.txt

(robots.txt)

A robots.txt file is used to instruct search engine spiders and co-operative bots which parts of your website they can or should not access.

To learn how to create, configure and add a robots.txt file to your server, see the tutorial below:

Blocking WordPress From Search Engines – Using Built-In WordPress Features

If you want to block or restrict certain parts of your website from being indexed by search engines, there are certain built-in features of WordPress you can use.

  • You can make pages and posts private,
  • You can password-protect pages and posts,
  • You can exclude pages and posts from XML sitemaps, HTML site maps, indexable custom menus, etc.

We have created step-by-step tutorials showing you how to block or restrict certain parts of your website from being indexed by search engines using all of the above built-in features of WordPress.

To learn more about protecting content in WordPress, see the tutorial below:

Blocking WordPress From Search Engines – Using WordPress Plugins

You can use WordPress plugins to prevent your site or restrict sections of your site or content from being indexed by search engines. Some of the strategies you can use with plugins include:

  • Creating a Membership Site using plugins that restrict access to content to specific users and can be configured to prevent search engines and bots from also accessing and indexing private areas and content on your site.
  • Putting your site in Maintenance Mode. This is useful if you want to block everyone from accessing your site (e.g. putting your site “under construction” while you work on improving it).
  • Installing a WordPress SEO plugin and configuring its settings to select which parts of your site should and shouldn’t be indexed by search engines.
Blocking Content From Search Engines With Yoast SEO

If you have the Yoast SEO WordPress plugin installed, you can configure its settings to prevent search engines from indexing Posts, Pages, and Media on your site.

To access these settings, go to your WordPress admin menu and choose SEO > Titles & Metas

Yoast SEO - Titles & Metas

(Yoast SEO – Titles & Metas)

When the Titles & Metas screen opens, click on the Post Types tab …

Yoast SEO: Titles & Metas - Post Types

(Yoast SEO: Titles & Metas – Post Types)

In the Post Types section, enable no index in the Meta Robots options to stop search engines from indexing your Posts, Pages, and Media …

Prevent Search Engines From Indexing Posts, Pages, And Media

(Prevent Search Engines From Indexing Posts, Pages, And Media)

You can also prevent search engines from indexing individual posts or pages on your site by clicking on the plugin’s ‘Advanced’ tab below the Post or Page editor and selecting the following:

  • Meta Robots Index – noindex
  • Meta Robots Follow – nofollow

Yoast SEO - Advanced Options

(Yoast SEO – Advanced Options)

You can also choose to noindex a certain category, tag, or custom taxonomy, by configuring the plugin’s options in that feature’s settings. For example, the screenshot below shows the WordPress SEO plugin’s settings when editing a category and how to prevent search engines from indexing that entire category …

Yoast SEO - Post Category Settings

(Yoast SEO – Post Category Settings)

So far, you have learned how to block search engine results by restricting indexing of your site, sections of your site, and individual posts or pages. Restricting indexing on your site, however, will not remove a page or post that was previously indexed by a search engine, at least until the next time the search engine bots attempt to spider your site (or perhaps even longer).

If a page of your site is already indexed and you would like it removed from the search engines, then do the following:

First, restrict indexing of the page or post using the methods described above. Once you have done this, you can request Google to remove an indexed site, page or post from its search database.

How To Remove Content That Has Already Been Indexed

Let’s say that you have published or created a post that Google has already found and indexed, and now you want it removed from Google’s search engine results.

Normally, when a page is updated or removed, it will eventually and automatically fall out of Google’s search results. You don’t need to do anything to make this happen. However, if you urgently need to remove content from Google’s search results, you can request expedited removal of those URLs.

For this example, let’s go through the process of asking Google to stop displaying content from a post that it has already indexed.

First, make sure you restrict indexing of the post that you would like removed from Google using one of the methods discussed earlier, then log into your Google Search Console

Google Search Console Login

(Google Search Console Login)

To learn how to set up a Google Search Console account, see the tutorial below:

Once you have logged into your account, click on your site’s name. This will bring up a list of options for managing your domain …

Search Console Dashboard

(Search Console Dashboard)

Go to Search Console > Dashboard > Google Index

Search Console > Dashboard > Google Index

(Search Console > Dashboard > Google Index)

Inside the ‘Google Index’ menu section, click on Remove URLs

Google Index > Remove URLs

(Google Index > Remove URLs)

This brings you to the ‘Remove URLs’ screen.

Google will temporarily remove your URLs from its search results. Enter the relative URL of the post, page or directory you want Google to remove from its indexing database and click Continue …

Google Search Console - Remove URLs

(Remove URLs)

Important

The relative URL is the part of the URL without the main domain name. So, in the example above:

  • Absolute URLhttp://yourdomain.com/private-pricing-info.php
  • Relative URL – private-pricing-info.php

Make sure you pay careful attention to the URL you are entering into this field because URLs are case sensitive.

  • http://yourdomain.com/PageTitle is not the same as
  • http://yourdomain.com/pagetitle

Read the ‘More information’ section to learn more about removing content permanently.

Once you have specified the URL to be removed, the next step is to indicate the request type. From the drop-down menu, choose one of the following options:

  • Temporarily hide from search results and remove from cache – Hides the page from Google search results for about 90 days, and also clears the cached copy of the page and snippet. The page can reappear in search results after the blackout period. Google will recrawl the page during the blackout period and refresh the page cache and snippet, but will not show them until the blackout period expires.
  • Remove page from cache only – Clears the cached page and snippet, but does not remove the page from search results. Google will refresh the page cache and snippet.
  • Temporarily hide directory – Hides an entire directory from search results for about 90 days and also clears cached pages and snippets for all pages in the specified directory. The directory can reappear in search results after the blackout period. Google will recrawl the pages during the blackout period and refresh the page caches and snippets.

Remove URLs - Request Type

(Remove URLs – Request Type)

Click Submit Request …

Remove URLs - Submit Request

(Remove URLs – Submit Request)

Your request will be submitted.

For more information about using Google Search Console to block search engine access to your content or remove URLs from its search results, refer to the Search Console help pages below:

Hopefully, now you have a better understanding of methods that you can use to block your site and prevent search engines from accessing areas of your site and your content.

How To Block Your WordPress Site From Search Engines

***

"I have used the tutorials to teach all of my clients and it has probably never been so easy for everyone to learn WordPress ... Now I don't need to buy all these very expensive video courses that often don't deliver what they promise." - Stefan Wendt, Internet Marketing Success Group

Disclaimer: This site is not directly associated with WordPress or any WordPress products mentioned or reviewed on this website. We may derive an affiliate commission from purchases of third-party services and products advertised on this website. All images remain the copyright of their respective owners and have been used only for training or illustrative purposes.

***

Have you found this tutorial useful? If so, please share this information using the share buttons below.

Author: Martin Aranovitch

Martin Aranovitch is the founder of WPCompendium.org and the author of The Small Business Digital Manager. WPCompendium.org provides hundreds of FREE detailed step-by-step tutorials that will teach you how to use WordPress to grow your business online at minimal cost with no coding skills required!

Originally published as How To Block WordPress From Search Engines.