This tutorial is part of our WordPress configuration step-by-step training series, where we show you how to set up and configure your WordPress site or blog with no coding skills required.
***
How To Block Your WordPress Site From Search Engines
WordPress combines an easy-to-use content publishing and management system with a powerful business marketing tool that when expertly configured, allows search engines to easily find your site and quickly index your posts and pages. Most of our tutorials assume that you actually want search engines to index your site.
There may be instances, however, when you may want to block search engines from indexing your site.
This tutorial covers some of the possible reasons why you would want to block your site from being indexed by search engines and shows you how to prevent search engines from indexing your site using various methods, including configuring WordPress settings and using different WordPress features and plugins.
Why Block Your WordPress Site From Search Engines?
Some of the reasons why you may want to block search engines from indexing your WordPress site can include:
- Personal Use. You may want a personal blog to post news or updates about yourself to friends or family only (you could use Facebook for this).
- Restrict Information To Trade Or Staff. You may want a private site to post confidential or restricted information about your business to clients and/or staff only, such as pricing, company news, updates, downloads or internal training.
- Inform Members Of A Local Community. You may want a site for posting administrative information pertaining only to members of a local community such as a business network, local community organization, church, network marketing downline, or special interest group.
- Hide Subscriber-Only Content – You may want to send subscribers to a site containing exclusive information that you don’t want listed in search engines and don’t want to set up a membership site.
- Testing & Experimental Purposes. You may want to set up a WordPress site for internal testing or experimental purposes, such as installing and testing different themes and plugins, or creating draft content.
- Demo Purposes. You may want to use a WordPress site for demo purposes, such as educational or training seminars and workshops, sales presentations of your products, etc.
Blocking Your WordPress Site From Search Engines – Step-By-Step Tutorial
There are various methods you can use to prevent search engines from indexing your site.
These include:
- Configuring WordPress settings
- Using built-in WordPress features
- Using WordPress plugins
How To Block WordPress From Search Engines – Configuring WordPress Settings
The first method you can use to help prevent search engines from crawling and indexing your site is to configure your WordPress Reading Settings …
(You can discourage search engines from indexing your website)
After configuring your WordPress Reading Settings to discourage search engines from indexing your site, WordPress will also stop automatically notifying the update services listed in your WordPress Writing Settings …
(You can block WordPress from automatically notifying update services)
To learn more about configuring your WordPress Writing Settings, see the tutorial below:
To learn more about configuring your WordPress Reading Settings, see the tutorial below:
Configuring the above settings will allow visitors continued access to your site and will instruct search engines not to index your site, but it cannot force-block access to your site. It is up to search engines to honor your request. Malware and spam bots will typically ignore it.
(Search Engines Discouraged)
robots.txt File
The next step in the process of blocking search engines from accessing your site is to configure a file in your server called robots.txt …
(robots.txt)
A robots.txt file is used to instruct search engine spiders and co-operative bots which parts of your website they can or should not access.
To learn how to create, configure and add a robots.txt file to your server, see the tutorial below:
Blocking WordPress From Search Engines – Using Built-In WordPress Features
If you want to block or restrict certain parts of your website from being indexed by search engines, there are certain built-in features of WordPress you can use.
- You can make pages and posts private,
- You can password-protect pages and posts,
- You can exclude pages and posts from XML sitemaps, HTML site maps, indexable custom menus, etc.
We have created step-by-step tutorials showing you how to block or restrict certain parts of your website from being indexed by search engines using all of the above built-in features of WordPress.
To learn more about protecting content in WordPress, see the tutorial below:
How To Block WordPress From Search Engines – Using WordPress Plugins
You can use WordPress plugins to block your site or restrict sections of your site or content from being indexed by search engines. Some of the strategies you can use with plugins include:
- Creating a Membership Site using plugins that restrict access to content to specific users and can be configured to prevent search engines and bots from also accessing and indexing private areas and content on your site.
- Putting your site in Maintenance Mode. This is useful if you want to block everyone from accessing your site (e.g. putting your site “under construction” while you work on improving it).
- Installing a WordPress SEO plugin and configuring its settings to select which parts of your site should and shouldn’t be indexed by search engines.
Blocking Content From Search Engines With Yoast SEO
If you have the Yoast SEO WordPress plugin installed, you can configure its settings to prevent search engines from indexing Posts, Pages, and Media on your site.
To access these settings, go to your WordPress admin menu and choose SEO > Titles & Metas …
(Yoast SEO – Titles & Metas)
When the Titles & Metas screen opens, click on the Post Types tab …
(Yoast SEO: Titles & Metas – Post Types)
In the Post Types section, enable no index in the Meta Robots options to stop search engines from indexing your Posts, Pages, and Media …
(Prevent Search Engines From Indexing Posts, Pages, And Media)
You can also prevent search engines from indexing individual posts or pages on your site by clicking on the plugin’s ‘Advanced’ tab below the Post or Page editor and selecting the following:
- Meta Robots Index – noindex
- Meta Robots Follow – nofollow
(Yoast SEO – Advanced Options)
You can also choose to noindex a certain category, tag, or custom taxonomy, by configuring the plugin’s options in that feature’s settings. For example, the screenshot below shows the WordPress SEO plugin’s settings when editing a category and how to prevent search engines from indexing that entire category …
(Yoast SEO – Post Category Settings)
So far, you have learned how to block search engine results by restricting indexing of your site, sections of your site, and individual posts or pages. Restricting indexing on your site, however, will not remove a page or post that was previously indexed by a search engine, at least until the next time the search engine bots attempt to spider your site (or perhaps even longer).
If a page of your site is already indexed and you would like it removed from the search engines, then do the following:
First, restrict indexing of the page or post using the methods described above. Once you have done this, you can request Google to remove an indexed site, page or post from its search database.
Removing Content That Has Already Been Indexed
Let’s say that you have published or created a post that Google has already found and indexed, and now you want it removed from Google’s search engine results.
Normally, when a page is updated or removed, it will eventually and automatically fall out of Google’s search results. You don’t need to do anything to make this happen. However, if you urgently need to remove content from Google’s search results, you can request expedited removal of those URLs.
For this example, let’s go through the process of asking Google to stop displaying content from a post that it has already indexed.
First, make sure you restrict indexing of the post that you would like removed from Google using one of the methods discussed earlier, then log into your Google Search Console …
(Google Search Console Login)
To learn how to set up a Google Search Console account, see the tutorial below:
Once you have logged into your account, click on your site’s name. This will bring up a list of options for managing your domain …
(Search Console Dashboard)
Go to Search Console > Dashboard > Google Index …
(Search Console > Dashboard > Google Index)
Inside the ‘Google Index’ menu section, click on Remove URLs …
(Google Index > Remove URLs)
This brings you to the ‘Remove URLs’ screen.
Google will temporarily remove your URLs from its search results. Enter the relative URL of the post, page or directory you want Google to remove from its indexing database and click Continue …
(Remove URLs)
The relative URL is the part of the URL without the main domain name. So, in the example above:
- Absolute URL – http://yourdomain.com/private-pricing-info.php
- Relative URL – private-pricing-info.php
Make sure you pay careful attention to the URL you are entering into this field because URLs are case sensitive.
- http://yourdomain.com/PageTitle is not the same as
- http://yourdomain.com/pagetitle
Read the ‘More information’ section to learn more about removing content permanently.
Once you have specified the URL to be removed, the next step is to indicate the request type. From the drop-down menu, choose one of the following options:
- Temporarily hide from search results and remove from cache – Hides the page from Google search results for about 90 days, and also clears the cached copy of the page and snippet. The page can reappear in search results after the blackout period. Google will recrawl the page during the blackout period and refresh the page cache and snippet, but will not show them until the blackout period expires.
- Remove page from cache only – Clears the cached page and snippet, but does not remove the page from search results. Google will refresh the page cache and snippet.
- Temporarily hide directory – Hides an entire directory from search results for about 90 days and also clears cached pages and snippets for all pages in the specified directory. The directory can reappear in search results after the blackout period. Google will recrawl the pages during the blackout period and refresh the page caches and snippets.
(Remove URLs – Request Type)
Click Submit Request …
(Remove URLs – Submit Request)
Your request will be submitted.
For more information about using Google Search Console to block search engine access to your content or remove URLs from its search results, refer to the Search Console help pages below:
Hopefully, now you have a better understanding of methods that you can use to block your site and prevent search engines from accessing areas of your site and your content.
***
"This is an awesome training series. I have a pretty good understanding of WordPress already, but this is helping me to move somewhere from intermediate to advanced user!" - Kim Lednum
***