How To Set Up Webmaster Accounts

Learn how to set up webmaster accounts for improving your WordPress site SEO and search results.

How To Set Up Webmaster AccountsThis tutorial is part of our WordPress Configuration tutorial series. In this tutorial, you will learn how to set up webmaster accounts for improving your WordPress site SEO and search results.

***

How To Set Up Webmaster Accounts

You want online users to find your website.  One of the most important steps of configuring a WordPress website or blog to drive traffic to your site is to set up webmaster accounts with search engines. Google, Bing, and other search engines provide tools that make it easier to find and rank your site pages.

What Is A Webmaster Account?

Search engines like Google and Bing provide website owners with useful tools for communicating and improving their search results. Basically, think of a webmaster account as Google’s (or Bing’s or another search engine’s) way of helping you to better help them understand what your site is all about and how to properly index and classify your web pages for search users.

Website owners can access these tools by registering and setting up an account and then adding and confirming ownership of their website(s).

Benefits Of Using Webmaster Accounts

Webmaster accounts allow you to set up a ‘feedback loop’ that will help you improve your website’s performance and hence improve the indexing of your webpages and your search results.

Setting up and registering your site(s) with webmaster accounts provide many benefits, including access to:

  • Tools and information that allow you to run a healthy website efficiently
  • Sitemap submissions and crawling and indexing activity on your site
  • Real user data like search statistics
  • Latest data about incoming links and internal links to your web properties
  • Notification of crawl errors, malware, etc.
  • Effective keyword targeting data
  • Vital SEO insights
  • And more.

In this tutorial, you will learn how to set up webmaster accounts that let you integrate information about your WordPress site with search engines like Google, Bing, and Yandex via the Yoast SEO plugin …

Yoast SEO lets you integrate Webmaster Tools with WordPress

(Yoast SEO lets you integrate Webmaster Tools with WordPress)

To learn how to set up and configure the Yoast SEO plugin, see the tutorial below:

Google Search Console

With Google’s Search Console (formerly called Webmaster Tools), once you verify your website, you are able to view a wealth of information, data, and insights about how Google sees your website and access tools that let you adjust and improve your results. Instead of employing ‘middle agents’ to help improve your website’s SEO, you have direct access to the source itself and can make important adjustments.

Google Search Console provides you with important information about how your site is indexed, such as a list of external and internal links to your site, the keywords users typed in to arrive on your site, keyword clickthrough rates, statistics, the last time the Googlebot crawled your homepage successfully, page errors, etc. You can also submit your site’s XML sitemap directly to Google in order to help Google index all of your websites pages. After submitting a new sitemap, you can view the status of the submission and see how many URLs have been indexed and view a detailed report on any errors detected by Google.

Google also provides essential diagnostic data about your website, including web crawl (mobile included) and content analysis data, which lets you know if you have errors that affect your site’s indexing results and SEO performance such as duplicate title tags, meta descriptions or pages with missing titles.

Info

We recommend integrating your Google Analytics account details to your Google Search Console account for even better data insights and results (see ‘Google Search Console’ tutorial below).

If you are concerned about privacy issues, you may not want to allow Google and other analytical tools to access your data. If this is the case, see our tutorial on how to block search engines from accessing your WordPress site.

Register Your Website With Google Search Console

To register your website with Google Search Console and access your webmaster account tools, go to Google Search Console

Google Search Console Login Screen

(Google Search Console Login Screen)

Log in with your Google Account details.

If you need help creating a Google account, see the tutorial below:

Once you are logged into Search Console, click on ‘Add A Property’ to add your website …

Google Search Console

(Google Search Console)

Enter your website URL and click the ‘Add’ button …

Add your site to Google Search Console

(Add your site to Google Search Console)

You will be presented with a number of website verification methods. You can verify your website using any of the available methods.

If you plan to use the Yoast SEO plugin (which we recommend you do), then go to the ‘Alternate methods’ tab, click on HTML tag, and select and copy the meta tag Google generates for you to your clipboard …

(Copy the website verification code to your clipboard)

If you have installed the Yoast SEO plugin, log into your WordPress administration area, go to Yoast SEO > Dashboard > Webmaster Tools and paste the verification code into the Google Search Console field, then click ‘Save Changes’ …

Add Google Verification Code To Yoast SEO Webmaster Tools

(Add Google Verification Code To Yoast SEO Webmaster Tools)

After pasting in the verification code, go back to the Google Search Console screen and click ‘Verify’ to verify and confirm …

(Verify and confirm your website with Google)

After your site has been verified, you can access important information about how Google sees your website …

Your site is now verified with Google

(Your site is now verified with Google)

Add Your XML Sitemap To Google Search Console

We recommend adding your XML sitemap file to Google Search Console.

To learn more about creating an XML sitemap, see the tutorial below:

To add your sitemap, go to Crawl > Sitemap and click on the ‘Add/Test Sitemap’ button …

Add a new sitemap to Google

(Add a new sitemap to Google)

Type or paste your sitemap URL into the ‘Add/Test Sitemap’ field and click ‘Submit’ …

Submit your XML sitemap

(Submit your XML sitemap)

After your sitemap has been submitted, click on ‘Refresh the page’ …

Sitemap submitted

(Sitemap submitted)

Google will now start crawling and indexing the URLs you have submitted.

After setting up your account with Google Search Console, you will have access to a range of useful information about your website and tools that will help you improve your site’s ranking and indexing …

Google Search Console - Google Index

(Google Search Console – Google Index)

To learn more about using Google Search Console, see the site’s help section below:

Search Console Help Center

(Search Console Help Center)

Congratulations, you have setup your webmaster account with Google!

Bing Webmaster Tools

Bing provides a suite of reporting and diagnostic tools and information about your website to help you get more of your site indexed and ranked in the Bing search engine.

Create An Account With Bing Webmaster Tools

To create an account, go to Bing Webmaster Tools

Bing Webmaster Tools

(Bing Webmaster Tools)

To create an account with Bing Webmaster Tools, you will need a Microsoft account (formerly known as a Windows Live ID).

Important

If you have a Hotmail or Outlook.com account you likely already have a Microsoft account. If not, you can get an account for free here. Alternatively, just click the link Sign-Up with Microsoft account and create a Microsoft account as part of the Webmaster Tools sign-up process.

If you have not registered before, click on ‘Sign Up’ …

Bing Webmaster Tools - Sign Up

(Bing Webmaster Tools – Sign Up)

After you have set up your Microsoft account, sign in to access the Bing Webmaster Tools area …

Log into Bing Webmaster Tools using your Microsoft account

(Log into Bing Webmaster Tools using your Microsoft account)

Register Your Website With Bing Webmaster Tools

To add your website(s) to Bing Webmaster Tools, enter your website URL into the My Sites > Add a Site field and click on the ‘Add’ button …

Add your website URL to Bing Webmaster Tools

(Add your website URL to Bing Webmaster Tools)

Once your site has been added to Bing Webmaster Tools, you will be provided with a verification code. Select this code and copy it to your clipboard …

Bing authentication code

(Bing authentication code)

If you have installed the Yoast SEO plugin, log into your WordPress administration area, go to Yoast SEO > Dashboard > Webmaster Tools and paste the code into the Bing Webmaster Tools field, then click ‘Save Changes’ …

Add Bing Verification Code To Yoast SEO Webmaster Tool

(Add Bing Verification Code To Yoast SEO Webmaster Tools)

After pasting in the verification code, go back to the Bing Webmaster screen, scroll down to the bottom of the form and click ‘Verify’ to verify and confirm your site …

(Verify and confirm your website with Bing)

Once your site has been verified, you will have access to webmaster tools and information about how Bing sees your website …

Once your site is verified Bing gives you access to tools and information about your website

(Once your site is verified Bing gives you access to tools and information about your website)

Add Your XML Sitemap To Bing Webmaster Tools

We recommend adding your XML sitemap file to Bing Webmaster Tools.

To learn more about creating an XML sitemap, see the tutorial below:

When adding a new website to Bing Webmaster Tools, you are given the opportunity to also add a sitemap while registering your site. Type or paste your sitemap URL into the ‘Add a sitemap’ field and click ‘Add’ …

Add your sitemap

(Add your sitemap)

To add a new sitemap to Bing, go to Dashboard > Configure My Site > Sitemaps, enter your sitemap URL and click on ‘Submit’ …

Submit a new sitemap to Bing

(Submit a new sitemap to Bing)

Bing bots will regularly crawl and index the URLs listed on your sitemap.

To learn more about using Bing Webmaster Tools, see the site’s help section below:

Bing Webmaster Tools Help & How-To Center

(Bing Webmaster Tools Help & How-To Center)

Congratulations, you have set up your webmaster account with Bing!

Yandex Webmaster

Yandex is a Russian multinational technology company specializing in Internet-related services and products. Yandex operates the largest search engine in Russia with about 60% market share in that country.

(Source: Wikipedia)

Yandex.com

(Yandex.com)

Yandex.Webmaster is a service provided by Yandex.com that offers information on how your sites are indexed. You can use it to let Yandex know about new or deleted pages, make changes to how your site is indexed, and improve your site’s search ranking.

With Yandex.Webmaster, you have access to information such as:

  • Which problems have come up during indexing
  • Which pages have dropped in the search ranking
  • Who is linking to pages on your site
  • Which search queries lead users to your site
  • How to protect your site from viruses
  • How to make your site look better in search results
  • And more …

In this section, we’ll show you how to register and set up an account with Yandex.Webmaster.

Create An Account With Yandex Webmaster

To create an account with Yandex.Webmaster, go to Webmaster.Yandex.com …

Yandex.Webmaster

(Yandex.Webmaster)

Click on Log In

Yandex.Webmaster Log in

(Yandex.Webmaster Log in)

If you haven’t set up an account yet, then click on the Register button to create a new account …

Create an account with Yandex.Webmaster

(Create an account with Yandex.Webmaster)

Complete the online registration form and confirm your mobile phone number, then click the ‘Register’ button …

Yandex.Webmaster online registration form

(Yandex.Webmaster online registration form)

Register Your Website With Yandex Webmaster

After registration, you will be taken to the Webmaster > My sites screen. Click on the ‘+’ button to add your site …

Add your site to Yandex Webmaster tools

(Add your site to Yandex Webmaster tools)

Enter your website URL into the Site address field and click on the ‘Add’ button …

Add your website URL to Yandex Webmaster

(Add your website URL)

Once your site has been added to Yandex Webmaster Tools, you will be provided with a verification code. Select this code and copy it to your clipboard …

Yandex Verification Code

(Yandex Verification Code)

If you have installed the Yoast SEO plugin, log into your WordPress administration area, go to Yoast SEO > Dashboard > Webmaster Tools and paste the verification code into the Yandex Webmaster Tools field, then click ‘Save Changes’ …

Add Yandex Verification Code To Yoast SEO Webmaster Tools

(Add Yandex Verification Code To Yoast SEO Webmaster Tools)

After pasting in the verification code, go back to the Yandex Webmaster screen and click ‘Check’ to verify and confirm …

Verify and confirm your website with Yandex

(Verify and confirm your website with Yandex)

Once your site has been verified, you will see your details listed in the ‘Access rights’ screen under ‘Users who manage site’ …

Your site is now verified with Yandex

(Your site is now verified with Yandex)

After setting up your account with Yandex Webmaster, you will have access to a range of useful information about your website and tools that will help you improve your site’s ranking and indexing …

Yandex Webmaster provides data, insights, and tools for your website

(Yandex Webmaster provides data, insights, and tools for your website)

Add Your XML Sitemap To Yandex Webmaster

We recommend adding your XML sitemap file to Yandex.

To learn more about creating an XML sitemap, see the tutorial below:

To add your sitemap, go to Indexing settings > Sitemap files, then type or paste your sitemap URL into the ‘Add Sitemap file’ field and click ‘Add’ …

Add your sitemap file to Yandex

(Add your sitemap file to Yandex)

Yandex bots crawl and index sitemap files every two weeks, so allow a few weeks for your site’s URLs to be crawled before you see results.

To learn more about using Yandex Webmaster Tools, see the site’s help section below:

Yandex Webmaster Help Site

(Yandex Webmaster Help Site)

Useful Tip

If you use the Google Chrome browser to browse the Yandex Help site and find all of the tutorials written in Russian, you can use Google’s translation features to translate the content into your own language.

Congratulations, you have setup your webmaster account with Yandex!

Add Your WordPress RSS Feed To Google Search Console

We also recommend adding your WordPress RSS feed to Google’s Search Console (and the other search engines listed above). This will also help to index your content faster.

To learn more about the WordPress RSS feed, see the tutorial below:

To add your RSS feed, go to Crawl > Sitemap and click on the ‘Add/Test Sitemap’ button …

Add a new feed to Google

(Add a new feed to Google)

Type or paste your feed URL into the ‘Add/Test Sitemap’ field and click ‘Submit’ …

Submit your WordPress RSS feed

(Submit your WordPress RSS feed)

After your feed has been submitted, click on ‘Refresh the page’ …

Sitemap submitted

(WordPress RSS feed submitted)

Your WordPress RSS feed is now added to Google’s search console. Google will now start crawling and indexing the URLs you have submitted.

WordPress RSS feed added to Google Search Console

(WordPress RSS feed added to Google Search Console)

Useful Tip

Repeat the above process to add your WordPress RSS feed to Bing’s search consoles.

Congratulations! Now you know how to set up webmaster accounts and how to add XML sitemaps and RSS feeds to notify search engines about your content.

How To Set Up Webmaster Accounts

(Source: Pixabay)

***

"This is AMAZING! I had learnt about how to use WordPress previously, but this covers absolutely everything and more!! Incredible value! Thank you!" - Monique, Warrior Forum

***

How To Block WordPress From Search Engines

Learn how to block WordPress from search engines by configuring your WordPress settings and plugins in this tutorial

How To Block WordPress From Search EnginesThis tutorial is part of our WordPress configuration step-by-step training series, where we show you how to set up and configure your WordPress site or blog with no coding skills required.

***

How To Block Your WordPress Site From Search Engines

WordPress combines an easy-to-use content publishing and management system with a powerful business marketing tool that when expertly configured, allows search engines to easily find your site and quickly index your posts and pages. Most of our tutorials assume that you actually want search engines to index your site.

There may be instances, however, when you may want to block search engines from indexing your site.

This tutorial covers some of the possible reasons why you would want to block your site from being indexed by search engines and shows you how to prevent search engines from indexing your site using various methods, including configuring WordPress settings and using different WordPress features and plugins.

Why Block Your WordPress Site From Search Engines?

Some of the reasons why you may want to block search engines from indexing your WordPress site can include:

  • Personal Use. You may want a personal blog to post news or updates about yourself to friends or family only (you could use Facebook for this).
  • Restrict Information To Trade Or Staff. You may want a private site to post confidential or restricted information about your business to clients and/or staff only, such as pricing, company news, updates, downloads or internal training.
  • Inform Members Of A Local Community. You may want a site for posting administrative information pertaining only to members of a local community such as a business network, local community organization, church, network marketing downline, or special interest group.
  • Hide Subscriber-Only Content – You may want to send subscribers to a site containing exclusive information that you don’t want listed in search engines and don’t want to set up a membership site.
  • Testing & Experimental Purposes. You may want to set up a WordPress site for internal testing or experimental purposes, such as installing and testing different themes and plugins, or creating draft content.
  • Demo Purposes. You may want to use a WordPress site for demo purposes, such as educational or training seminars and workshops, sales presentations of your products, etc.

How To Block WordPress From Search Engines – Step-By-Step Tutorial

There are various methods you can use to prevent search engines from indexing your site.

These include:

  • Configuring WordPress settings
  • Using built-in WordPress features
  • Using WordPress plugins

Blocking Your WordPress Site From Search Engines – Configuring WordPress Settings

The first method you can use to help prevent search engines from crawling and indexing your site is to configure your WordPress Reading Settings

You can discourage search engines from indexing your site

(You can discourage search engines from indexing your site)

After configuring your WordPress Reading Settings to discourage search engines from indexing your site, WordPress will also stop automatically notifying the update services listed in your WordPress Writing Settings

You can block WordPress from automatically notifying update services.

(You can block WordPress from automatically notifying update services)

To learn more about configuring your WordPress Writing Settings, see the tutorial below:

To learn more about configuring your WordPress Reading Settings, see the tutorial below:

Important

Configuring the above settings will allow visitors continued access to your site and will instruct search engines not to index your site, but it cannot force-block access to your site. It is up to search engines to honor your request. Malware and spam bots will typically ignore it.

Search Engines Discouraged

(Search Engines Discouraged)

robots.txt File

The next step in the process of blocking search engines from accessing your site is to configure a file in your server called robots.txt

robots.txt

(robots.txt)

A robots.txt file is used to instruct search engine spiders and co-operative bots which parts of your website they can or should not access.

To learn how to create, configure and add a robots.txt file to your server, see the tutorial below:

How To Block WordPress From Search Engines – Using Built-In WordPress Features

If you want to block or restrict certain parts of your website from being indexed by search engines, there are certain built-in features of WordPress you can use.

  • You can make pages and posts private,
  • You can password-protect pages and posts,
  • You can exclude pages and posts from XML sitemaps, HTML site maps, indexable custom menus, etc.

We have created step-by-step tutorials showing you how to block or restrict certain parts of your website from being indexed by search engines using all of the above built-in features of WordPress.

To learn more about protecting content in WordPress, see the tutorial below:

Blocking Your WordPress Site From Search Engines – Using WordPress Plugins

You can use WordPress plugins to block your site or restrict sections of your site or content from being indexed by search engines. Some of the strategies you can use with plugins include:

  • Creating a Membership Site using plugins that restrict access to content to specific users and can be configured to prevent search engines and bots from also accessing and indexing private areas and content on your site.
  • Putting your site in Maintenance Mode. This is useful if you want to block everyone from accessing your site (e.g. putting your site “under construction” while you work on improving it).
  • Installing a WordPress SEO plugin and configuring its settings to select which parts of your site should and shouldn’t be indexed by search engines.
Blocking Content From Search Engines With Yoast SEO

If you have the Yoast SEO WordPress plugin installed, you can configure its settings to prevent search engines from indexing Posts, Pages, and Media on your site.

To access these settings, go to your WordPress admin menu and choose SEO > Titles & Metas

Yoast SEO - Titles & Metas

(Yoast SEO – Titles & Metas)

When the Titles & Metas screen opens, click on the Post Types tab …

Yoast SEO: Titles & Metas - Post Types

(Yoast SEO: Titles & Metas – Post Types)

In the Post Types section, enable no index in the Meta Robots options to stop search engines from indexing your Posts, Pages, and Media …

Prevent Search Engines From Indexing Posts, Pages, And Media

(Prevent Search Engines From Indexing Posts, Pages, And Media)

You can also prevent search engines from indexing individual posts or pages on your site by clicking on the plugin’s ‘Advanced’ tab below the Post or Page editor and selecting the following:

  • Meta Robots Index – noindex
  • Meta Robots Follow – nofollow

Yoast SEO - Advanced Options

(Yoast SEO – Advanced Options)

You can also choose to noindex a certain category, tag, or custom taxonomy, by configuring the plugin’s options in that feature’s settings. For example, the screenshot below shows the WordPress SEO plugin’s settings when editing a category and how to prevent search engines from indexing that entire category …

Yoast SEO - Post Category Settings

(Yoast SEO – Post Category Settings)

So far, you have learned how to block search engine results by restricting indexing of your site, sections of your site, and individual posts or pages. Restricting indexing on your site, however, will not remove a page or post that was previously indexed by a search engine, at least until the next time the search engine bots attempt to spider your site (or perhaps even longer).

If a page of your site is already indexed and you would like it removed from the search engines, then do the following:

First, restrict indexing of the page or post using the methods described above. Once you have done this, you can request Google to remove an indexed site, page or post from its search database.

How To Remove Content That Has Already Been Indexed

Let’s say that you have published or created a post that Google has already found and indexed, and now you want it removed from Google’s search engine results.

Normally, when a page is updated or removed, it will eventually and automatically fall out of Google’s search results. You don’t need to do anything to make this happen. However, if you urgently need to remove content from Google’s search results, you can request expedited removal of those URLs.

For this example, let’s go through the process of asking Google to stop displaying content from a post that it has already indexed.

First, make sure you restrict indexing of the post that you would like removed from Google using one of the methods discussed earlier, then log into your Google Search Console

Google Search Console Login

(Google Search Console Login)

To learn how to set up a Google Search Console account, see the tutorial below:

Once you have logged into your account, click on your site’s name. This will bring up a list of options for managing your domain …

Search Console Dashboard

(Search Console Dashboard)

Go to Search Console > Dashboard > Google Index

Search Console > Dashboard > Google Index

(Search Console > Dashboard > Google Index)

Inside the ‘Google Index’ menu section, click on Remove URLs

Google Index > Remove URLs

(Google Index > Remove URLs)

This brings you to the ‘Remove URLs’ screen.

Google will temporarily remove your URLs from its search results. Enter the relative URL of the post, page or directory you want Google to remove from its indexing database and click Continue …

Google Search Console - Remove URLs

(Remove URLs)

Important Info

The relative URL is the part of the URL without the main domain name. So, in the example above:

  • Absolute URLhttp://yourdomain.com/private-pricing-info.php
  • Relative URL – private-pricing-info.php

Make sure you pay careful attention to the URL you are entering into this field because URLs are case sensitive.

  • http://yourdomain.com/PageTitle is not the same as
  • http://yourdomain.com/pagetitle

Read the ‘More information’ section to learn more about removing content permanently.

Once you have specified the URL to be removed, the next step is to indicate the request type. From the drop-down menu, choose one of the following options:

  • Temporarily hide from search results and remove from cache – Hides the page from Google search results for about 90 days, and also clears the cached copy of the page and snippet. The page can reappear in search results after the blackout period. Google will recrawl the page during the blackout period and refresh the page cache and snippet, but will not show them until the blackout period expires.
  • Remove page from cache only – Clears the cached page and snippet, but does not remove the page from search results. Google will refresh the page cache and snippet.
  • Temporarily hide directory – Hides an entire directory from search results for about 90 days and also clears cached pages and snippets for all pages in the specified directory. The directory can reappear in search results after the blackout period. Google will recrawl the pages during the blackout period and refresh the page caches and snippets.

Remove URLs - Request Type

(Remove URLs – Request Type)

Click Submit Request …

Remove URLs - Submit Request

(Remove URLs – Submit Request)

Your request will be submitted.

For more information about using Google Search Console to block search engine access to your content or remove URLs from its search results, refer to the Search Console help pages below:

Hopefully, now you have a better understanding of methods that you can use to block your site and prevent search engines from accessing areas of your site and your content.

How To Block Your WordPress Site From Search Engines

***

"Learning WordPress has been a huge stumbling block for me. I've been looking for something that covers absolutely everything but doesn't cost an arm and a leg. Thank you so much ... you have just provided me with what I have been looking for! Truly appreciated!" - Tanya

***

How To Create A Robots.txt File

This tutorial explains what a robots.txt file is and how to create a robots.txt file and upload it to your server.

Get Thrive Leads for WordPress

How To Create A Robots.txt FileAlways backup your WordPress site before making changes to important files and make copies of any files that you are about to modify.

If you don’t want to edit WordPress files yourself, ask a professional to help you.

***

Important Info

WordPress generates a virtual robots.txt file if there is no physical robots.txt file found on your server. In this tutorial, you will learn how to create and upload a physical robots.txt file to your server to override the virtual file, how to edit your robots.txt file and how to test your robots.txt file instructions to search engines.

Robots File

A robots.txt file instructs search engine spiders and co-operative ‘bots’ about the parts of your website that you want to keep private, and which areas they can and cannot access. Bots, or ‘robots’, are programs used by search engines like Google and other software to gather information for their databases.

About robots.txt

The following entry is sourced from robotstxt.org:

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

It works likes this: a robot wants to visit a website URL, say http://www.yourdomain.com/welcome.php. Before it does so, it firsts checks for http://www.yourdomain.com/robots.txt, and finds:

How To Create A Robots.txt File

The “User-agent: *” part contains an asterisk.

As stated in Wikipedia,

In software, a wildcard character is a single character, such as an asterisk (*), used to represent a number of characters or an empty string. It is often used in file searches so the full name need not be typed.

(Source: Wikipedia)

This means that this section applies to all robots.

The “Disallow: /” part tells the robot that it should not visit any pages on the site.

There are two important things to consider when using /robots.txt:

  • Robots can ignore your /robots.txt instructions. This includes malware robots that scan the web for security vulnerabilities and email address harvesters used by spammers.
  • The /robots.txt file is a publicly available file. Anyone can see what sections of your server you don’t want robots to use.

So, don’t use /robots.txt to try and hide information.

If you need to protect content on your WordPress site, see the tutorial below:

Accessing Your robots.txt File

Your robots.txt file must be located in the same directory as your WordPress installation …

Location of robots.txt file

(Location of robots.txt file)

To view your site’s robots.txt file after it has been installed, simply type your site’s URL into your browser and add “robots.txt” to the end of the URL, e.g.:

http://www.yoursite.com/robots.txt

This will bring up your robots.txt file …

robots.txt file

(robots.txt file)

How To Configure Your Robots File

Typically, with a new WordPress installation, if you configure your WordPress Reading Settings to allow search engines to index your site, your robots.txt file will look something like this …

WordPress Reading Settings - Search Engine Visibility Allowed

(WordPress Reading Settings – Search Engine Visibility Allowed)

If you disable the ‘Search Engine Visibility’ option in the Reading settings, the instructions in your robots.txt file will be modified to discourage search engines from crawling your site …

WordPress Reading Settings - Search Engine Visibility Disallowed

(WordPress Reading Settings – Search Engine Visibility Disallowed)

A robots.txt file can be configured in many different ways, depending on what instructions you want to give to search engine spiders and other visiting bots (robots).

Here are just some examples …

Examples of robots.txt file instructions

(Examples of robots.txt file instructions. Source: Wikipedia)

How To Create A Robots.txt File – Step-By-Step Tutorial

Important

Before creating a robots.txt file, we recommend creating an XML sitemap for your site.

  • If you plan to use a WordPress SEO plugin like Yoast SEO (recommended), the plugin will automatically create an XML sitemap for your website and add these to your WordPress installation.
  • If you do not plan to use the Yoast SEO plugin, you can use a standalone WordPress XML Sitemap plugin like Google XML Sitemaps to generate an XML sitemap for your site.

Let’s show you how to manually create, configure and upload a robots.txt file to your server that performs a number of important functions …

robots.txt file

(robots.txt file)

At the very top of the robots.txt file (i.e. the first line) you will add a link to your site’s XML sitemap. This sitemap enables search engines to find and index all of your site’s pages faster.

Sitemap: http://www.yoursite.org/sitemap.xml

Under the sitemap entry, a recommended entry is Google’s ‘Mediapartners’ agent. By allowing this agent access, you prevent white space or public service ads from appearing on your pages if you have added Google AdSense to your site. This happens if Google has not had the opportunity to index a page on your site with Google AdSense ads or determine what the page is about yet.

The entry looks like this:

User-agent: Mediapartners-Google Disallow:

The rest of the text in your file can be organized however you like. In the sample robots.txt file provided below, you’ll see that the file text has been organized into segments or groupings for ease of readability and management.

In our sample robots.txt file, two groupings are preceded by “User-agent: *” followed by a list of directories or files with each list item preceded by “Disallow“. This represents the directories and files we don’t want search engines to access.

For example, if you have a folder on your site called “private” that you do not want search robots to crawl, then you would add the following line into your robots.txt file:

Disallow: /private/

Of particular note are the image files – since searches for some images (e.g. system images) can result in non-profitable access to your site, you may want to disallow search engines from accessing these.

What follows these sections in our example robots.txt file is a long list of user-agents that are prevented from accessing the site. These are selected not necessarily because they are ‘bad’ but because they would simply use up your server’s bandwidth and other resources if allowed to access and spider your site. The entries for these items look similar to the example below:

User-agent: ia_archiver Disallow: /

Having “/” as a “Disallow” value prevents any access to any file or directory on your site.

Adding A Robots.txt File To Your Site

Click the share button below to download sample robots.txt files that you can use for your own site:

Important

If using the sample robots.txt file provided above, feel free to copy and use the file as is, but make sure to make the changes indicated in the file as shown below:

How To Create A Robots.txt File

Modify the portions in red (i.e. replace yourdomain.com with your URL), and remove the instructions (including the brackets), then resave and reupload the file to your server (see the next section for details on adding a robots.txt file to your server).

Editing Your Robots.txt File

If using the sample robots.txt file provided, either download the zip file provided above or click on the link provided to view the robots.txt file in a new browser window.

Do one of the following:

Option #1

  1. Click “File” > “Save Page As …”
  2. Save the page to a location on your hard drive. The file will be saved as “robots.txt”
  3. Open up the file and change the first line of the file as per the above instructions.
  4. Resave and upload the robots.txt file via FTP to the root directory of your site.

Option #2

  1. Select all of the content in the browser window and copy it to your clipboard.
  2. Create a new plain text file (e.g. notepad).
  3. Paste the contents of your clipboard into the text file.
  4. Save the file as “robots.txt”
  5. Open up the file and change the first line of the file as per the above instructions.
  6. Resave and upload via FTP to the root directory of your site.

After uploading the robots.txt file to your server, verify that it has been uploaded correctly (i.e. open a browser and go to http://www.yoursite.com/robots.txt).

Also, check the following:

  • All the folders you want disallowed have been entered correctly
  • “Disallow” is correctly defined. For example, “Disallow: /” means no access at all, “Disallow:” (with no proceeding value) means full access.
  • You have not left spaces or extra characters. Search engines are very particular and may not respond in the way you expect them to. For example, too many spaces between groupings is not advised (one is ok). Also, special characters like ‘#’ have special meaning in the robots.txt file so be sure to leave those out.

Testing Your Robots.txt File

You can test your robots.txt file using a checking tool like the robots.txt Tester provided by Google Search Console (formerly Google Webmaster Tools).

Important

To use the robots.txt file checker tool you will need to have a Google Search Console  account set up.

If you haven’t set up your webmaster accounts yet, see the tutorial below:

To test a site’s robots.txt file, do the following:

Log into your Google Search Console account …

Google Search Console Login

(Google Search Console Login)

On the Google Search Console home screen, click the site you want to check …

Google Search Console Home Page

(Google Search Console Home Screen)

In the Search Console menu, select Crawl > robots.txt Tester …

Crawl > robots.txt Tester

(Crawl > robots.txt Tester)

This brings you to the robots.txt Tester screen …

robots.txt Tester screen

(robots.txt Tester screen)

If your site has been set up correctly and already indexed by Google, the tool will populate the fields on this screen with information about your site.

You can view the content of your robots.txt file and check for errors or warnings …

Inspect your robots.txt file content

(Inspect your robots.txt file content)

You can also enter URLs into a URL test field, select the user-agents you want to test against from the ‘User-agents’ drop-down menu list (e.g. Googlebot, Googlebot mobile, Mediapartners-Google, etc.) …

Test your URLs to see if any are blocked

(Test your URLs to see if any are blocked)

Click the Test button to analyze your site and view your results.

The tool will show you if the URL or directory is allowed (i.e. can be indexed by Google) …

Allowed - Google will crawl this URL or directory

(Allowed – Google will crawl this URL or directory)

Or if the URL or directory is not allowed …

Blocked - Google will not crawl this URL or directory

(Blocked – Google will not crawl this URL or directory)

Important Info

Any changes you make in this tool will not be saved. To save any changes, you’ll need to copy the contents and paste them into your robots.txt file.

Useful Tip

Refer to the Google Webmaster Tools Help documentation for more details on how to use the tools and how to analyze your results.

Robots.txt – Additional Information

If you have a WordPress SEO Plugin like Yoast SEO installed and discourage search engines from indexing your site (see How To Configure WordPress Reading Settings), you will block access to robots and get an error message like the one shown below …

Blocking access to robots can affect some SEO plugin settings

(Blocking access to robots can affect some SEO plugin settings)

This makes sense … why would you want to optimize your website for search engines if you are instructing WordPress to block access to search engines from visitng your site?

To learn more about configuring WordPress SEO Plugins, see the tutorial below:

To learn more about configuring your WordPress Reading Settings, see the tutorial below:

To learn how to effectively block your WordPress site from being accessed by search engines, see the tutorial below:

For technical information on the benefits and advantages of using a robots.txt file, visit this site:

Congratulations! Now you know how to add a robots.txt file to prevent search engines and compliant bots from indexing pages or sections of your site.

How To Create A Robots.txt File

(Source: Pixabay)

***

"I am beyond impressed with what you have put together. I can tell that you put a ton of hard work into building what you have. You have the absolute best content on WordPress I have ever seen!" - Robert T. Jillie

***