Community
51 viewsIt’s a fact that as a website owner, you always seek more online visibility and higher rankings on search results.
But there is another side of the coin as well.
Are you okay with revealing your new product before its launch date?
Will you be ready to spoil the surprise of your upcoming promotional event or contest?
We know your answer will be a loud and clear ‘NO‘.
That is why it is essential to direct search engines on how to crawl and index different web pages of your websites that yield maximum benefits.
Robots meta directives allow you to complete that challenging task. They are pieces of code that instruct site crawlers how you want them to crawl and index your web pages.
This article will discuss everything you need to know about robots meta directives and their role in SEO.
So, let’s dive in!
Types Of Robots Meta Directives
There are following two types of robots meta directives,
- Meta Robots Tag
- X-Robots Tag
What Is A Meta Robots Tag?
A meta robots tag is a snippet of HTML code instructing search engines how to crawl, index, and display web page content in search results.
It is usually placed in the head section of a web page.
Example: <meta name=”robots” content=”noindex”>
The above meta robot tag tells search bots not to index the web page’s content.
What Is An X-Robots Tag?
An X-Robots tag is the meta directive added in the HTTP header to control the crawling and indexing of non-HTML files.
X-Robots tag offers more flexibility and functionality compared to robots tags.
How Robot Tags Are Different From Robots.Txt?
Both of them offer similar functions but serve different purposes.
Robots.txt is a standalone file located in the root directory of a domain that is applied to the entire website to direct search bots to which web pages to crawl.
A meta robots tag is an HTML tag applied to a particular web page to direct search bots on how to crawl, index, and display information on that specific web page.
What Are The Uses Of Meta Robot Tags?
This section shall discuss the use of meta robot tags, which will help you decide whether your web pages need one.
Besides instructing search bots how to crawl, index, and display the content of a web page, robots tags also direct them to the following instructions for the web page,
- Whether to include that web page in search results.
- Whether to follow the links on that web page.
- Whether to index the images of the web page.
- Whether to display the cached version of the web page in SERPs.
- Whether to display a snippet of the web page in search results.
Why Are Robots Tags Important For SEO?
A meta robot tag helps search bots crawl and index web pages of a website, which is the first step towards getting your web page rank in search engine results pages.
That is why these tags play a vital role in robots SEO.
The bigger your website, the more you need to manage your crawlability and indexation. Ideally, you want only some web pages of your website to rank in search results.
Instead, you might be eager to prevent the following web pages of your website from getting indexed by search engines,
- Web pages with thin content that offer little valuable information to the users.
- Web pages in the staging environment.
- Internal search results pages.
- PPC landing pages.
- ‘Admin’ and ‘Thank You’ pages.
- Web pages containing duplicate content.
- Web pages about product launches, contests, or upcoming promotional events.
On the other hand, you also want search engines to crawl and index your important pages efficiently to achieve a higher ranking.
That is why the correct combination of robots meta directives and sitemap is crucial for your website’s robot SEO.
What Are The Attributes Of Robots SEO Tags?
There are two attributes of a meta robot tag,
- Name
- Content
Name
The name attribute indicates which search bots you want to follow the directives,
For example,
meta name=”googlebot-image”
That will ensure that Googlebots will follow the instructions for images.
If you want all search bots to follow an instruction, you need to write the name attribute as follows,
meta name=”robots”
Content
The content attribute provides instructions (values) to the search bots.
For example,
content=”noindex”
That will prevent search bots from indexing the web page content.
If you do not use an HTML robots tag, the search bots will index the web page content to display in search results and follow the links on the web page by default.
Google supports the following ‘values’ (instructions) for content,
Noindex
It tells search bots not to index the web page content to prevent it from appearing on the search results.
Nofollow
It instructs search bots not to crawl and follow the links on the web page content. However, bots can still index those links, especially if they contain backlinks.
None
None instruction is the combination of noindex and nofollow. However, you must remember that no search bots except Googlebot support it.
Noarchive
It prevents search bots from displaying a cached copy of the web page in search results. However, you must use ‘nocache’ instruction instead of ‘noarchive’ for ‘Firefox’ and ‘Internet Explorer’.
Notranslate
It prevents Googlebots from offering a translation of the web page content in search results.
Noimageindex
It stops search bots from indexing the images embedded in the web page content.
Nositelinksearchbox
It instructs Google not to show a search box for your website in search results.
If you do not provide this value, Google may show a search box as a part of your site links in SERP.
Nosnippet
It stops search bots from displaying text snippets and video previews of the web page in SERPs.
Max-snippet
The max-snippet value specifies search bots the maximum number of characters they can display in text snippets.
Setting the value as ‘0’ will prevent the display of text snippets, and the value ‘-1’ will instruct no limit on the maximum number of characters.
The above tag limits the maximum number of characters to 160 for text snippets.
Max-image-preview
It specifies the maximum size of the preview image search bots can display in image snippets.
The directive can have the following three values,
None: Google will not display a preview image in image snippets.
Standard: A default preview image may be shown in image snippets.
Large: Google may show the largest possible preview image in image snippets.
Max-video-preview
It specifies the maximum duration of the video search bots can display in video snippets.
Setting the value to ‘0’ will prevent the display of video snippets, and the value ‘-1’ will instruct no limit on the duration of video snippets.
The above tag limits the maximum duration to 15 seconds for video snippets.
Indexifembedded
The directive allows Google to index web page content embedded in another web page through iframes or similar HTML elements despite a noindex directive.
However, you must always use the instruction accompanied by a noindex directive.
Unavailable_after
The directive works like a noindex directive with a timer. It prevents bots from displaying the web page content after a specified date and time.
However, you must specify the date and time using RFC 822, RFC 850, or ISO 8601 formats.
You can use this meta directive for time-sensitive and event pages you want to make available for a limited time.
How To Implement Robots Tags?
Now that you know the functions of the SEO robots tags, it is time to know how to implement them in reality.
Either you can insert them in the head section of the page by editing the HTML code, or you need to put them in the place specified by your content management system with SEO plug-ins.
Let’s consider the example that you want to prevent search bots from indexing the web page content but allow them to follow the links on the page.
Let’s see how to add the meta robot tag for that directive.
HTML Code
You can add your meta tags in the head section of your web page by editing the HTML code.
WordPress
If you are using one of the most popular content management systems, WordPress with Yoast SEO or Rank Math SEO plug-in, you need to execute the following steps,
Yoast SEO
Open the block below the ‘Page Editor’ and click on the ‘Advanced’ tab.
Implement the ‘noindex’ directive by toggling the “Allow search engines to show this page in search results?” drop-down to ‘No’.
Next, you need to toggle the “Should search engines follow links on this page?” drop-down to “Yes” to allow search following the links on the page.
Rank Math
If you are using the Rank Math plug-in, your job will become much easier, as you can directly select the meta directives from the meta box’s ‘Advanced’ tab.
Shopify
You need to edit the head section of the theme.liquid layout file to implement robots tags in Shopify.
You need to add the following code to the layout file,
Once you finish implementing a noindex tag to your desired web pages, you must cross-check if that is really preventing Googlebots from indexing them by using the RankWatch Google index checker for bulk web pages.
How To Implement X-Robots Tag?
X-Robots tag is the meta directive added in the HTTP header to control the crawling and indexing of non-HTML files.
We will consider the same example in which you want to prevent search bots from indexing the web page content but allow them to follow the links on the page.
Apache Server
You need to add the following code to your website’s .htaccess file or httpd.config file on an Apache server,
Nginx Server
You need to add the following code to your website’s .conf file on an Nginx server,
How To Avoid Common Robots Meta Directives Mistakes?
You can avoid common robots meta directives mistakes by following the below tips,
- Do not add Noindex directives to pages blocked in Robots.txt.
- Do not add Noindex directives to Robots.txt file.
- Ensure a proper Sitemap management.
- You must remember to remove the Noindex directives in the production environment.
- Avoid adding ‘Secret’ URLs in Robots.txt file.
Check Your Site’s Crawlability
It is recommended to check your website’s crawlability at regular intervals to identify technical SEO errors using a reliable tool like the RankWatch site auditor.
The tool provides you with all the technical SEO issues stopping your website from ranking higher and categorizes the issues depending on their criticality.
You can resolve the issues to ensure a smooth crawling of your website by the search bots.
Final Thoughts
No website owner targets each web page of the website to rank in search engines.
It is essential to optimize the essential pages offering valuable information for users to rank well in search results.
On the other hand, it is equally important to restrict a few pages from appearing in the SERPs. For example, pages with thin content, the ‘admin’ page, or the ‘thank you’ page.
Hence, it is essential to manage the crawlability and indexability of your web pages as per the requirements.
The various robot meta directives allow you to achieve that, and you now know how to implement them.
However, it is essential to check the crawlability and indexability using a reliable tool after implementing the robots tags.
Share Your Thoughts