How To Use Robots Meta Tag How do you explain a spider of a search engine you only want him to index just the first page of your website or he is allowed to index the ...

How To Use Robots Meta Tag

How do you explain a spider of a search engine you only want him to index just the first page of your website or he is allowed to index the whole website. You use the so called ROBOTS tag.

The robots meta tag is not the same as the file called robots.txt You should use these two together. Both are used by the seach engines like Yahoo and Google. If you use this meta tag the wrong way you might shut the search engines out. So the influence of this meta tag is significant.

Robots meta tag

Meta robots tag controls spiders from crawling and indexing your webpage or post. In this post, we will see all types of tags in detail. We can use more than one meta tags in a webpage but its not a good practice because it can cause conflicts. Hence it should occur only once in a webpage.

Using the robots meta tag

The robots meta tag lets you utilize a granular, page-specific approach to controlling how an individual page should be indexed and served to users in search results. Place the robots meta tag in the <head> section of a given page, like this: section of a given page,
<!DOCTYPE html>
<meta name="robots" content="noindex" />

Valid Indexing & Serving Directives

The following table shows all the directives that Google honors and their meaning. Note: it is possible that these directives may not be treated the same by all other search engine crawlers. Multiple directives may be combined in a comma-separated list (see below for the handling of combined directives). These directives are not case-sensitive.

all There are no restrictions for indexing or serving. Note: this
directive is the default value and has no effect if explicitly
noindex Do not show this page in search results and
do not show a "Cached" link in search results.
nofollow Do not follow the links on this page
none Equivalent to noindex, nofollow
noarchive Do not show a "Cached" link in search results.
nosnippet Do not show a snippet in the search results
for this page
noodp Do not use metadata from the
Open Directory project for titles or
snippets shown for this page.
notranslate Do not offer translation of this page in
search results.
noimageindex Do not index images on this page.
unavailable_after: [RFC-850 date/time] Do not show this
page in search results after the specified date/time. The date/time
must be specified in the
RFC 850 format.

Example meta tag robots

<meta name="robots" content="index, follow">
<meta name="robots" content="noindex, nofollow" />
<meta name="robots" content="index, nofollow">
<meta name="robots" content="noindex, follow">
<meta name="robots" content="noarchive" />
<meta content='all' name='robots'/>
<meta content='noodp, noydir' name='robots'/>

Noodp= No Open Directory Project
Noydir= No Yahoo Directory

These are used if your website is listed in one of these directories with information you do not want used in the results pages. This might be the case if you have old, outdated listings that no longer apply. They tell robots not to use information from these sources, and they are optional.

What does “noydir,noodp” mean?

The noydir and noodp are Meta Robots commands that instruct search engine bots on how to handle the usage of data for your site in the Yahoo! Directory and the Open Directory Project/DMOZ Directory.

The noydir command corresponds to the Yahoo! Directory and the noodp command correlates to DMOZ.

noydir Explained

When a site is listed in the Yahoo! Directory there is a title and description associated with that listing. So, Yahoo! started showing that title and description as the linking text and snippet text in the search engine results page (SERP) whenver that website would show up (most commonly only the homepage). The noydir command overrides this functionality and gives control back to the website owner to control the linking and snippet text in the SERPs via the Title tag and Meta Description tag on their website. By regaining this control you can adjust those fields whenever desired.

noodp Explained

The same functionality goes for the noodp command. However, this is directed at Google. Google partnered with ODP to power the Google Directory so if you website was listed in DMOZ then Google would pull that listing information as the linking and snippet text in the Google SERPs. The noodp overrides that functionality and give you control back.


The most common implementation is to do both commands simultaneously as in your question like such:
<meta name=”robots” content=”noydir,noodp” />
(content field order does not matter)

They can also be listed separately as such:
<meta name=”robots” content=”noydir” />
<meta name=”robots” content=”noodp” />

You can also direct the commands to the specific bots as such:
<meta name=”googlebot” content=”noodp” />
<meta name=”slurp” content=”noydir” />

It is hoped that these examples illustrate several of the methods at your disposal that will help you to tailor the way search engines index a given web page.

For further reading:

Google and NOODP
Yahoo! and NOYDIR

How to block specific search engine bot

By specifying meta name as “robots”,you are actually blocking all search engine robots. However if you want to block specific search engine bot then you can specify bot name instead of “robots”

BOT Name:


Yahoo: SLURP



So, if you want to make your page nofollow for Google bot only then you can use below tag.
<meta name="Googlebot" content="noindex" />
Note: In meta name I have given “Googlebot” instead of “robots” to make it applicable for Google search engine bot only.
Share it:


How To



Post A Comment: