How to make robots work for you

[cthulhu_waves]

(lifted from the Dating Factory blog)

Мета-tags represent information from particular pages within a website. Robots are sometimes called ?bots? or ?spiders? or ?crawlers? which are really types of software used by search engine companies to search and scan website pages. Meta tags are pieces of software used in webpage development to send specific messages to the robots. There is a range of meta tags used by robots, so it is important for you to understand how robots work on your pages and what you can get from them:

? <meta name=?Keywords? content=???/> ? this tag is taken into account when robots check to see if your pages comply with search queries

? <meta name=?Description? content=???/> ? the content in this tag will be used in a brief description of your site in search results

? <meta http-equiv=?Content-Type? content=?type; charset=??/> ? this tag helps robots identify the type of the document and its coding method

? <meta http-equiv=?refresh? content=?N;url=_URL of redirection_?/> ? this one redirects a visitor to the page given in the url address after having been on the primary page for N seconds. When N is low, the structure is treated as timely redirection and is similar to redirects set up on the server side, like HTTP 302/303/307.


Some more commands for robots:

? <meta name=?robots? content=?all?/> ? tells the robot it is allowed to index the text and the links on the pages, it is similar to <meta name=?robots? content=?index, follow?/>

? <meta name=?robots? content=?noindex?/> ? do not index the text of the page

? <meta name=?robots? content=?nofollow?/> ? do not follow links on the page

? <meta name=?robots? content=?none?/> ? it is prohibited to index text and follow links on the page, this works equally to <meta name=?robots? content=?noindex, nofollow?/>

? <meta name=?robots? content=?noarchive?/> ? do not show a link to a saved copy on the search results page.


In the attribute name you can specify what exact robots you do not want to visit the page:

<meta name=?google? content=?all?/>

Robots will not get to know about files if links to these files contain the nofollow tag. However, if there are other links available without the nofollow tag, the pages will be shown. If meta tags are not specified, robots take them as allowing all actions. If there is a conflict between tags, the robot will create a rule for reading tags as allowing all actions.

For example:

<meta name=?robots? content=?all?/>

<meta name=?robots? content=?noindex, follow?/>

The robot will select the all attribute and your links will be indexed.

Exception! The attribute ?all? does not affect the noarchive attribute:

<meta name=?robots? content=?all?/>

<meta name=?robots? content=?noarchive?/>


In this case both text and links will be indexed, but when giving the search results there will be no link for the saved copy of your page.

<noindex> tag

In order to prohibit indexation of service texts you can use the <noindex> tag. It works the same way as the <nofollow> tag but relates to the content included into the tag, like in the example below:

<noindex>the text you do not want to get indexed</noindex>

The <noindex> tag can be inserted anywhere in the HTML of your page.


rel=?nofollow? attribute for the <a> tag

This attribute is used in the <a> tag in the following format

<a href=?url? rel=?nofollow?>the URL of your link</a>

This attribute is very similar to nofollow but works for the link specified inside the <a> tag.


rel=?canonical? attribute for the <link> tag

If there are similar pages on your site, you can give a preferable address to each of the pages (canonic address).

For example:

www.yourwebsite.com/pages?id=2

www.yourwebsite.com/blog

If the /blog is preferable, then:

<link rel=?canonical? href=?http://www.yourwebsite.com /blog?/>

The rel=?canonical? attribute is not a mandatory directive but is an offered option, please be advised, robots may ignore it.

Hope this helps and please do send us questions you want us to talk about.