robots.txt file?
 

is this mandatory if u want your site indexed by all SE's?

and, it looks like it that a robots txt file is the same as the robot meta tags.... am i right?
mean this tag: <META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">

bump 4 u

bump, im pretty sure the meta tags and the file serve the same purpose

I think it's used more to define parts you don't want crawled. The meta tags should take care if you just want to be crawled.

It's changing also, the spiders are looking for more actual page content to make ranking. I'm starting to notice that my description tags are not getting in the listing but any text from the page is including link text.

It's a changing world for search engines.

yeah thats what i think too... so no robots text file for me :upsidedow

yeah, when i search for my page on google and my galleries and free sites pop up, some dont have a description text below the link at all. But thats when i do a pure domain search, when u search for words it lists parts of the sentences form the page that contain those words.