robots.txt file? - GoFuckYourself.com

nico-t · 03-03-2005, 12:18 PM

is this mandatory if u want your site indexed by all SE's?

and, it looks like it that a robots txt file is the same as the robot meta tags.... am i right?
mean this tag: <META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">

fl_prn_str · 03-03-2005, 01:52 PM

bump 4 u

tungsten · 03-03-2005, 01:53 PM

bump, im pretty sure the meta tags and the file serve the same purpose

psili · 03-03-2005, 01:54 PM

I think it's used more to define parts you don't want crawled. The meta tags should take care if you just want to be crawled.

GigaAlan · 03-03-2005, 02:03 PM

It's changing also, the spiders are looking for more actual page content to make ranking. I'm starting to notice that my description tags are not getting in the listing but any text from the page is including link text.

It's a changing world for search engines.

nico-t · 03-03-2005, 06:00 PM

Quote:

Originally Posted by psili

I think it's used more to define parts you don't want crawled. The meta tags should take care if you just want to be crawled.

yeah thats what i think too... so no robots text file for me

nico-t · 03-03-2005, 06:02 PM

Quote:

Originally Posted by GigaAlan

It's changing also, the spiders are looking for more actual page content to make ranking. I'm starting to notice that my description tags are not getting in the listing but any text from the page is including link text.

It's a changing world for search engines.

yeah, when i search for my page on google and my galleries and free sites pop up, some dont have a description text below the link at all. But thats when i do a pure domain search, when u search for words it lists parts of the sentences form the page that contain those words.

03-03-2005, 12:18 PM	#1
nico-t emperor of my world Join Date: Aug 2004 Location: nethalands Posts: 29,903	robots.txt file? is this mandatory if u want your site indexed by all SE's? and, it looks like it that a robots txt file is the same as the robot meta tags.... am i right? mean this tag: <META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">

03-03-2005, 01:53 PM	#3
tungsten Too lazy to set a custom title Join Date: Mar 2004 Posts: 10,579	bump, im pretty sure the meta tags and the file serve the same purpose __________________ VOYEUR /HOMEMADE, HENTAI / CARTOON, Reality, Amateur, Shemale, Hardcore, Cuckold, Celebrity, Retro/Vintage, ect...ALL OUR SITES >> Unbelievable Ratio \| High % of Rebills \| Bi-Monthly Payments (also to E-Passporte) Ton's of EXCLUSIVE Free content & FHG's \|=> GREAT REVENUE $$$ GUARANTEED!

03-03-2005, 01:54 PM	#4
psili Confirmed User Join Date: Apr 2003 Location: Loveland, CO Posts: 5,526	I think it's used more to define parts you don't want crawled. The meta tags should take care if you just want to be crawled. __________________ Your post count means nothing.

03-03-2005, 02:03 PM	#5
GigaAlan Confirmed User Join Date: Feb 2005 Location: Los Angeles, Ca. Posts: 196	It's changing also, the spiders are looking for more actual page content to make ranking. I'm starting to notice that my description tags are not getting in the listing but any text from the page is including link text. It's a changing world for search engines. __________________ Alan Hawkins Operations Manager www.gigacash.com www.feedsnow.com ICQ#242-838-909

03-03-2005, 01:52 PM	#2
fl_prn_str Confirmed User Industry Role: Join Date: Aug 2004 Location: Tampa, FL Posts: 5,736	bump 4 u