|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
| View Poll Results: does Crawl-Delay affect seo for robots.txt | |||
| yes |
   
|
0 | 0% |
| no |
  
|
2 | 66.67% |
| no idea |
  
|
1 | 33.33% |
| Voters: 3. You may not vote on this poll | |||
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
12-05-2011, 01:43 AM
|
#1 |
|
Femcams.com
Industry Role:
Join Date: Jul 2011
Location: bjcam.com
Posts: 12,220
|
does Crawl-Delay affect seo for robots.txt
does Crawl-Delay affect seo
i.e. if i set my Crawl-Delay for robots.txt 5 seconds 10 seconds 20 seconds 60 seconds 120 seconds |
|
|
|
12-05-2011, 01:46 AM
|
#2 |
|
Confirmed User
Industry Role:
Join Date: May 2010
Posts: 5,735
|
I'm curious...what would be the benefit of setting a crawl-delay?
|
|
|
|
|
12-05-2011, 01:49 AM
|
#3 | |
|
Femcams.com
Industry Role:
Join Date: Jul 2011
Location: bjcam.com
Posts: 12,220
|
Quote:
|
|
|
|
|
|
12-05-2011, 02:10 AM
|
#4 |
|
Femcams.com
Industry Role:
Join Date: Jul 2011
Location: bjcam.com
Posts: 12,220
|
google ignores Crawl-delay anyway
|
|
|
|
|
12-05-2011, 02:21 AM
|
#5 |
|
Confirmed User
Industry Role:
Join Date: May 2010
Posts: 5,735
|
"You can change the crawl rate (the time used by Googlebot to crawl the site) for sites that are at the root level—for example, www.example.com and http://subdomain.example.com. You can't change the crawl rate for sites that are not at the root level—for example, www.example.com/folder.
Note: Crawl rate refers to the speed of Googlebot's requests during the crawl process. It doesn't have any effect on how often we crawl or how deeply we crawl your URL structure." |
|
|
|
|
12-05-2011, 03:01 AM
|
#6 | |
|
Raise Your Weapon
Industry Role:
Join Date: Jun 2003
Location: Outback Australia
Posts: 15,605
|
Quote:
|
|
|
|
|
|
12-05-2011, 04:40 AM
|
#7 | |
|
Confirmed User
Industry Role:
Join Date: Oct 2010
Location: US
Posts: 2,078
|
Quote:
__________________
I want to buy contextual links on quality blogs For both Adult and mainstream niche - Small to massive packages |
|
|
|
|
|
12-05-2011, 06:09 AM
|
#8 | |
|
Femcams.com
Industry Role:
Join Date: Jul 2011
Location: bjcam.com
Posts: 12,220
|
Quote:
problem seems to have gone know i went in to web-master tools and set all the crawl rates to the lowest they do ill monitor the uptime and see what happens or i could try iinet vps |
|
|
|
|
|
12-05-2011, 07:19 AM
|
#9 | |
|
Raise Your Weapon
Industry Role:
Join Date: Jun 2003
Location: Outback Australia
Posts: 15,605
|
Quote:
if by iinet vps you mean iiNet in AU, forget it , you can't host porn anywhere inside AU geographically. edit: it seems to me you have a configuration or other issue with setup. |
|
|
|
|
|
12-05-2011, 07:25 AM
|
#10 |
|
Junior Achiever
Industry Role:
Join Date: Nov 2004
Location: Walled Garden
Posts: 17,066
|
Find a new host.
|
|
|
|
|
12-05-2011, 08:17 AM
|
#11 | |
|
Femcams.com
Industry Role:
Join Date: Jul 2011
Location: bjcam.com
Posts: 12,220
|
Quote:
|
|
|
|
|
|
12-05-2011, 06:18 PM
|
#12 | |
|
Raise Your Weapon
Industry Role:
Join Date: Jun 2003
Location: Outback Australia
Posts: 15,605
|
Quote:
|
|
|
|
|
|
12-05-2011, 06:24 PM
|
#13 |
|
So Fucking Banned
Industry Role:
Join Date: Feb 2006
Posts: 25,214
|
Tell them to increase your rams...
|
|
|
|
|
12-05-2011, 06:28 PM
|
#14 | |
|
Femcams.com
Industry Role:
Join Date: Jul 2011
Location: bjcam.com
Posts: 12,220
|
Quote:
|
|
|
|
|
