Googlebot jerks me 100,000 times a day but Google won't show me more love - GoFuckYourself.com

rowan · 08-05-2009, 12:11 PM

It's that weird capped Google traffic issue... no matter how many pages on my site get indexed (generating more potential longtail search results), Google refers the same amount of traffic every day.

G is adding 100k+ new profile pages each day to their index but the inbound traffic remains the same. It's the same now at 5.8m indexed as it was at 500k indexed. This effectively caps my income because most of my traffic comes from G.

Does anyone have anything solid on this phenomenon?

rowan · 08-05-2009, 12:13 PM

PS: This phenomenon is also causing overloading issues because it's a database driven profile site... Googlebot is hitting me pretty much as fast as it can (around 85% of all fetches are from Googlebot), yet there seems to be no additional long term benefit from this. If the referral traffic scaled with the number of profile pages indexed then I'd just add more capacity, but it's not worth it.

d-null · 08-05-2009, 12:13 PM

I've noticed it too, there is definitely a cap system in place

off your topic a bit, but I've got a site that is pr5, up from pr4 last update, that googlebot visits all the time and lately google is sending the site zero traffic, very strange

google is a strange Beast

Phoenix · 08-05-2009, 12:15 PM

yeah i can see that and i hate it...its almost constant

MovieMaster · 08-05-2009, 12:33 PM

been there done that

rowan · 08-05-2009, 12:35 PM

Yesterday's referral counts

google - 33500
bing - 209
yahoo - 23

I'm going nowhere

gwidomains · 08-05-2009, 12:44 PM

Quote:

Originally Posted by rowan

It's that weird capped Google traffic issue... no matter how many pages on my site get indexed (generating more potential longtail search results), Google refers the same amount of traffic every day.

G is adding 100k+ new profile pages each day to their index but the inbound traffic remains the same. It's the same now at 5.8m indexed as it was at 500k indexed. This effectively caps my income because most of my traffic comes from G.

Does anyone have anything solid on this phenomenon?

From what I have read on an interesting black hat site, there is a sandbox on adult oriented content. My guess is that goog knows you have adult content and has sandboxed you into adult, given goog also can find networks of sites -- they've capped you based on your footprint given the adult nature of your network.

I'm an seo noob though.

seeandsee · 08-05-2009, 12:46 PM

Quote:

Originally Posted by rowan

It's that weird capped Google traffic issue... no matter how many pages on my site get indexed (generating more potential longtail search results), Google refers the same amount of traffic every day.

G is adding 100k+ new profile pages each day to their index but the inbound traffic remains the same. It's the same now at 5.8m indexed as it was at 500k indexed. This effectively caps my income because most of my traffic comes from G.

Does anyone have anything solid on this phenomenon?

google is god!

Agent 488 · 08-05-2009, 12:47 PM

Quote:

Originally Posted by gwidomains

From what I have read on an interesting black hat site, there is a sandbox on adult oriented content. My guess is that goog knows you have adult content and has sandboxed you into adult, given goog also can find networks of sites -- they've capped you based on your footprint given the adult nature of your network.

I'm an seo noob though.

no adult cap.

rowan · 08-05-2009, 01:04 PM

Referrals sampled on the first day of each month...

4622 01/Oct/2008
6343 01/Nov/2008
9534 01/Dec/2008
15682 01/Jan/2009
27923 01/Feb/2009
31315 01/Mar/2009
31235 01/Apr/2009
30495 01/May/2009
31308 01/Jun/2009
31259 01/Jul/2009
32537 01/Aug/2009

You can see it clearly tops out in Feb and is capped from there on.

Jet - BANNED FOR LIFE · 08-05-2009, 01:07 PM

Quote:

Originally Posted by rowan

Yesterday's referral counts

google - 33500
bing - 209
yahoo - 23

I'm going nowhere

that;s a lot of google traffic, you should not complain

LaoTzu · 08-05-2009, 01:07 PM

Quote:

Originally Posted by rowan

PS: This phenomenon is also causing overloading issues because it's a database driven profile site... Googlebot is hitting me pretty much as fast as it can (around 85% of all fetches are from Googlebot), yet there seems to be no additional long term benefit from this. If the referral traffic scaled with the number of profile pages indexed then I'd just add more capacity, but it's not worth it.

You probably know this, but you can set a crawl delay in your robots.txt file to slow down googlebot.

rowan · 08-05-2009, 01:20 PM

Quote:

Originally Posted by LaoTzu

You probably know this, but you can set a crawl delay in your robots.txt file to slow down googlebot.

From what I can tell Google now ignores crawl-delay, and requires webmasters to submit/verify/configure their site via webmaster tools.

Ronzo · 08-05-2009, 03:46 PM

You have tons more traffic than I do. About the only thing we have in common is that most traffic by far comes from Google. I've always relied heavily on using Blogger (Google's own software) for SEO. I have 15+ blogs and a few of them are 3+ years old. They've consistently delivered traffic based on page-1 results for several good keywords. About 2 weeks ago, my traffic began slowing, so I started digging to find out what was happening.

I use statcounter for tracking and I noticed that my traffic from Google was dropping based on keywords. One by one, this started happening with all my blogs. Within one day, 4 of my blogs lost 60% of their traffic and they haven't recovered since. So, I went to Google's webmaster tools and started tweaking and managing my sites and sitemaps.
I've been trying to conform as much as possible to what Google's reporting there with crawls, etc. It's not working. On Monday, I added 5 pages to one of my better blogs in hopes of getting back on track. Today, my submitted URLs via my sitemap reflect the addition of the 5 pages, but I've actually dropped by 3 the number of pages Google is now indexing. I don't know what's going on... but all of a sudden what I've been doing successfully for the past 3+ years doesn't work at all. A lot of hard and diligent work down the tubes.

In fact, on a couple of my blogs the only traffic I'm getting through Google is "image" traffic. I've ceased getting "keyword" traffic because the pages I had on Google's first results page have slipped back to the 5th, 6th or 7th and beyond results pages. My sites are slowly disappearing. And, at this point, I'm not sure what to do. I'm not doing any black hat stuff, or anything that Google's had a known problem with (webmaster guidelines). My conclusion is that Google newest algorythms no longer like its own blogging platform.

rowan · 08-05-2009, 04:14 PM

I just thought of a sticky tape and bandaids solution to the crawl-delay issue... since Googlebot appears to fetch sequentially, why not just add a delay yourself?

if (strpos($_SERVER["HTTP_USER_AGENT"], " Googlebot/") > 0) sleep(2);

Basic_man · 08-05-2009, 04:27 PM

Quote:

Originally Posted by rowan

I just thought of a sticky tape and bandaids solution to the crawl-delay issue... since Googlebot appears to fetch sequentially, why not just add a delay yourself?

if (strpos($_SERVER["HTTP_USER_AGENT"], " Googlebot/") > 0) sleep(2);

Why not setup it in the webmasters area from Google directly...?? It's free and a valid solid solution...

rowan · 08-05-2009, 04:43 PM

Quote:

Originally Posted by Basic_man

Why not setup it in the webmasters area from Google directly...?? It's free and a valid solid solution...

I don't like Google knowing too much about my sites.

By removing support for robots.txt crawl-delay (they used to support it) they're forcing people to use their proprietary webmaster tools system.

Nicky · 08-05-2009, 04:46 PM

5,8 million pages indexed and 33k hit's a day, can only be black hat lol

NoWhErE · 08-05-2009, 04:52 PM

Quote:

Originally Posted by rowan

It's that weird capped Google traffic issue... no matter how many pages on my site get indexed (generating more potential longtail search results), Google refers the same amount of traffic every day.

G is adding 100k+ new profile pages each day to their index but the inbound traffic remains the same. It's the same now at 5.8m indexed as it was at 500k indexed. This effectively caps my income because most of my traffic comes from G.

Does anyone have anything solid on this phenomenon?

How can you be generating 100k profile pages per day? What kind of site are you running?

$5 submissions · 08-05-2009, 04:54 PM

Quote:

Originally Posted by gwidomains

From what I have read on an interesting black hat site, there is a sandbox on adult oriented content.

I believe it. Same with casino sites.

rowan · 08-05-2009, 07:30 PM

Nothing blackhat about it, the profile pages are related to domains, which is why there's hundreds of millions of them...

Pleasurepays · 08-05-2009, 07:39 PM

why do you think that just because you add a page... that page is important and should be showing up in the SERP's and your traffic should be growing?

JeremyB · 08-05-2009, 07:45 PM

Quote:

Originally Posted by rowan

Nothing blackhat about it, the profile pages are related to domains, which is why there's hundreds of millions of them...

something like aboutus.org?

Let google do what it wants to do - fucking around with it could invoke the Law of Unintended Consqeuences.

Iron Fist · 08-05-2009, 09:01 PM

Google is like the micro manager... always checking up on you and giving you no love....

d-null · 08-05-2009, 11:54 PM

Quote:

Originally Posted by Ronzo

...My conclusion is that Google newest algorythms no longer like its own blogging platform.

Google's own blogging platform IS a cesspool of spam and linkschemes and piracy and other assorted garbage, generally speaking

sandman! · 08-06-2009, 01:30 AM

there is a simple reason

cant post it on a board tho

08-05-2009, 12:11 PM	#1
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Googlebot jerks me 100,000 times a day but Google won't show me more love It's that weird capped Google traffic issue... no matter how many pages on my site get indexed (generating more potential longtail search results), Google refers the same amount of traffic every day. G is adding 100k+ new profile pages each day to their index but the inbound traffic remains the same. It's the same now at 5.8m indexed as it was at 500k indexed. This effectively caps my income because most of my traffic comes from G. Does anyone have anything solid on this phenomenon?

08-05-2009, 12:13 PM	#3
d-null . . . Industry Role: Join Date: Apr 2007 Location: NY Posts: 13,724	I've noticed it too, there is definitely a cap system in place off your topic a bit, but I've got a site that is pr5, up from pr4 last update, that googlebot visits all the time and lately google is sending the site zero traffic, very strange google is a strange Beast __________________ __________________ Looking for a custom TUBE SCRIPT that supports massive traffic, load balancing, billing support, and h264 encoding? Hit up Konrad! Looking for designs for your websites or custom tubesite design? Hit up Zuzana Designs Check out the #1 WordPress SEO Plugin: CyberSEO Suite

08-05-2009, 12:15 PM	#4
Phoenix BACON BACON BACON Industry Role: Join Date: Nov 2002 Location: Poems everybody, the laddie fancies himself a poet Posts: 35,457	yeah i can see that and i hate it...its almost constant __________________ Skype Phoenixskype1 Telegram PhoenixBrad https://quantads.io

08-05-2009, 12:33 PM	#5
MovieMaster Confirmed User Join Date: Apr 2003 Posts: 897	been there done that __________________ 100% Exclusive Program Babe-Bucks.com = HotModelClub.com \| NextDoor-Models.com Payments by: CCBILL ICQ Three 02 Five 638 Five 6 ReliableServers.com - NEED BANDWIDTH?

08-05-2009, 04:46 PM	#18
Nicky Judge Jury and Executioner Industry Role: Join Date: Mar 2003 Location: Sweden Posts: 30,069	5,8 million pages indexed and 33k hit's a day, can only be black hat lol __________________ gfynicky @ gmail.com

08-05-2009, 12:35 PM	#6
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Yesterday's referral counts google - 33500 bing - 209 yahoo - 23 I'm going nowhere

08-05-2009, 01:04 PM	#10
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Referrals sampled on the first day of each month... 4622 01/Oct/2008 6343 01/Nov/2008 9534 01/Dec/2008 15682 01/Jan/2009 27923 01/Feb/2009 31315 01/Mar/2009 31235 01/Apr/2009 30495 01/May/2009 31308 01/Jun/2009 31259 01/Jul/2009 32537 01/Aug/2009 You can see it clearly tops out in Feb and is capped from there on.

08-05-2009, 03:46 PM	#14
Ronzo Confirmed User Industry Role: Join Date: Jan 2007 Location: On the Ranch. Posts: 160	You have tons more traffic than I do. About the only thing we have in common is that most traffic by far comes from Google. I've always relied heavily on using Blogger (Google's own software) for SEO. I have 15+ blogs and a few of them are 3+ years old. They've consistently delivered traffic based on page-1 results for several good keywords. About 2 weeks ago, my traffic began slowing, so I started digging to find out what was happening. I use statcounter for tracking and I noticed that my traffic from Google was dropping based on keywords. One by one, this started happening with all my blogs. Within one day, 4 of my blogs lost 60% of their traffic and they haven't recovered since. So, I went to Google's webmaster tools and started tweaking and managing my sites and sitemaps. I've been trying to conform as much as possible to what Google's reporting there with crawls, etc. It's not working. On Monday, I added 5 pages to one of my better blogs in hopes of getting back on track. Today, my submitted URLs via my sitemap reflect the addition of the 5 pages, but I've actually dropped by 3 the number of pages Google is now indexing. I don't know what's going on... but all of a sudden what I've been doing successfully for the past 3+ years doesn't work at all. A lot of hard and diligent work down the tubes. In fact, on a couple of my blogs the only traffic I'm getting through Google is "image" traffic. I've ceased getting "keyword" traffic because the pages I had on Google's first results page have slipped back to the 5th, 6th or 7th and beyond results pages. My sites are slowly disappearing. And, at this point, I'm not sure what to do. I'm not doing any black hat stuff, or anything that Google's had a known problem with (webmaster guidelines). My conclusion is that Google newest algorythms no longer like its own blogging platform.

08-05-2009, 04:14 PM	#15
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	I just thought of a sticky tape and bandaids solution to the crawl-delay issue... since Googlebot appears to fetch sequentially, why not just add a delay yourself? if (strpos($_SERVER["HTTP_USER_AGENT"], " Googlebot/") > 0) sleep(2);

08-05-2009, 07:30 PM	#21
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Nothing blackhat about it, the profile pages are related to domains, which is why there's hundreds of millions of them...

08-05-2009, 07:39 PM	#22
Pleasurepays BANNED - SUPPORTING TUBES Join Date: Aug 2002 Location: I live in a pile of boogers Posts: 11,913	why do you think that just because you add a page... that page is important and should be showing up in the SERP's and your traffic should be growing?

08-05-2009, 09:01 PM	#24
Iron Fist Too lazy to set a custom title Join Date: Dec 2006 Posts: 23,400	Google is like the micro manager... always checking up on you and giving you no love.... __________________ i like waffles

08-06-2009, 01:30 AM	#26
sandman! Icq: 14420613 Industry Role: Join Date: Mar 2001 Location: chicago Posts: 15,432	there is a simple reason cant post it on a board tho __________________ Need WebHosting ? Email me for some great deals [email protected]