GoFuckYourself.com - Adult Webmaster Forum

GoFuckYourself.com - Adult Webmaster Forum (https://gfy.com/index.php)
-   Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)
-   -   Do you block cuil.com bot from crawling. (https://gfy.com/showthread.php?t=856048)

sortie 09-18-2008 07:55 PM

Do you block cuil.com bot from crawling.
 
Damn!

I'm thinking to just block this search engine.
I've never seen a bot use this much resources.

They are crawling every single video page on all my sites at once.

This shit will last at least a week maybe two weeks.

I have yet to see a single hit of traffic from that engine so WTF?

Emil 10-11-2008 03:37 AM

It's crawling my sites like crazy right now, but I don't think it's a bad sign. (until the server goes down :P )


All times are GMT -7. The time now is 01:38 PM.

Powered by vBulletin® Version 3.8.8
Copyright ©2000 - 2025, vBulletin Solutions, Inc.
©2000-, AI Media Network Inc123