Google Fetch issue
 

Got an email from Google webmaster tools saying they could not access my robots file. I've checked the robots file and htaccess and there does not appear to be anything that would block the googlebot from accessing my site, yet when I try to fetch the robots file, or any other page of my site for that matter, I get that it is unreachable.

Any thoughts on what could be causing this, or anyone else seeing a similar issue with the googlebot?

Google IPs blocked by hoster or some antiddos protection

Thanks, that's what I was beginning to suspect as well.

Sometimes you get temporary connectivity problems and Goog reports 404 and other server errors on URIs but when you go in and check them they work. That's more likely than DDOS protection or hosting but I wouldn't rule these out!

I'd do a Fetch as Google via the Webmaster tools now and see if there's still a problem.

That was my first thought, but I've made multiple fetch requests over the past 9 hours with the same results. Also, there does not appear to be any unusual load on the system - traffic volume and load times are what I would expect.

Can you post or PM the domain and I'll do a fetch myself in my own browser with a mofified user agent (Googlebot) and from a Google IP (using Google translate)?

Network traffic is not 100% spot on... so from time to time they will not access your site. Who fucking cares?

Sure, any additional info would be great. :thumbsup It's my main site at http://qualityadultaffiliates.com

I'm fairly certain the issue is with the googlebot IP getting blocked at the server level, as it's not just the robots file, but the entire site returning unreachable or temporarily unreachable results when fetching as desktop. Fortunately though, none of the mobile fetches cause the error.

Yup your hosting company doesn't like Google Translate IP's (I'd raise a ticket with the following and welcome them to the 21st century);


Google Translate

403. That’s an error.

Your client does not have permission to get URL /translate_c?depth=1&hl=en&rurl=translate.google.co m&sl=es&tl=en&u=http://qualityadultaffiliates.com/&usg=ALkJrhhEnWWb25JBHCeCWUkOsaWv9imUWw from this server. That’s all we know.

Thank you. :thumbsup