GoFuckYourself.com - Adult Webmaster Forum - View Single Post - Tech Anyone following Google's latest May 2022 Core Update?

dcortez · 05-30-2022, 12:31 PM

Given all that's been discussed here thus far, I've decided to limit Googlebot's access to directories that contain bounce scripts off-site to sponsors, using robots.txt disallow.

I don't like doing this, as linking to other "authoritative sites" is a good thing for a website - healthy outbound links can be a positive thing.

Google claims it respects this, and its testing of my disallowed URLs against my robots.txt shows that Google recognizes this.

BUT...

Even though the following works:

User-agent: *
Disallow: /STFOOH/

This will NOT prevent Google's Adbots from following into the disallowed areas to their final destinations. Such Adbots must be explicitly declared as User-agent (and there are more than one).

I have decided to use .htaccess 403 for all User-agents with "adbot' in their name.

05-30-2022, 12:31 PM
dcortez DINO CORTEZ™ Industry Role: Join Date: Jun 2003 Location: Vancouver Island Posts: 2,145	Given all that's been discussed here thus far, I've decided to limit Googlebot's access to directories that contain bounce scripts off-site to sponsors, using robots.txt disallow. I don't like doing this, as linking to other "authoritative sites" is a good thing for a website - healthy outbound links can be a positive thing. Google claims it respects this, and its testing of my disallowed URLs against my robots.txt shows that Google recognizes this. BUT... Even though the following works: User-agent: * Disallow: /STFOOH/ This will NOT prevent Google's Adbots from following into the disallowed areas to their final destinations. Such Adbots must be explicitly declared as User-agent (and there are more than one). I have decided to use .htaccess 403 for all User-agents with "adbot' in their name.