Thread: Tech Claude Code research preview - the future is now!
View Single Post
Old 03-06-2025, 09:25 AM  
CrystalPalace
Confirmed User
 
Industry Role:
Join Date: Feb 2025
Posts: 36
the claude crawler bot is very intrusive on your website if you do not have it blocked in robots.txt, it will send over 50,000 hits in a few hours

I have blocked them all now
CrystalPalace is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote