Search engines and password protected directories?
 

Search engine robots can't get to content password protected by htaccess right?

A crawler is just like any other software requesting a page, so yeah, they should get back a 401 also.

Some sites have set up their system so that googlebot gets access to protected content, so it is indexed.

if you do let googlebot in anybody with a user agent switcher is in too.

I have another question, is water good if I'm thirsty?

Will searchengines index my lawn?

Is the searchengine-bot really a spider?

yes :2 cents:

Of course, that's why you restrict access with both the user-agent and IP range.