Robot files and SEO
I was watching some SEO videos lately just to get an idea of certain things, but I thought of something that I can never seem to find anything on.
Do webpages that are blocked by a robots file affect SEO for the pages that are indexed?
Does a search engine scan the whole site, but then only index what the robots allows, or does it just scan that pages allowed by the robot and totally ignores the rest, for SEO ranking purposes etc.. ?
Anyone have any idea?
|