Quote:
Originally Posted by mineistaken
Conclusion that robots.txt should be only used when you need to disallow something? And if you do not want to disallow something there is ZERO point of using it (except that error message)?
|
The whole point of robots.txt is to exclude bots. If you don't want to exclude anything, then you don't need one. I always add one anyways, though.