AM I SMART OR AM I SMART ?
create file called robots.php at wp root,
Quote:
<html>
<body>
User-agent: IRLbot<br />
Crawl-delay: 3600<br />
<br />
User-agent: *<br />
Disallow: /next/<br />
<br />
# har har<br />
User-agent: *<br />
Disallow: /activate/<br />
<br />
User-agent: *<br />
Disallow: /wp-login.php<br />
<br />
User-agent: *<br />
Disallow: /wp-admin/<br />
<br />
User-agent: *<br />
Disallow: /signup/<br />
<br />
User-agent: *<br />
Disallow: /tag/<br />
<br />
# MT refugees<br />
User-agent: *<br />
Disallow: /cgi-bin/<br />
<br />
User-agent: *<br />
Disallow:<br />
<br />
Sitemap: http://<?php echo $_SERVER['HTTP_HOST']; ?>/sitemap.xml<br />
</body>
</html>
|
add this line in your .htaccess file after # uploaded files
Quote:
|
RewriteRule ^(.*/)?robots.txt robots.php [L]
|
dont forget., if they can do it., then u can TOOOOOOOOOO.
p.s. make sure to edit ur sitemap path or u can use plugin
\sitemaps-and-seo-wordpress-mu-style-1.0.4\simple-sitemaps
from wpmu dev., it will automatically create sitemaps for each blog at
and now try to access your robots.txt
http://subdomain.maindomain.com/robots.txt