[pmwiki-users] big problems with robots.txt

pmwiki at 911networks.com pmwiki at 911networks.com
Sat May 15 11:01:22 CDT 2010


Hi,

Here's my Google sitemap.xml.gz
URLs submitted: 729
Indexed URLs: 	389

Crawl errors:  Restricted by robots: 715

Here's the start of the list:

http://www.foto-biz.com/Biz/32gb?action=browse 	URL restricted by robots.txt 	May 11, 2010
http://www.foto-biz.com/Biz/640k?action=browse 	URL restricted by robots.txt 	May 11, 2010
http://www.foto-biz.com/Biz/A-sucker-s-born-everyday?action=browse 	URL restricted by robots.txt 	May 11, 2010
http://www.foto-biz.com/Biz/Anonymous-guilty-party?action=browse 	URL restricted by robots.txt 	May 10, 2010

There are hundreds more

I have enabled in local/config.php: $EnableRobotCloakActions = 1;

Where do these ?action=browse come from? 

I have added in .htaccess: 
==========BEGIN=====================================
RewriteCond %{QUERY_STRING} ^action=browse [NC]
RewriteRule ^(.*)$ http://www.foto-biz.com/$1? [R=301]
==========END=======================================


My robots.txt is:

==========BEGIN=====================================
User-agent: Googlebot
Disallow:
Disallow: *RecentChanges$
Disallow: /cookbook/
Disallow: /pub/
Disallow: /local/
Disallow: /docs/
Disallow: /scripts/
Disallow: /wikilib.d/
Disallow: /wiki.d/Site/
Disallow: /wiki.d/SiteAdmin/
Disallow: /wiki.d/Site*
Disallow: /wiki.d/Keywords/Keywords.Cancel
Disallow: /wiki.d/Keywords/Keywords.Success
Disallow: /PmWiki/
Disallow: /Category.GroupFooter
Allow: /wiki.d/Biz/
Allow: /wiki.d/Canon/
Allow: /wiki.d/Foto-biz/
Allow: /wiki.d/Foto-Biz/
Allow: /wiki.d/Keywords/Keywords.Lightroom-keywords
Allow: /wiki.d/Keywords/Keywords.Lightroom-keywords-price-increase
Allow: /wiki.d/Keywords/Keywords.Lightroom-keywords-update
Allow: /wiki.d/Lightroom/
Allow: /wiki.d/Main/
Allow: /wiki.d/Seo/
==========END=======================================

-- 
Thanks



More information about the pmwiki-users mailing list