[pmwiki-users] big problems with robots.txt
pmwiki at 911networks.com
pmwiki at 911networks.com
Sat May 15 11:01:22 CDT 2010
Hi,
Here's my Google sitemap.xml.gz
URLs submitted: 729
Indexed URLs: 389
Crawl errors: Restricted by robots: 715
Here's the start of the list:
http://www.foto-biz.com/Biz/32gb?action=browse URL restricted by robots.txt May 11, 2010
http://www.foto-biz.com/Biz/640k?action=browse URL restricted by robots.txt May 11, 2010
http://www.foto-biz.com/Biz/A-sucker-s-born-everyday?action=browse URL restricted by robots.txt May 11, 2010
http://www.foto-biz.com/Biz/Anonymous-guilty-party?action=browse URL restricted by robots.txt May 10, 2010
There are hundreds more
I have enabled in local/config.php: $EnableRobotCloakActions = 1;
Where do these ?action=browse come from?
I have added in .htaccess:
==========BEGIN=====================================
RewriteCond %{QUERY_STRING} ^action=browse [NC]
RewriteRule ^(.*)$ http://www.foto-biz.com/$1? [R=301]
==========END=======================================
My robots.txt is:
==========BEGIN=====================================
User-agent: Googlebot
Disallow:
Disallow: *RecentChanges$
Disallow: /cookbook/
Disallow: /pub/
Disallow: /local/
Disallow: /docs/
Disallow: /scripts/
Disallow: /wikilib.d/
Disallow: /wiki.d/Site/
Disallow: /wiki.d/SiteAdmin/
Disallow: /wiki.d/Site*
Disallow: /wiki.d/Keywords/Keywords.Cancel
Disallow: /wiki.d/Keywords/Keywords.Success
Disallow: /PmWiki/
Disallow: /Category.GroupFooter
Allow: /wiki.d/Biz/
Allow: /wiki.d/Canon/
Allow: /wiki.d/Foto-biz/
Allow: /wiki.d/Foto-Biz/
Allow: /wiki.d/Keywords/Keywords.Lightroom-keywords
Allow: /wiki.d/Keywords/Keywords.Lightroom-keywords-price-increase
Allow: /wiki.d/Keywords/Keywords.Lightroom-keywords-update
Allow: /wiki.d/Lightroom/
Allow: /wiki.d/Main/
Allow: /wiki.d/Seo/
==========END=======================================
--
Thanks
More information about the pmwiki-users
mailing list