[pmwiki-users] 404 on Abort?

H. Fox haganfox at users.sourceforge.net
Sun Jan 7 20:35:20 CST 2007


On 1/7/07, porneL <pornel at pornel.net> wrote:
> On Sun, 07 Jan 2007 17:57:26 -0000, H. Fox
> <haganfox at users.sourceforge.net> wrote:
>
> > I hadn't considered bots.  Is that of any concern?  If so, please
> > explain what I'm missing.
>
> I think search-engine bots are a concern here, because they will index
> error pages that are sent with status 200.

Oh.  That's a point in favor of the suggestion, then.

> Many bots can be excluded by looking for "+http://".

Thanks, that's good to know.

A robots.php script for controlling bots is bundled with PmWiki.  It's
not documented on pmwiki.org, but it includes some documentation in
the script itself.  Excerpt:

    This file provides various features to allow PmWiki to control
    what web crawlers (robots) see when they visit the site.  Of course
    it's still possible to control robots at the webserver level
    and via robots.txt, but this page provides some finer level
    of control.

You can set $RobotPattern to identify the bots you want to treat in a
special way.  It would be easy to add "+http://" to the default
pattern as a local customization.

Maybe it should be added by default.  Might be worth a PITS entry.

> > Isn't sending 400, 403, etc. status codes to *some* bots better than
> > sending them to none?
>
> It is, but sending 200 to any of them is bad already.

This is another point in favor of the suggestion, which is to make
PmWiki's behavior "less bad" in this regard, even if it does not
become perfect.  I'm not losing any sleep over it, though.  :-)

> > Which non-IE browsers have "MSIE[space][digit]" in their user agent
> > string?
>
> Opera, although it doesn't care about status of the page, so there's no
> harm done.

Maybe it's worth having a test to detect Opera so it's grouped with
the other non-IE browsers.

Oh... Maybe not.  Having reread the other thread, I realize I've
misstated something in this one.  It seems we would only be changing
the status code, not returning a different error page.

Hagan




More information about the pmwiki-users mailing list