[pmwiki-users] Testing and quality control (Was: Pre-announcing 2.2.0 non-beta release)

Peter Bowers pbowers at pobox.com
Thu Jan 22 10:57:08 CST 2009


> -----Original Message-----
> From: pmwiki-users-bounces at pmichaud.com [mailto:pmwiki-users-
> bounces at pmichaud.com] On Behalf Of Patrick R. Michaud
...
> I'm a very big fan of automated testing.  That said, in the very
> early days of working on PmWiki 2.0 (yes, 2.0), I was developing
> PmWiki with a testing system in place.  It didn't work out very
> well in the long run.  Perhaps this is because of the tools I had
> available for doing the testing, but after maintaining it the
> testing subsystem for many months I found it to be a much bigger
> drain on my time/resources than benefit I was receiving for it,
> so I abandoned it.
> 
> I'd welcome a way for us to have automated tests for PmWiki,
> and I think we can support it on pmwiki.org.  However, I
> warn that it's not as trivial as it sounds at first, and
> someone other than me will need to implement a prototype or
> proof-of-concept.

Perhaps if you share your approach it might get some creative juices
flowing...?  In that spirit I'll offer a thought...

===my very off-the-cuff tho'ts===

I was thinking through using wget, saving off known-good-html files, and
then looping through with various configurations on various servers and etc.
Something like this [caveat - all theoretical at this point]:

for domain in domains.d/*
do
   curdomain=$(basename $domain)
   for htmlfile in $domain/html.d/*
   do
      curhtml=$(basename $htmlfile)
      wget http://${curdomain}/${curhtml} #what options needed?
      echo -n ${curhtml}
      if diff -q ${htmlfile} ${curhtml}   #don't remember how wget names 
      then
         echo "!!!FAILED!!!"
      else
         echo "Success."
      fi
      rm ${curhtml}
   done
done

Obviously some more error checking has to be thrown in around wget and etc,
but that should give the basic idea...

===end===

Anyway, I don't know if that helps the discussion along or not -- if
something like this seemed workable and helpful I'd be happy to put together
a proof of concept on it.  Basically it would mean Petko would install the
new version on whatever test sites already held the test pages.  Then he
would start off this script and a few hours come back later and scan down
for any failed tests...

Of course the difficulty is not in coding the test script nor in running the
test script, but in choosing the appropriate test data and checking it by
hand and then saving off the known-good-copies (although this saving off can
be automated as well).  The good thing is that you don't have to be complete
to have it be useful.  If you have even 10 pages in a single configuration
that you check presumably that's probably 10 pages better testing than it
would have been otherwise...

> I also agree with Petko that I'd prefer that pmwiki.org not
> become the central testing system for all recipes.  I'll
> comment more on this in a reply to his message.

Wherever the test platform is has to be protected so that nobody could make
*any* change without authorization.  It would obviously be vital that the
wiki pages and the "correct output" files be exactly synchronized and even a
change of a deleted space would alter that.  

-Peter




More information about the pmwiki-users mailing list