[pmwiki-users] Trouble with .pageindex when too much _new_ data to index (+ sqlite)

Thu Jan 29 07:36:58 CST 2015

(success indexing at step 6 ; answer to Peter below)

Indeed, issue is running out of time, no memory related message. Pardon my
negligence, error message varies (see further).
About 60 mo of new data, I'm talking about a large number of pages. There
should not be any big page among it. All short.

Is there a ".pageindex,new" file created?  If not, then PmWiki must be
> terminating
> long before page indexing is starting -- perhaps when doing the search
> itself.
> One of the first things that the indexing function does is to create the
> ".pageindex,new" file.  So if that file isn't being created, then the
> problem
> is occuring *before* indexing ever gets started, not during.

No .pageindex,new is created.
This is my opinion as well. Fail happens at the very beginning of the
indexing mecanism.

0. Native settings, searching for word : Bob/ endormir :
Fatal error: Maximum execution time of 30 seconds exceeded in
D:\xampp3\htdocs\abclf\scripts\xlpage-utf-8.php on line 75

1. Settings changed to set_time_limit(60); and searching for Bob/ endormir :
Fatal error: Maximum execution time of 60 seconds exceeded in
D:\xampp3\htdocs\abclf\cookbook\sqlite.php on line 403

No .pageindex is created ; only sqlite database and .flock in wiki.d

2. RecentChanges pages are still virgin ; I edit a Bob page, do a few
changes, and save. I do have now an existing Bob/RecentChanges page. A
.pageindex (74 octets) has been created along with .lastmod empty file.

3a. set_time_limit(60); searching for link : link=Bob/RecentChanges :
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to
allocate 1792 bytes) in D:\xampp3\htdocs\abclf\pmwiki.php(456) :
runtime-created function on line 1

url was:
http://localhost/abclf/pmwiki.php?n=PmWikiFr.Recherche&fmt=PmWikiFr.Recherche%23recherche&action=search&q=link%3DBob%2FRecentChanges

3b. set_time_limit(60); searching for word : Bob/ roupiller :
Fatal error: Maximum execution time of 60 seconds exceeded in
D:\xampp3\htdocs\abclf\cookbook\sqlite.php on line 403

url was:
http://localhost/abclf/pmwiki.php?n=Citation.2000&action=search&q=bob%2F+roupiller

4. set_time_limit(60); and setting up a pagelist – althoug index is not
done – (:pagelist group=Citation order=time count=50:) gives that error :
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to
allocate 1792 bytes) in D:\xampp3\htdocs\abclf\pmwiki.php(456) :
runtime-created function on line 1

5. set_time_limit(60); and Site.Site.php script prints out an error :
Fatal error: Call to undefined function PageIndexQueueUpdate() in
D:\xampp3\htdocs\abclf\local\Site.Site.php on line 8

Nothing new in wiki.d ; looking at the filetime, .pageindex (74 o) and
.lastmod are 13:22 (when a page is saved to generate a RecentChanges page)
; .flock a 13:45 (one of the last test).

6a. set_time_limit(90); searching for link : link=Bob/RecentChanges :
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to
allocate 1792 bytes) in D:\xampp3\htdocs\abclf\pmwiki.php(456) :
runtime-created function on line 1

6b. set_time_limit(90); searching for word : Bob/ roupiller :
Success. I get resultats.
.pageindex is 1,2 mo big now.

=Peter==========================================================================

Script saved in local/Site.Reindex.php

set_time_limit is set to native setting ;
I ran Site.Reindex one time :
no fatal error
at the end, message at the top is
DEBUG: A
DEBUG: count=166588

In wiki.d, .pageindex is now 11,8 mo (and there is a 2 mo .reindex file
left).

(For information .pageindex done yesterday is 36,9 mo)

Thank you,
Gilles.

2015-01-29 14:03 GMT+01:00 Peter Bowers <pbowers at pobox.com>:

>
> On Wed, Jan 28, 2015 at 10:10 PM, ABClf <languefrancaise at gmail.com> wrote:
>
>> Is there something to do with the native search engine to avoid it
>> failing each time amount of new data is too big ?
>
>
> Try reindexing by multiple re-loads of the page Site.Reindex using the
> following as the contents of Site.Reindex.php:
>
> ===(snip)===
> <?php
>
> # NOTE: For this to work it expects wiki.d/.reindex to be deleted before
> # starting the reindex. Then simply reload this page multiple times until
> # you get the message that the reindexing process is complete
>
> #global $PageIndexTime, $WikiDir, $FarmD;
> include_once("$FarmD/scripts/stdconfig.php");
> include_once("$FarmD/scripts/pagelist.php");
>
> SDV($ReindexFile, "$WorkDir/.reindex");
> #echo "DEBUG: Attempting to delete $ReindexFile<br />\n";
> #unlink($ReindexFile);
>
> set_time_limit(120);
> $PageIndexTime = 60;
> $fp = @fopen($ReindexFile, "r");
> if (!$fp) { // no .pageindex - start from scratch
>     echo "DEBUG: A<br />\n";
>     $pagelist = $WikiDir->ls();
>     sort($pagelist);
>     file_put_contents($ReindexFile, implode("\n", $pagelist));
>     fixperms($ReindexFile);
> } else { // we are assuming .pageindex has been created in order
>     echo "DEBUG: B<br />\n";
>     $pagelist = explode("\n", file_get_contents($ReindexFile));
>     $lastpage = '';
>     $ifp = @fopen($PageIndexFile, 'r');
>     if ($ifp) {
>         while (!feof($ifp)) {
>           $line = fgets($ifp, 4096);
>           while (substr($line, -1, 1) != "\n" && !feof($ifp))
>             $line .= fgets($ifp, 4096);
>           $i = strpos($line, ':');
>           if ($i === false) continue;
>           $n = substr($line, 0, $i);
>           if ($n > $lastpage) $lastpage = $n;
>           else break;
>         }
>         fclose($ifp);
>         for ($i = 0; $i < sizeof($pagelist); $i++)
>             if ($pagelist[$i] >= $lastpage) break;
>         if ($pagelist[$i] == $lastpage)
>             $pagelist = array_slice($pagelist, $i+1);
>     }
> }
> echo "DEBUG: count=".count($pagelist)."<br />\n";
> if (!count($pagelist)) {
>     echo "Indexing complete. Deleting $ReindexFile<br />\n";
>     if (file_exists($ReindexFile)) {
>         fixperms($ReindexFile);
>         unlink($ReindexFile); // for some reason this is giving err in
> windows
>     }
> }
> PageIndexUpdate($pagelist);
> ===(snip)===
>
> I've only done a bare minimum of testing so there will probably be
> issues.  If you get it working it may be worthwhile to package it as a
> recipe for others who may want an explicit reindex.
>
> -Peter
>

-- 

---------------------------------------
| A | de la langue française
| B | http://www.languefrancaise.net
| C | languefrancaise at gmail.com
---------------------------------------
       @bobmonamour
---------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pmichaud.com/pipermail/pmwiki-users/attachments/20150129/73c24cee/attachment-0001.html>