[pmwiki-users] Trouble with .pageindex when too much _new_ data to index (+ sqlite)

ABClf languefrancaise at gmail.com
Thu Jan 29 07:50:50 CST 2015


Bad choice with pagelist tested :
(:pagelist group=Citation order=time count=50:) is not expected to work, as
the quotes are imported without time field (import was limited to : name,
author, charset, targets, text).

Gilles.

2015-01-29 14:36 GMT+01:00 ABClf <languefrancaise at gmail.com>:

> (success indexing at step 6 ; answer to Peter below)
>
> Indeed, issue is running out of time, no memory related message. Pardon my
> negligence, error message varies (see further).
> About 60 mo of new data, I'm talking about a large number of pages. There
> should not be any big page among it. All short.
>
> Is there a ".pageindex,new" file created?  If not, then PmWiki must be
>> terminating
>> long before page indexing is starting -- perhaps when doing the search
>> itself.
>> One of the first things that the indexing function does is to create the
>> ".pageindex,new" file.  So if that file isn't being created, then the
>> problem
>> is occuring *before* indexing ever gets started, not during.
>
>
> No .pageindex,new is created.
> This is my opinion as well. Fail happens at the very beginning of the
> indexing mecanism.
>
> 0. Native settings, searching for word : Bob/ endormir :
> Fatal error: Maximum execution time of 30 seconds exceeded in
> D:\xampp3\htdocs\abclf\scripts\xlpage-utf-8.php on line 75
>
> 1. Settings changed to set_time_limit(60); and searching for Bob/ endormir
> :
> Fatal error: Maximum execution time of 60 seconds exceeded in
> D:\xampp3\htdocs\abclf\cookbook\sqlite.php on line 403
>
> No .pageindex is created ; only sqlite database and .flock in wiki.d
>
> 2. RecentChanges pages are still virgin ; I edit a Bob page, do a few
> changes, and save. I do have now an existing Bob/RecentChanges page. A
> .pageindex (74 octets) has been created along with .lastmod empty file.
>
> 3a. set_time_limit(60); searching for link : link=Bob/RecentChanges :
> Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to
> allocate 1792 bytes) in D:\xampp3\htdocs\abclf\pmwiki.php(456) :
> runtime-created function on line 1
>
> url was:
> http://localhost/abclf/pmwiki.php?n=PmWikiFr.Recherche&fmt=PmWikiFr.Recherche%23recherche&action=search&q=link%3DBob%2FRecentChanges
>
> 3b. set_time_limit(60); searching for word : Bob/ roupiller :
> Fatal error: Maximum execution time of 60 seconds exceeded in
> D:\xampp3\htdocs\abclf\cookbook\sqlite.php on line 403
>
> url was:
> http://localhost/abclf/pmwiki.php?n=Citation.2000&action=search&q=bob%2F+roupiller
>
> 4. set_time_limit(60); and setting up a pagelist – althoug index is not
> done – (:pagelist group=Citation order=time count=50:) gives that error :
> Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to
> allocate 1792 bytes) in D:\xampp3\htdocs\abclf\pmwiki.php(456) :
> runtime-created function on line 1
>
> 5. set_time_limit(60); and Site.Site.php script prints out an error :
> Fatal error: Call to undefined function PageIndexQueueUpdate() in
> D:\xampp3\htdocs\abclf\local\Site.Site.php on line 8
>
> Nothing new in wiki.d ; looking at the filetime, .pageindex (74 o) and
> .lastmod are 13:22 (when a page is saved to generate a RecentChanges page)
> ; .flock a 13:45 (one of the last test).
>
> 6a. set_time_limit(90); searching for link : link=Bob/RecentChanges :
> Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to
> allocate 1792 bytes) in D:\xampp3\htdocs\abclf\pmwiki.php(456) :
> runtime-created function on line 1
>
> 6b. set_time_limit(90); searching for word : Bob/ roupiller :
> Success. I get resultats.
> .pageindex is 1,2 mo big now.
>
>
>
>
> =Peter==========================================================================
>
>
> Script saved in local/Site.Reindex.php
>
> set_time_limit is set to native setting ;
> I ran Site.Reindex one time :
> no fatal error
> at the end, message at the top is
> DEBUG: A
> DEBUG: count=166588
>
> In wiki.d, .pageindex is now 11,8 mo (and there is a 2 mo .reindex file
> left).
>
> (For information .pageindex done yesterday is 36,9 mo)
>
>
> Thank you,
> Gilles.
>
>
> 2015-01-29 14:03 GMT+01:00 Peter Bowers <pbowers at pobox.com>:
>
>>
>> On Wed, Jan 28, 2015 at 10:10 PM, ABClf <languefrancaise at gmail.com>
>> wrote:
>>
>>> Is there something to do with the native search engine to avoid it
>>> failing each time amount of new data is too big ?
>>
>>
>> Try reindexing by multiple re-loads of the page Site.Reindex using the
>> following as the contents of Site.Reindex.php:
>>
>> ===(snip)===
>> <?php
>>
>> # NOTE: For this to work it expects wiki.d/.reindex to be deleted before
>> # starting the reindex. Then simply reload this page multiple times until
>> # you get the message that the reindexing process is complete
>>
>> #global $PageIndexTime, $WikiDir, $FarmD;
>> include_once("$FarmD/scripts/stdconfig.php");
>> include_once("$FarmD/scripts/pagelist.php");
>>
>> SDV($ReindexFile, "$WorkDir/.reindex");
>> #echo "DEBUG: Attempting to delete $ReindexFile<br />\n";
>> #unlink($ReindexFile);
>>
>> set_time_limit(120);
>> $PageIndexTime = 60;
>> $fp = @fopen($ReindexFile, "r");
>> if (!$fp) { // no .pageindex - start from scratch
>>     echo "DEBUG: A<br />\n";
>>     $pagelist = $WikiDir->ls();
>>     sort($pagelist);
>>     file_put_contents($ReindexFile, implode("\n", $pagelist));
>>     fixperms($ReindexFile);
>> } else { // we are assuming .pageindex has been created in order
>>     echo "DEBUG: B<br />\n";
>>     $pagelist = explode("\n", file_get_contents($ReindexFile));
>>     $lastpage = '';
>>     $ifp = @fopen($PageIndexFile, 'r');
>>     if ($ifp) {
>>         while (!feof($ifp)) {
>>           $line = fgets($ifp, 4096);
>>           while (substr($line, -1, 1) != "\n" && !feof($ifp))
>>             $line .= fgets($ifp, 4096);
>>           $i = strpos($line, ':');
>>           if ($i === false) continue;
>>           $n = substr($line, 0, $i);
>>           if ($n > $lastpage) $lastpage = $n;
>>           else break;
>>         }
>>         fclose($ifp);
>>         for ($i = 0; $i < sizeof($pagelist); $i++)
>>             if ($pagelist[$i] >= $lastpage) break;
>>         if ($pagelist[$i] == $lastpage)
>>             $pagelist = array_slice($pagelist, $i+1);
>>     }
>> }
>> echo "DEBUG: count=".count($pagelist)."<br />\n";
>> if (!count($pagelist)) {
>>     echo "Indexing complete. Deleting $ReindexFile<br />\n";
>>     if (file_exists($ReindexFile)) {
>>         fixperms($ReindexFile);
>>         unlink($ReindexFile); // for some reason this is giving err in
>> windows
>>     }
>> }
>> PageIndexUpdate($pagelist);
>> ===(snip)===
>>
>> I've only done a bare minimum of testing so there will probably be
>> issues.  If you get it working it may be worthwhile to package it as a
>> recipe for others who may want an explicit reindex.
>>
>> -Peter
>>
>
>
>
> --
>
> ---------------------------------------
> | A | de la langue française
> | B | http://www.languefrancaise.net
> | C | languefrancaise at gmail.com
> ---------------------------------------
>        @bobmonamour
> ---------------------------------------
>



-- 

---------------------------------------
| A | de la langue française
| B | http://www.languefrancaise.net
| C | languefrancaise at gmail.com
---------------------------------------
       @bobmonamour
---------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pmichaud.com/pipermail/pmwiki-users/attachments/20150129/5e39e484/attachment.html>


More information about the pmwiki-users mailing list