[pmwiki-devel] [pmwiki-users] Newsletter throttling

[The Editor]

On 11/1/06, Patrick R. Michaud <pmichaud at pobox.com> wrote:
> On Mon, Oct 30, 2006 at 02:55:32PM -0400, The Editor wrote:
> > Also, my current strategy is to just start a php process when the
> > newsletter is sent that runs invisibly in the background till all the
> > emails are sent, throttling it by the sleep function, and returning an
> > immediate, simple message to the screen that messages have been fed to
> > a queue so the admin can continue clicking around.
>
> Before detailing with all of the other code details, let's just
> look at the issue of throttling email, because that's critical
> to the overall design of the module.
>
> What value of $throttle were you planning to use for the sleep()
> function, and what are the mailing constraints that you have that
> are forcing the use of a throttle?

I'm very much open to suggestions on how to do this.  My current plan
is to allow the form to set a value for how many seconds the function
will sleep between emails.  I have several small lists, and may create
more for forums, classes, teams, etc.  They can go out with sleep set
to 0 or 1.  I also have a larger mailing list of about 3000 emails
(monthly newsletter).  My default setting will be a sleep of 10 or 20
seconds for this group. Or send out in clusters of 10 or 20 and sleep
for several minutes.

My mailing constraint that calls for throttling is my ISP which puts a
cap of 800 emails/hour.  I want to keep mine to at least half that for
my large list to accommodate all the other mail being sent via my
domain.

> Also, the statement that "a php process runs invisibility in
> the background until all the emails are sent" worries me --
> what happens if that process dies in the middle for some
> reason?  Do some of the email messages not get sent?
> Is there a provision for restarting the send somehow?

My original plan was to break the maillists into packets of 100
emails, save them as separate text files and then mail them out 1
packet at a time using cron.  When that was discouraged I considered a
single function with sleeps between individual emails, or every 10-20
emails, etc., until the whole batch is gone.

My plan was to include an option for sending email notifications to
the sender every 100 or so emails indicating how far in the process
the emailing has gone.  So if there was a problem, the email could be
manually resent to the remaining emails.  Another option might be to
make a copy of the maillist on a temporary PmWiki page of all the
emails, then delete them every so often as they are sent and resave
the list.  So if an email did have to be resent it could simply be
sent to the maillist containing the remaining emails.  Clunky perhaps,
but easy enough for an administrator to handle.

The problem at this point is probably not so much the coding, but
getting some recommendations on how exactly we want it to work.
Anyone have any suggestions?  Pm?  Crisses gave me some great help
with the technical side of using the register_shutdown-function so I
should be able to get something going pretty soon. The question is
what exactly do we want going?

> (A process dying in the middle is always a possibility, but
> it becomes much more likely if that process takes a long time
> to complete, such as would be the case if email sends are
> being throttled over several hours.)

By the way, some of my design goals include:

1) Being able to easily add/removes emails by simple ZAP forms, or
cutting and pasting email lists into a wiki page.
2) Being able to optionally require confirmation before allowing
subscription of an email to a list.
3) Being able to have any number of maillists.
4) Being able to simultaneously post to a maillist, and to a wiki
page, for archiving.
5) Being able to throttle the emails.
6) Being able to template the emails using wiki template pages.
7) Being able to have multiple email messages in queue (I have/will
have several interactive forums/blogs, etc.
8) Being able to automatically sort emails and remove duplicates.
9) Being able to optionally log or email sender report of progress.
10) Being able to run quietly in the background without slowing browsing.
11) Being able to run cross-platform based on php and pmwiki only.

Cheers,
Caveman