Perl Advent Calendar 2009-12-04

Let it go, let it go, let it go

by Bill 'N1VUX' Ricker

Maybe you have got an annual data load that, like Santa's annual child census update, that takes quite a while to process a slew of files from various sources, and the cpu and disk lights are not in your cube so you can't see progress. And should it fail, you'd like to restart close to where it failed.

While you could use Term::ProgressBar or some explicit state variables ¹, if we build it with Every module's iterator every(6) that returns ( (undef)x5 , 1)..., we have total control and can use the every(seconds=>999) function to also handle having regular-in-time snap backups to create recovery points for the long load sequence. Every is totally different from Time::Out, which does multiple alarms, as Every goes in an event loop.

It can be used as

$dbh->commit if every(100);
CopyToRecoverySite() if every(seconds=>3600);

Installation only has one dependency issue².

In a test

   1 use 5.010;
   2 use Every;
   3 
   4 sub sing { say @_; }
   5 while (1) {
   6     state $_iter;
   7     print ++$_iter, "\t";
   8     given (1) {
   9 
  10         when ( every(5) | every(6) ) { sing("Merry Christmas!!") }
  11         when ( every(8) )            { sing("On Dasher!") }
  12         default                      { sing("Ho, Ho, Ho!") };
  13     }
  14     sleep 1;
  15 }

we find that On Dasher! is scarcer than expected, as it does not count passes where the line is skipped when printing Merry Christmas!!. ³

So for Santa loading census updates, we'd like to see progress like

....v....x....v....x....v....x....v....x....v....x....v....x....v....x....v....x
....v....x....v....x....v....x....v....x....v....x....v....x....v....x....v....x
....v....x....v....x....v....x....v....x....v....x.
backup @ Thu Dec  3 22:22:06 2009
...v....x....v....x....v....x

with one dot every 100 files, roman v and x for 5th and 10th dots.⁴

mod4.pl

   1 use constant {
   2     N    => 100,     # number passees between progress dots
   3     COLS => 80,      # how many wide on progress
   4     HOUR => 3600,    # magic number, seconds per hour
   5 };
   6 use Every;
   7 
   8 while (1) {
   9     1;               # .... load files in input stream here ...
  10 }
  11 continue {
  12 
  13     # Progress
  14     print(
  15         (
  16             every(5)
  17             ? ( every(2) ? q(x) : q(v) ) # 5*2=10 
  18             : '.'
  19         ),
  20         ( every(COLS) ? "\n" : q() )
  21     ) if every(N);
  22 
  23     backup_incremental()
  24       and print( "\nbackup \@ ", scalar localtime, "\n" )
  25       if every( seconds => HOUR ) ;
  26 }
  27 
  28 exit;
  29 sub backup_incremental { 1 }

Thank you to David Westbrook for input today.

1. With the 'new' Perl 5.10 Xmas 2007 State variables use feature 'state'; state $_n; you no longer need closures or global variables to do this yourself.

2. Note that prerequisite Devel::Callsite 0.04 fails automated install. It has test problems. We had to do a force notest install.

3. And hence every(2)||every(3) is not what you think, since the lazy eval of || will call every(3) half as often as expected; coercion of undef to 0 makes every(2)|every(3) as one expects.

4. could use ccccDccccM for true roman counting :-) but that ties the print chars to the divisors.