Day 24: working with integer spans - Set::IntSpan::Fast

At my previous job I had to work with time ranges. Systems would go down ("red" event) and would come up again ("green" event). And later I had to decide if a new time span was adding new outage time or was already completely contained.

I thought about using sorted start and end times (as epoch time) and compare it via binary search. But the code would have been complex and nobody would understand it. I remembered the integer span modules on CPAN (Set::IntSpan). I could represent all previous time ranges as integer spans and then simply check for intersection with the new time span. If the intersection had the same amount of elements as my new time range, it was already completely contained. Otherwise it added some new outage time.

I choose Set::IntSpan::Fast, which also exists as XS version (Set::IntSpan::Fast::XS):

#!/usr/bin/perl
use strict;
use warnings;
use Set::IntSpan::Fast;
my @outage = (
[1293000000, 1293000060],
[1293000000, 1293000030],
[1293000030, 1293000080],
);
my $prev = Set::IntSpan::Fast->new;
$prev->add_range(@{shift @outage});
foreach (@outage) {
    my $new = Set::IntSpan::Fast->new($_->[0].'-'.$_->[1]);
    my $diff = $new->diff($prev);
    next if $diff->is_empty;
    printf("Added %d seconds outage time.\n", $diff->cardinality);
    $prev->merge($diff);
}
printf("Total outage time: %s seconds.\n", $prev->cardinality);

Instead of intersection I used diff, which is empty if no new outage time is added.

If you misspell the complement method like this:

Set::IntSpan::Fast->new->compliment

you get a nice error message. Try it. :)


This is the end of my Perl Advent calendar. Thanks for all your comments. I learned some new things, which I will try in the coming weeks. I can't promise to post again this year, because I'm quite exhausted. But I have some ideas for new blog entries (update on the UUID benchmark and my reason for switching from RDBO to DBIC).

I wish you a merry Christmas and an happy New Year.

Links:

• Set::IntSpan::Fast (distribution)
• Set::IntSpan::Fast (documentation)
• Set::IntSpan::Fast::XS (distribution)
• Set::IntSpan::Fast::XS (documentation)

Day 23: IO::All can do it all :)

Look at the impressive synopsis:

use IO::All;
# Some of the many ways to read a whole file into a scalar
io('file.txt') > $contents;         # Overloaded "arrow"
$contents < io 'file.txt';          # Flipped but same operation
$io = io 'file.txt';                # Create a new IO::All object
$contents = $$io;                   # Overloaded scalar dereference
$contents = $io->all;               # A method to read everything
$contents = $io->slurp;             # Another method for that
$contents = join '', $io->getlines; # Join the separate lines
$contents = join '', map "$_\n", @$io; # Same. Overloaded array deref
$io->tie;                           # Tie the object as a handle
$contents = join '', <$io>;         # And use it in builtins
# Other file operations:
@lines = io('file.txt')->slurp;         # List context slurp
$content > io('file.txt');              # Print to a file
io('file.txt')->print($content, $more); # (ditto)
$content >> io('file.txt');             # Append to a file
io('file.txt')->append($content);       # (ditto)
$content << $io;                        # Append to a string
io('copy.txt') < io('file.txt');        # Copy a file
io('file.txt') > io('copy.txt');        # Invokes File::Copy
io('more.txt') >> io('all.txt');        # Add on to a file
...
# Miscellaneous:
@lines = io('file.txt')->chomp->slurp;  # Chomp as you slurp
$binary = io('file.bin')->binary->all;  # Read a binary file
io('a-symlink')->readlink->slurp;       # Readlink returns an object
print io('foo')->absolute->pathname;    # Print absolute path of foo
# IO::All External Plugin Methods
io("myfile") > io->("ftp://store.org"); # Upload a file using ftp
$html < io->http("www.google.com");     # Grab a web page
io('mailto:worst@enemy.net')->print($spam); # Email a "friend"

IO::All combines a lot of Perl's IO modules into one package. It is very expressive and terse at the same time. This is handy in one liners or small "throw away" scripts. Like Scalar::Andand (day 10) I would not use it in production code.

Links:

• IO::All (distribution)
• IO::All (documentation)

PS: Originally I wanted to write about Chart::Clicker, but installation problems delayed it. Now I'm glad - I found this extensive article today: Charting Weather with Chart::Clicker.

Day 22: exporting data as .xls file

Spreadsheet::ParseExcel and Spreadsheet::WriteExcel are two mature modules for reading and writing Microsoft Excel files.

Sometimes you just want to export some columns, Spreadsheet::WriteExcel::Simple is a perfect fit for that:

#!/usr/bin/perl
use strict;
use warnings;
use Spreadsheet::WriteExcel::Simple;

my $xls = Spreadsheet::WriteExcel::Simple->new;
$xls->write_bold_row([qw/Date Time Thing/]);
$xls->write_row(['12/22/10', '10:00', 'buy presents']);
$xls->write_row(['12/22/10', '16:00', 'wrap presents']);
$xls->write_row(['12/24/10', '18:00', 'give presents']);
$xls->save('todo.xls');

Besides from saving the file, this module has mainly two methods: write_bold_row and write_row. I use the first one for headings and the latter for every row of data.

Spreadsheet::WriteExcel::Simple ist just a wrapper around Spreadsheet::WriteExcel. book and sheet give you access to the underlying objects. So you can adjust the settings:

$xls->sheet->keep_leading_zeros

will allow you to write data like '01234'. (For example some German zip codes have a leading zero.)

The counterpart for reading is (you propably guessed it already) Spreadsheet::ParseExcel::Simple:

#!/usr/bin/perl
use strict;
use warnings;
use Data::Dump qw/pp/;
use Spreadsheet::ParseExcel::Simple;

my $xls = Spreadsheet::ParseExcel::Simple->read('todo.xls');
my $sheet = ($xls->sheets)[0];

my @output;
my @headlines = $sheet->next_row;
while ($sheet->has_data) {
    my @data = $sheet->next_row;
    my %item;
    foreach (@headlines) {
        $item{$_} = shift @data;
    }
    push @output, \%item;
}
print pp(\@output);

# Output:
[
  { Date => "12/22/10", Thing => "buy presents", Time => "10:00" },
  { Date => "12/22/10", Thing => "wrap presents", Time => "16:00" },
  { Date => "12/24/10", Thing => "give presents", Time => "18:00" },
]

If you want to do more complex stuff, have a look at Spreadsheet::ParseExcel and Spreadsheet::WriteExcel.

Links:

• Spreadsheet::ParseExcel (distribution)
• Spreadsheet::ParseExcel (documentation)
• Spreadsheet::ParseExcel::Simple (distribution)
• Spreadsheet::ParseExcel::Simple (documentation)
• Spreadsheet::WriteExcel (distribution)
• Spreadsheet::WriteExcel (documentation)
• Spreadsheet::WriteExcel::Simple (distribution)
• Spreadsheet::WriteExcel::Simple (documentation)

Day 21: parallel HTTP requests with Mojo::Client

Two years ago I followed the development of Mojo very closely. Plack/PSGI did not exist and I was appealed by Mojos very few dependencies. I even wrote an mod_perl2 handler for it. My interest for Mojo is gone (Plack filled that need much better), but Sebastian has some very nice modules in his distribution. One of them is Mojo::Client - an asynchronous HTTP 1.1 client:

#!/usr/bin/perl
# filename: mojo-client.pl
use strict;
use warnings;
use Mojo::Client;
my $mojo = Mojo::Client->new;
my $url  = 'http://localhost:5000/?sleep=';
my @tx   = ();
foreach (2 .. 4) {
    push @tx, scalar $mojo->build_tx(GET => $url.$_);
}
$mojo->queue(@tx);
$mojo->start;
my $slept = 0;
foreach my $tx (@tx) {
    $slept += $tx->res->json->{slept};
}
print "Slept $slept seconds.\n";

To test this, I wrote a small app.psgi:

#!/usr/bin/perl
# filename: app.psgi
use strict;
use warnings;
use Plack::Request;
my $app = sub {
    my $req = Plack::Request->new(shift);
    my $sleep = sleep($req->parameters->{sleep} || 5);
    return [
        200,
        ['Content-Type', 'application/json'],
        ['{"slept":'.$sleep.'}'],
    ];
};

Fire up starman (or any other PSGI web server capable of parallel requests) and try it:

$ time perl mojo-client.pl
Slept 9 seconds.
real 0m4.170s
user 0m0.130s
sys 0m0.030s

The numbers are the same as with the Gearman example. We "wait" 9 seconds, but it only takes a little bit over 4 seconds.

As a counter example the results with plackup (single-request server):

$ plackup &
HTTP::Server::PSGI: Accepting connections at http://0:5000/
$ time perl mojo-client.pl
Slept 9 seconds.
real 0m9.178s
user 0m0.180s
sys 0m0.000s

Mojo::Client has much more to offer than parallel processing. It also supports websocket requests.

Did you notice the $tx->res->json call? Mojo::Client has built-in JSON support. Truly Web 2.0 :)

Links:

• Mojolicious (distribution)
• Mojo::Client (documentation)

Day 20: a praise for Plack

Plack is a wonderful thing. If you don't know it, have a look at Miyagawa's slides. I gave an introduction at the 2010 German Perlworkshop, but it wasn't so good. I assumed a too much prior knowledge, so it was confusing. I won't try again in this short post, either. :)

Let me just say a few sentences: PSGI specifies - like CGI - a standard interface between web servers and Perl web applications (or web frameworks). And Plack is a toolkit for PSGI. With plackup you can start PSGI apps from the command line.

Save this short script as app.psgi and run plackup in the shell:

#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper;
$Data::Dumper::Sortkeys = 1;
my $app = sub {
    my $env = shift;
    return [
        200,
        ['Content-Type' => 'text/html'],
        ['<html><body><pre>'.Dumper($env).'</pre></body></html>'],
    ];
};

Now, point your browser to http://localhost:5000/ and you will see the PSGI environment variables.

With plackup -r you get a restarting server (very useful during development). And -e allows you to wrap middlewares around your web app:

plackup -e 'enable "Debug"' app.psgi

will give you a nice debug panel. (You have to install Plack::Middleware::Debug.)

Normally you do not use Plack directly, your web application framework does it. But for really small web apps, I use Plack::Request and Plack::Response (together with Template Toolkit) directly.

Links:

• PSGI (distribution)
• PSGI (specification)
• Plack (distribution)
• plackup (documentation)
• Plack::Request (documentation)
• Plack::Response (documentation)
• Plack::Middleware::Debug (distribution)
• Plack::Middleware::Debug (documentation)
• Plack at YAPC::NA (slides)

Day 19: memory usage with Devel::Size

Devel::Size helps you when you want to know how much memory a particular variable uses. It has two functions: size and total_size. For scalars size is enough, for array or hash references, total_size calculates the memory usage of the complete structure.

On my system (Linux 64bit, Perl 5.12), an integer takes at least 24 Bytes:

perl -MDevel::Size=size -e 'print size(1)'

24

And a string starts at 56 Bytes:

perl -MDevel::Size=size -e 'print size("1")'

56

Here is a comparison of a string and two arrays:

use Devel::Size qw/total_size/;
my $scalar = '123456789';
my $flat   = ['123', '456', '789'];
my $full   = [[1, 2, 3], [4, 5, 6], [7, 8, 9]];
printf "scalar: %5d\n", total_size $scalar;
printf "flat:   %5d\n", total_size $flat;
printf "full:   %5d\n", total_size $full;

# Output:
scalar:    64
flat:     368
full:     896

My (old) German Perl blog has a better example: Array-Overhead mit Devel::Size bestimmen.

But I want to make another point here. Sometimes you need something else:

use Devel::Size qw/total_size/;
use XML::LibXML;

my $html = <<HTML;
<html><head><title>test</title></head>
<body><h1>headline</h1>
<b><i>just</i> <u>testing</u></b>
</body></html>
HTML
my $xml = XML::LibXML->new;
my $doc = $xml->parse_html_string($html);
printf "HTML length: %5d\n", length $html;
printf "total_size:  %5d\n", total_size $doc;

# Output:
HTML length:   112
total_size:     72

Oh, the parsed document (which is a DOM tree), is smaller than the HTML string? Data::Dumper shows the problem:

$VAR1 = bless( do{\(my $o = 42302096)}, 'XML::LibXML::Document' );

XML::LibXML is a XS module, the DOM tree is not stored in Perl objects, but in C. So we need to ask the system how much memory our process is consuming. I could not find a good way in Perl. The best I found was ps:

ps -o rss --no-heading -p PID

Lets try this (I leave the first part out):

my $start = get_rss();
my $doc2  = $xml->parse_html_string($html);
my $final = get_rss();
printf "real size: %d KB\n", $final - $start;

sub get_rss {
    my $rss = qx/ps -o rss --no-heading -p $$/;
    $rss =~ s/\D//g;
    return $rss;
}

This reveals a size of 8 KB. I tried this with bigger documents and it grows. :)

But is measuring RSS (resident set size) really the right figure? Please comment, if you know it. Thanks.

Links:

• Devel::Size (distribution)
• Devel::Size (documentation)
• ps (man page)

Day 18: doing things in parallel (Gearman)

Parallel processing is a complex topic (or at least it can be complex). There are a lot of choices: POE, Coro, threads, processes and the list goes on ...

For a client I had to do three expensive calculations in parallel. I tried threads and processes, but communicating the result back needed some thought. POE would be a good candidate, but I do not have practical experience with it (only a few talks at various conferences). In the end I settled for Gearman. This has the additional charme of being able to spread parallel processing between different hosts.

If you do not know Gearman, have a look at it's Wikipedia page (which is quite brief).

Gearman was originally written in Perl, but was later rewritten in C. CPAN modules for both exist. My examples are for the Perl version. Gearman::Client contains the client and worker code. Gearman::Server the gearmand server script.

Without further ado, here comes the worker script (this contains our "real work" to be executed):

#!/usr/bin/perl
# filename: worker.pl
use strict;
use warnings;
use Gearman::Worker;

my $worker = Gearman::Worker->new(job_servers => ['127.0.0.1']);
$worker->register_function(sleep => \&_sleep);
while (1) {
    $worker->work;
}

sub _sleep {
    my $job     = shift;
    my $seconds = $job->arg;
    print "sleeping\n";
    sleep($seconds);
    return $seconds;
}

And now our client, where we will do three things in parallel:

#!/usr/bin/perl
# filename: client.pl
use strict;
use warnings;
use Gearman::Client;

my $client  = Gearman::Client->new(job_servers => ['127.0.0.1']);
my $waited  = 0;
my $taskset = $client->new_task_set;
for (2..4) {
    $taskset->add_task(sleep => $_, {
        on_complete => sub {
            my $ret = shift;
            $waited += $$ret;
            print "Done.\n";
        },
    });
}
$taskset->wait;
print "Waited $waited seconds.\n";

To test this, start gearmand and at least three workers:

gearmand &
perl worker.pl &
perl worker.pl &
perl worker.pl &
time perl client.pl

It gives the following output:

time perl client.pl
sleeping
sleeping
sleeping
Done.
Done.
Done.
Waited 9 seconds.

real 0m4.062s
user 0m0.030s
sys 0m0.030s

The three sleeping lines appear simultaneously. The jobs run 2, 3 and 4 seconds each. The total run time of the script is just a little bit over 4 seconds (instead of 9 if we were doing the jobs serially).

Two of my team mates have written modules for Gearman: Dennis Schoen has written Gearman::XS, which uses the Gearman C library. Johannes Plunien has written Gearman::Driver, which is very useful if you have a lot of worker processes.

At work we use all of them in production (and the C Gearman server).

Links:

• Gearman (project homepage)
• Gearman::Client (distribution)
• Gearman::Worker (documentation)
• Gearman::Client (documentation)
• Gearman::Server (distribution)
• gearmand (documentation)
• Gearman::XS (distribution)
• Gearman::Driver (distribution)