Sunday, December 19, 2010

Day 19: memory usage with Devel::Size

Devel::Size helps you when you want to know how much memory a particular variable uses. It has two functions: size and total_size. For scalars size is enough, for array or hash references, total_size calculates the memory usage of the complete structure.

On my system (Linux 64bit, Perl 5.12), an integer takes at least 24 Bytes:
perl -MDevel::Size=size -e 'print size(1)'
24

And a string starts at 56 Bytes:
perl -MDevel::Size=size -e 'print size("1")'
56

Here is a comparison of a string and two arrays:
use Devel::Size qw/total_size/;
my $scalar = '123456789';
my $flat   = ['123', '456', '789'];
my $full   = [[1, 2, 3], [4, 5, 6], [7, 8, 9]];
printf "scalar: %5d\n", total_size $scalar;
printf "flat:   %5d\n", total_size $flat;
printf "full:   %5d\n", total_size $full;
# Output:
scalar:    64
flat:     368
full:     896

My (old) German Perl blog has a better example: Array-Overhead mit Devel::Size bestimmen.

But I want to make another point here. Sometimes you need something else:
use Devel::Size qw/total_size/;
use XML::LibXML;

my $html = <<HTML;
<html><head><title>test</title></head>
<body><h1>headline</h1>
<b><i>just</i> <u>testing</u></b>
</body></html>
HTML
my $xml = XML::LibXML->new;
my $doc = $xml->parse_html_string($html);
printf "HTML length: %5d\n", length $html;
printf "total_size:  %5d\n", total_size $doc;
# Output:
HTML length:   112
total_size:     72

Oh, the parsed document (which is a DOM tree), is smaller than the HTML string? Data::Dumper shows the problem:
$VAR1 = bless( do{\(my $o = 42302096)}, 'XML::LibXML::Document' );

XML::LibXML is a XS module, the DOM tree is not stored in Perl objects, but in C. So we need to ask the system how much memory our process is consuming. I could not find a good way in Perl. The best I found was ps:
ps -o rss --no-heading -p PID

Lets try this (I leave the first part out):
my $start = get_rss();
my $doc2  = $xml->parse_html_string($html);
my $final = get_rss();
printf "real size: %d KB\n", $final - $start;

sub get_rss {
    my $rss = qx/ps -o rss --no-heading -p $$/;
    $rss =~ s/\D//g;
    return $rss;
}
This reveals a size of 8 KB. I tried this with bigger documents and it grows. :)

But is measuring RSS (resident set size) really the right figure? Please comment, if you know it. Thanks.

Links:

1 comment:

  1. A string is even bigger in 5.13!

    $ perlbrew switch perl-5.12.2
    $ perl -MDevel::Size=size -E 'say size "1"'
    56
    $ perlbrew switch perl-5.13.7
    $ perl -MDevel::Size=size -E 'say size "1"'
    72

    ReplyDelete