Perl: April 2008 Archives

On the other day, I just wanted to know how many memory my system is using really for its purposes. The modular debugger "mdb" has a nifty macro for it: it is called memstat - really straightforward.

This is the result:

# mdb -k
Loading modules: [ unix krtld genunix specfs dtrace cpu.AuthenticAMD.15 uppc pcplusmp ufs mpt ip hook neti sctp arp usba fcp fctl qlc lofs fcip cpc random crypto zfs logindmux ptm nfs ]
> ::memstat
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    4111973             16062   78%
Anon                       251805               983    5%
Exec and libs                6346                24    0%
Page cache                  37719               147    1%
Free (cachelist)           285302              1114    5%
Free (freelist)            547571              2138   10%

Total                     5240716             20471
Physical                  5121357             20005

Weird - a 16 GB Kernel? Yes, that is normal as we are using ZFS as a filesystem. And its cache is stored in kernel memory. You may use the famous arcstats.pl perl program from Neelakanth Nadgir to get detailed ARC statistics but to understand a little bit you also may start with the Sun::Solaris Perl-Modules shipped with Solaris 10.

For our ARC statistics we have to use Sun::Solaris::Kstat:

#!/usr/bin/perl

use Sun::Solaris::Kstat;

my $k=new Sun::Solaris::Kstat || die "No Kernel statistics module available.";

while (1)
{
  $k->update();
  my $kstats = $k -> {zfs}{0}{arcstats};
  my %khash = %$kstats;

  foreach my $key (keys %khash)
  {
    printf "%-25s = %-20s\n",$key,$khash{$key};
  }
  print "----------\n";
  sleep 5;
}

This example will print out something like this every 5 seconds:

mru_ghost_hits            = 31005
crtime                    = 134.940576581
demand_metadata_hits      = 7307803
c_min                     = 670811648
mru_hits                  = 4479479
demand_data_misses        = 1616108
hash_elements_max         = 1059239
c_max                     = 10737418240
size                      = 10737420288
prefetch_metadata_misses  = 0
hits                      = 14405090
hash_elements             = 940483
mfu_hits                  = 9925611
prefetch_data_hits        = 0
prefetch_metadata_hits    = 0
hash_collisions           = 2486320
demand_data_hits          = 7097287
hash_chains               = 280319
deleted                   = 1301979
misses                    = 2263351
demand_metadata_misses    = 647243
evict_skip                = 47
p                         = 10211474432
c                         = 10737418240
prefetch_data_misses      = 0
recycle_miss              = 595519
hash_chain_max            = 11
class                     = misc
snaptime                  = 12168.15032689
mutex_miss                = 13682
mfu_ghost_hits            = 139332

You can see fields for the size ("size"), the maximum size ("c_max") - this is set in my case to 10GB via

set zfs:zfs_arc_max=0x280000000

in /etc/system.

You see the counters for hits, misses, metadata misses, and so on. To get values per time unit, just take differences between to measures and format them - or just just use arcstats.pl, which will yield in an output like this:

# /var/home/pascal/bin/arcstat.pl
    Time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c
10:45:21   16M    2M     13    2M   13     0    0  653K    8    10G   10G
10:45:22   512   196     38   196   38     0    0    96   57    10G   10G
10:45:23   736   219     29   219   29     0    0    76   27    10G   10G
10:45:24   647   210     32   210   32     0    0    74   39    10G   10G


So - at the end - we know that 10 GB of the 16 GB kernel memory is used for the ZFS cache.

Editor note, July 16th 2008: Yes, yes, yes, you are right, there is "kstat" available as a command (/usr/bin/kstat) and you can write:

kstat zfs:0:arcstats:size

to get the actual ARC cache size.
Just take a look at the kstat  program, it is written in perl and using... Sun::Solaris::Kstat to retrieve the values... :)