Fun with DTrace and ZFS mirrors (Solaris 10)

| | Comments (3) | TrackBacks (2)

[Update Apr 11th, 2009:]

All ZFS movies on this site:

ZFS vs UFS
ZFS as a movie actor
ZFS scrub

and my Youtube Playlist:

http://www.youtube.com/view_play_list?p=3D4F9C2AD1EF1282

and my Channel:

http://www.youtube.com/pascalgienger

blocksgraph4s.0090.gif
As a little DTrace exercise, I wanted to show how often disk blocks are read and written on a ZFS volume. More, it should be possible to watch zfs' write mechanism (more or less sequential on forthcoming unused blocks).

To begin, I started with a DTrace script like this:


#!/usr/sbin/dtrace -s

#pragma D option quiet

io:::done
{
  printf("%i,%i,%i,%i,%i\n",(timestamp/1000000),args[1]->dev_minor,args[0]->b_lblkno,args[0]->b_flags & B_WRITE,args[0]->b_bcount);
}

This resulted in a file beginning like that:

4668607882,448,2405944908,256,51200
4668607882,448,2405945042,256,7680
4668607883,512,2300900368,0,36864
4668607883,512,1865201616,0,20480
4668607884,448,2405945344,256,38912
4668607884,512,2358096048,256,38912
4668607884,512,2358096184,256,17408

[.... continued ...]

The columns mean: Timestamp (in milliseconds), minor device node of device, logical block number, read/write (read=0,write=256), number of bytes read or written starting from the given logical block.

This data file was then processed with a nifty perl script (using GD and the libgd graphics library), displaying each pixel as a block range of the device. Each read requests makes the dot greener, each write makes the underlying dot more red. For 10 seconds of data, I made a frame - at the end I had an animated gif which is funny to watch. You see growing "red" areas (write) and more or less random read i/o. And: You see the spool area, where files are written and read afterwards (the middle of the picture) - this is the heavy loaded postfix spool queue.
The left side of the graphic shows device 448, the right side device 512. Both are the two parts of a zpool mirror - hence the rather identical access patterns.

View animated gif but be warned: The file is 5 MB in size and it depends on your browser whether it can handle such large animated gifs - or not.

2 TrackBacks

Listed below are links to blogs that reference this entry: Fun with DTrace and ZFS mirrors (Solaris 10).

TrackBack URL for this entry: http://southbrain.com/mt/mt-tb.cgi/10

Pascal Gienger von der Universität Konstanz hat ein paar raffinierte DTrace Skripte auf seinem Weblog veröffentlicht die die Festplattenaktivität von ZFS aufzeichnen. Um das ganze auch noch attraktiv zu visualisieren schrieb er mit Hilfe der GD Lib ... Read More

ZFS visualized from c0t0d0s0.org on March 24, 2009 2:00 PM

Constantin linked to an interesting playlist of videos at youtube: ZFS. How can you visualize a filesystem. Well ... Pascal Gienger used dtrace and some perl scripting to display usage of a block (green=read/red=write) of a filesystem in production use i Read More

3 Comments

can you share the perl script?

Great idea!

I put together a C# app to visualise the output of the dtrace script in a similar way to how you've done it. However, I did have a question:

My current setup is 4 450GB SAS drives as two mirrored pairs in a single zpool. Whilst most of my dtrace output is sensible, I'm seeing a lot of references to very low-value block numbers (e.g. 128 as opposed to 327164563) as well as certain very high-value numbers (879080270). With a default block size (128k?) it seems that the high value is more than the total number of blocks I'd expect on the device. The data for this is very localised around these.

Do you know if these are anything special? I'm currently using a 500x500 grid per device meaning that each pixel is 2048 blocks. How would I calculate the number of blocks per device?

Thanks :)

Could you post the perl script that you're using to create the movies? I'd like to make some visualizations for my own storage.

Leave a comment

July 2012

Sun Mon Tue Wed Thu Fri Sat
1 2 3 4 5 6 7
8 9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24 25 26 27 28
29 30 31        

About

This blog is owned by:

Pascal Gienger
Jägerstrasse 77
8406 Winterthur
Switzerland


Google+: Profile
YouTube Channel: pascalgienger