blktrace helps analyze io to a drive while you run any command/script. blkparse helps put that in human readable output. Seekwatcher helps visualize the IO in a movie or picture (look at 5th and 6th links below).

NOTE: seekwatcher generates visuals(pic/movie) from output traces generated by blktrace. it uses blkparse to make sense of it all and generate human readable output it can parse for times,read/writes, and locations on disk and to generate the visuals

Good articles for checking out first on blktrace:

Good articles for checking out first on seekwatcher:

Note: blktrace needs debugfs to work (more on this below) it allows trace information to be accessed by system (which is how blktrace runs its course). blktrace is first run to generate unreadable by human trace file, then blkparse can optionally be run on the tracefile to analyze the work (if you know how to read it – articles above describe it). then seekwatcher is run on output trace file to generate picture visualization or movie. OR you can use seekwatcher to run the blktrace program for you and then it will movie the picture/movie.

Note: when I used the word shell below that refers to the putty or xterm or teraterm etc.. terminal that you have open

PREREQS

########

FOR THE MOVIES TO WORK:

or:

NOTE: I used mencoder

CHECK PYTHON VERSION

=====================

Note: im sure this will work with other pythons

Python 2.7.3

OTHER INFO ON MY PYTHON:

Python 2.7.3 (default, Jan  2 2013, 16:53:07)

[GCC 4.7.2] on linux2

GCC VERSION

============

Note: im sure this will work with other gccs

gcc (Debian 4.7.2-5) 4.7.2

OTHER RELATED PROGRAMS I HAD INSTALLED (if curious)

===================================================

I noticed that seekwatcher was hard to get working with all of its python dependencies (im just a noob), so if your having troubles maybe try to match what I had installed python related. Those above are the ones that count, here is every python package I have just incase your curious:

# dpkg -l | egrep python | awk ‘{print $2}’

DEBUGFS needs to be mounted

============================

Make sure you have this debugfs mount setup on your system – check like this:

OUTPUT SHOULD BE: debugfs on /sys/kernel/debug type debugfs (rw,relatime)

IF THIS OUTPUT IS MISSING:

INSTALLING

############

LATEST COMMIT all 3 should be same

https://oss.oracle.com/mercurial/mason/seekwatcher/archive/tip.tar.gz

http://oss.oracle.com/mercurial/mason/seekwatcher/archive/tip.tar.gz

https://oss.oracle.com/mercurial/mason/seekwatcher/file/b392aeaf693b/

Now seekwatcher should be installed into one your PATH directories so to run it you can just type “seekwatcher” without being in the correct directory and running “./seekwatcher” <— (because thats annoying)

NOTES ON RUNNING BLKTRACE AND SEEKWATCHER

##########################################

Can run on disk where testing just dont do tests onto same spot as saving (irroneous data might come up maybe)

Do not run from directory where debugfs is mounted (debugfs being mounted just gives the right tools for blktrace)

blktrace needs to be specified the disks/device with -d, I see examples pointing to the /dev/sda instead of the working partition. In my example below I point to the partition ex: /dev/sda5, i still get results. Either way it should work. blktrace is on a higher layer as i understand from the reading.

METHOD1: RUN A TRACE WITH SEEKWATCHER AND GET OUTPUT IMAGE/MOVIE

#################################################################

The easiest way to run seekwatcher is to make it run blktrace for you.

Seekwatcher options:

-t <tracefile> saves output blktrace output of the command ran with -p, or uses the blktrace file specified as the input trace file

-o <outputfile> the image file or movie we will save

-d <device to monitor> the harddrive we are monitoring

–movie: makes moves in .mpg format (use vlc to run them, and speed them up with vlc they look better)

-l label the traces (if doing multiple traces, each l corresponds to each t in order, 1st t corresponds to 1st l)

2 examples (What they do should be self evident based on the parameter descriptions I explained above):

2 more examples:

METHOD2: BLKTRACE followed by SEEKWATCHER

###########################################

blktrace args that I use (the common ones are -d and -o listed below for device and output file, there is more like adjust the buffer with -b or -n which might be useful on big traces, the program will warn you if your buffer is too small at the end of the trace so you can rerun your test – it will tell you if you lost any events and how many percent of events were lost – i lost 0.1% of events once so I didnt redo the test for that small lost #)

-d <device> the device that will be read with blktrace for io. note blktrace goes between userland and devices so you can put the partition or device to analyze here like -d /dev/sda5 or -d /dev/sda in the parse of the trace (with blkparse) you can see when data goes from sda5 to sda – however going from sda5 to sda or seeing from sda that it got io coming from sda5 is irellevant for seekwatcher (it will still look like a write, or in the other direction a read)

-o <output file> the save file of the trace, note best not to save to the same device as reading from with -d option or else you get extra writes (if you dont mind seeing the writes on your results then its okay to leave it).

Note: blktrace – analyzes IO with very little overhead (2% CPU hit) output by itself is not human readable (some pieces are) – doing work with blktrace requires 2 shells (one for running blktrace and one for running the commands)

blkparse helps make blktrace output human readable

Note: The above seekwatcher method used -p to give the command that was traced, however we can use blktrace to generate that command and skip the -p in seekwatcher.

Note: first we need some blktrace output for seekwatcher if we are not using -p. that way we can do alot of operations in a shell rather then specify everything in a -p argument within seekwatcher

Note: if using seekwatcher with -p, dont forget -d for the device name (remember blktrace needs a -d all the time)

OUTPUT TO SCREEN (not useful with seekwatcher)  – METHOD1 ( no seekwatcher)

=======================================================================

then in another shell run commands and you will see your io on blktrace shell. not useful for seekwatcher (Because there isnt a file seekwatcher can be point at)

OUTPUT TO FILE (can be used by seekwatcher) – METHOD2a (no seekwatcher)

=====================================================================

Then do your operations in antoher shell when done go back to blktrace shell and press CONTROL-C

When done you will have a file called trace123 (it might have an extra extention added on, like trace123 will become trace123.blktrace.0)

To convert the output to humanreadable text

OR TO TAKE CARE OF EXTRA SUFFIX use *:

OR instead of running blkparse you can just run seekwatcher

BLKTRACE WITH SEEKWATCHER – METHOD2b (With seekwatcher)

=========================================================

First run blktrace (as described in above “OUTPUT TO FILE” section)

To end the capture on blktrace go to the shell where its dumping all of the info and press CONTROL-C (it will show summary info) (or read Note and you can just kill the operation, just dont kill -9 it)

Note: you can kill blktrace (aka: properly stop blktrace) with SIGTERM instead of doing the CONTROL-C button press – it will have the same effect, find out its pid with “ps aux | grep blktrace” then “kill -15 PID” or “kill PID” (put in the actual pid # inplace of PID). When you kill with SIGTERM (signal 15) or CONTROL-C blktrace will spit out some summary information (im assuming if you used kill -9 you would cut blktrace and it wouldnt show you that information, so dont kill -9 the blktrace). Note kill by default (without any parameters) uses a -15 signal (SIGTERM signal).

Note: Since we can use kill to properly stop blktrace, that means you can stick to using one shell by backgrounding blktrace with &. “blktrace -o trace123 -d /dev/sda &” (hitting enter after this will give you the pid of the backgrounded blktrace app on the terminal output so you dont have to look it up) and then do your operations you want to trace, then when done just “kill -15 PID” or “kill PID” (note the pid was shown to you, if you missed it, just type “jobs -l” to see all background processes started by you and their pids with -l argument – or of course just use “ps aux | grep blktrace” or a similar method)

3 examples (What they do should be self evident based on the parameter descriptions I explained above):

Seekwatcher creates 3 graphs by default: a plot of each sector read or written, the number of seeks per second, and the throughput per second during the run. Sometimes the plot of each sector can be difficult to read, especially when comparing two runs with very different IO patterns. seekwatcher –no-io-graph will disable this plot. Another option is to use seekwatcher –zoom start:end to specify the range on the yaxis you want included in the graph. The range is specified in MB, and using a range of 0:0 will turn off seekwatcher’s attempt at autoscaling the plot.

IF GET ERROR “bad fd error”:

=============================

ATTACHED: below is seekwatcher.py which has this fix applied to it – NOT TAKING ANY CREDIT FOR THE PROGRAMMING, Im just distributing it with the fix.

If you get this error – when running seekwatcher:

Empty blktrace run found, exiting

The problem arises because /bin/sh is linked to “dash” and they use the wrong redirections. The fix is described here:

THIS LINK:

OR THIS LINK:

You have to change the line (note this diff might be showing it backwards):

613c613

—

The error comes about that blkparse is run with “>& /dev/null” so that fd1 and fd2 go to /dev/null, but in bash its done differently (with bash3 and bash4 this method works “> /dev/null 2>&1”)

Open the file/program seekwatcher with vi (or your fav editor) and edit the code since its python you can do that:

OUTPUT: /usr/local/bin/seekwatcher

OR can open like this:

Then type “/fh.name” without quotes to find the line that says this:

And change it to say:

It just the last part of the line that you need to chang after the  -O >

NOTE: do not mess up with the trailing spaces on the left (leave the tabs/spaces on the left alone) or else the program wont work. Im not showing the spaces in my fix instructions, but the diff file output above shows them (you will see them in the code)

Now save and exit and it should work

IF GET OTHER ERROR:

===================

Use google

My example script: makes a trace of reading every file in a dir and makes a seekwatcher picture and movie

###########################################################################################################

Read the comments at the top