How to find per-process I/O statistics on Linux
Newer Linux kernels have per-process I/O accounting and you can use the iotop tool to find out what’s performing I/O, but in many cases I’m trying to find the source of an I/O problem in an older kernel. I found sort of a hack-ish way to do that today, while trying to figure out why a system was basically unresponsive.
I found a post on Stack Overflow that showed a way you can get per process I/O statistics from the kernel even in older kernels. I adapted this to my needs, and wrote a little script.
Here’s how you use it. First, get it:
wget http://maatkit.googlecode.com/svn/trunk/util/iodump
Then turn on kernel messages about I/O:
echo 1 > /proc/sys/vm/block_dump
This makes the kernel start writing messages about every I/O operation that takes place. Now all you have to do is get those messages and feed them into my script:
while true; do sleep 1; dmesg -c; done | perl iodump
Wait a little while, then cancel the script. The results should look something like the following:
root@kanga:~# while true; do sleep 1; dmesg -c; done | perl iodump ^C# Caught SIGINT. TASK PID TOTAL READ WRITE DIRTY DEVICES firefox 4450 4538 251 4287 0 sda4, sda3 kjournald 2100 551 0 551 0 sda4 firefox 28452 185 185 0 0 sda4 kjournald 782 59 0 59 0 sda3 pdflush 31 30 0 30 0 sda4, sda3 syslogd 2485 2 0 2 0 sda3 firefox 28414 2 2 0 0 sda4, sda3 firefox 28413 1 1 0 0 sda4 firefox 28410 1 1 0 0 sda4 firefox 28307 1 1 0 0 sda4 firefox 28451 1 1 0 0 sda4
I deliberately generated a bunch of I/O by deleting my Firefox history and cache.
Got any better ideas, warnings, etc? Post them in the comments.
iotop (http://guichaz.free.fr/iotop/) is not too dissimilar from this, only giving real-time usage statistics, without the need to sample for a while before reviewing.
__
23 Aug 09 at 4:42 am
is there similar way to see I/O dump or mac osx? i use osx86 on my laptop and week ago , I badly needed to see I/O write activity which was in MB/s constantly with no real activity carried by me.
anshuman gholap.
anshuman
23 Aug 09 at 6:47 am
Good tip, thanks!
It’s minor, but I think “while true; do sleep 1″ can be replaced by “watch 1″
Ludwig
23 Aug 09 at 6:58 am
Another good tool is iotop. Debian users can simply “sudo aptitude install iotop”. iotop provides real-time disk i/o stats. Even without cumulative totals, it’s still fairly useful!
Sohail Mirza
23 Aug 09 at 7:02 am
sysstat 8.1.7-1 from ubuntu contains pidstat tool
It’s useful for per-proc io statistic with -d key
This is an example.
pidstat -d 1
Linux 2.6.31-rc5-generic (vega) 08/23/2009 _i686_ (1 CPU)
04:20:05 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command
04:20:06 PM 6452 41009.71 0.00 0.00 dd
04:20:06 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command
04:20:07 PM 6452 39920.79 0.00 0.00 dd
Nickolay Ihalainen
23 Aug 09 at 8:20 am
Right, pidstat, iotop, iopp — all are very useful, but not on an old CentOS server that doesn’t have per-process IO accounting :-)
Xaprb
23 Aug 09 at 1:41 pm
I/O accounting should be part of RHEL/CentOS 5.4.
http://dag.wieers.com/blog/red-hat-backported-io-accounting-to-rhel5
Bruce Locke
23 Aug 09 at 3:09 pm
Have a look at blktrace (the rpm or deb should include blktrace, blkparse, btrace and btt). Just point it at a block device and run for a while. Very useful. Lots of good howto’s turn up on google if you care to look.
John McNulty
23 Aug 09 at 6:16 pm
[...] How to find per-process I/O statistics on Linux at Xaprb (tags: linux sysadmin performance monitoring tools process kernel tips) [...]
links for 2009-08-23 at DeStructUred Blog
23 Aug 09 at 10:03 pm
Why not write a single script to do everything required, something like:
#!/bin/sh
dmesg -c
/etc/init.d/klogd stop
echo 1 > /proc/sys/vm/block_dump
# allow 30 seconds of stats to be logged
sleep 30
dmesg -c | perl iodump
echo 0 > /proc/sys/vm/block_dump
/etc/init.d/klogd start
Tom Collins
24 Aug 09 at 12:31 am
John, I’ll look into that, thanks for the tip.
Tom, I wrote it as a separate script because some systems actually don’t have a klogd running as a daemon (and/or you might not want to disable it), and I also want to be able to log on one system and analyze on another system.
Xaprb
24 Aug 09 at 8:27 am
[...] How to find per-process I/O statistics on Linux at Xaprb (tags: linux sysadmin performance io monitoring) [...]
andy.edmonds.be › links for 2009-08-25
25 Aug 09 at 8:35 pm
[...] Schwartz shared How to find per-process I/O statistics on Linux. He writes, “Newer Linux kernels have per-process I/O accounting and you can use the iotop [...]
Log Buffer #159: a Carnival of the Vanities for DBAs | Pythian Group Blog
28 Aug 09 at 1:05 pm