Profiling a process’s IO usage with ioprofile
I’ve written a tool to profile a process’s IO usage. It works by gathering lsof and strace from a process, and then figuring out how the file descriptors, function calls, and filenames are all related to each other. The manual page has examples. I’m curious to see how it works for you. Note that it might be a good idea to run this on your development or staging system before you go running it against your production MySQL server — there are rumors of strace misbehaving on some kernels.
Hi Baron,
this is a great tool! – I’d like just to add a small suggestion which seems to me will increase the value of the collected information: it’ll be great to use strace with the “-T” option – it’ll also collect the TIME spent in each system call — this will point you to the most critical files (not those which are the mostly accessed but probably well cached by the filesystem for ex., but for those which are really limiting you from I/O level)..
Thank you!
Rgds,
-Dimitri
Dimitri
8 Oct 10 at 3:07 am
Hi Dimitri,
That is what it does, by default. You can use the command-line options to switch from displaying timings, to displaying the count of calls, or the amount of data read or written with each call.
Xaprb
8 Oct 10 at 6:53 am
Hi Baron,
you’re the best! :-))
seems I was not completely waked up this morning :-))
Thanks for the tool!
Rgds,
-Dimitri
Dimitri
8 Oct 10 at 9:07 am
Thanks for the tool Baron. Definitely will become a staple for me.
It’s funny that it’s just above “all measurements are wrong” :)
Arie Kachler
8 Oct 10 at 11:09 am
Hi,
Does it use lsof periodically to see files which are created during runtime (like on-disk temporary tables)?
I should probably just look at the source :)
–Justin
Justin Swanhart
8 Oct 10 at 1:53 pm
Oh, the description of the tool says it uses only one lsof. Should have read that first.
Justin Swanhart
8 Oct 10 at 1:57 pm
It only needs one lsof. Afterwards, it catches open() calls.
Xaprb
8 Oct 10 at 6:25 pm
I was in the process of writing a tool that collected and decoded repeated ‘lsof’ output, to get a sense of write behaviors.
Using lsof and strace in combination is much more clever! I have an I/O issue I am working on right now, and looking forward to trying this.
Peter Bach
18 Oct 10 at 5:12 pm
Hi,
nice tool. I have recently published IOprofiler based on the strace as well. The idea behind that is very similar to your tool, but it is a bit heavier and thus it has more features (taking into account children processes as well, plotting, etc).
Check it out here, if you are interested.
http://code.google.com/p/ioapps/
Jiri Horky
3 Nov 10 at 5:28 am
That looks very nice. For database servers, pread and pwrite are often the most important calls to trace. Also see http://code.google.com/p/aspersa/issues/detail?id=52 for a really important functionality that I need to develop, but it looks like you have partially implemented.
Xaprb
3 Nov 10 at 7:30 am
Hi, I’ve just added support for pread and pwrite system calls :-)
Only thing that is really missing now is support for non-complete trace files (i.e. strace -p, where some open() calls are missing) – as you have it now.
Jiri Horky
2 Jan 11 at 8:16 am