New Aspersa I/O analysis tool, diskstatsSat, Feb 5, 2011 in Databases Open Source
I’ve just committed some changes to diskstats, an I/O analysis tool in Aspersa that’s actually been in the Subversion repository for a long time, but in a barely usable fashion and with no documentation. Now it’s usable and documented.
It is basically a reimplementation of iostat in awk. Why on earth would I reinvent that wheel? Because I spend a lot of time gathering and analyzing raw data from /proc/diskstats, which is vital to really understanding what the storage subsystem is doing. The iostat tool hides important details. Seeing that detail has immediately solved many a disk performance problem and proven SAN vendors wrong, for instance. (I used to do this the old-fashioned way.) Disk performance, of course, is one of the most important things to analyze in a database server that’s struggling.
Also, iostat isn’t interactive, and I wanted an interactive, menu-driven tool to quickly slice and dice the data and drill down into what is happening with I/O. The data it accepts is in the same format as that stored by the stalk and collect tools, which is my default post-mortem toolset. And finally – and I know this might be hard to believe – I’ve been asked to fix problems many times on systems that don’t have iostat and I am not allowed to install it.
And wouldn’t you know it, as I wrote the user’s manual I found a bug, after all my ranting about how other tools show I/O stats wrong. I don’t have time to diagnose or fix the bug right now, so maybe someone else can contribute that. There is a test suite (remind me to explain sometime how I make Bash scripts highly testable) so if we find the problem and fix it, it’ll stay fixed. Contribute your fix to the bug report :-)