Archive for the ‘Aspersa’ Category
I spent a little time yesterday doing some things with the “summary” tool from Aspersa. I added support for summarizing status and configuration of Adaptec and LSI MegaRAID controllers. I also figured out how to write a test suite for Bash scripts, so most major parts of the tool are fully tested now. I learned a lot more sed and awk this weekend.
There is really only one way to get status of Adaptec controllers (/usr/StorMan/arcconf), but the LSI controllers can be queried through multiple tools. I added support for MegaCli64, as long as it’s located in the usual place at /opt/MegaRAID/MegaCli/MegaCli64. I am looking for feedback and/or help on supporting other methods of getting status from the LSI controllers, such as megarc and omreport. If you can contribute sample output from these tools, please attach them as a file to a new issue report on the project’s issue tracker. (Don’t paste them as text, please — formatting and whitespace will get mangled. Tabs and spaces need to be preserved.)
I am slowly gaining insight into how best to write a similar summary tool for MySQL servers. The goals of this tool are very specific — including things like diff’able output. I’m figuring out what went wrong with Maatkit’s mk-audit tool and how to go about it differently.
I frequently encounter MySQL servers with intermittent problems that don’t happen when I’m watching the server. Gathering good diagnostic data when the problem happens is a must. Aspersa includes two utilities to make this easier.
The first is called ‘stalk’. It would be called ‘watch’ but that’s already a name of a standard Unix utility. It simply watches for a condition to happen and fires off the second utility.
This second utility does most of the work. It is called ‘collect’ and by default, it gathers stats on a number of things for 30 seconds. It names these statistics according to the time it was started, and places them into a directory for analysis.
Here’s a sample of how to use the tools. In summary: get them and make them executable, then configure them; then start a screen session and run the ‘stalk’ utility as root. Go do something else and come back later to check! A code sample follows.
$ wget http://aspersa.googlecode.com/svn/trunk/stalk $ wget http://aspersa.googlecode.com/svn/trunk/collect $ chmod +x stalk collect $ mkdir -p ~/bin $ mv stalk collect ~/bin $ vim ~/bin/stalk # Configure it $ screen -S stalking.the.server $ sudo ~/bin/stalk
Inside the ‘stalk’ tool, you’ll see a few things you can configure. By default, it tries to connect to mysqld via mysqladmin and see how many threads are connected to the server. If this increases over 100 (a sample number you should almost certainly change), or if it can’t connect to mysqld, then it fires off the ‘collect’ tool, or whatever else you configure it to execute.
The ‘collect’ tool, by default, captures a variety of things including disk usage, cpu usage, internal status from mysqld, and even oprofile (which it saves using the standard oprofile save feature; you must use opreport to get your report later). There is also a commented-out section to run GDB if you want stack traces. This is not enabled by default because that’ll freeze mysqld briefly. Usually this is OK if mysqld is already unresponsive during the problem!
Some of the utilities we were adding to Maatkit really did not belong there. Yes, this included some of the functionality in the now-retired mk-audit tool. We really learned a lesson about what it’s possible to support, design, spec, code, and test in a single tool.
I’ve moved those tools to a new project, Aspersa. Some folks are revolting and calling it Asparagus, because apparently that’s easier to say. Aspersa is the name of the common garden snail, which turns out to be a fascinating creature. It is also slow. Draw your own conclusions.
This project is more of a home for simple scripts and snippets — a simple place I can grab all the little utilities I use to make my life easy. There is a “summary” tool that largely replaces mk-audit’s functionality outside the database, and I plan to add a “mysql-summary” tool to summarize the database.
I don’t plan to make “releases.” You get the tools with wget directly from SVN trunk. There is no separate website, and little to no documentation, but there is a mailing list, and you’re invited to join and contribute.