Xaprb

Note: the bt-aggregate tool has been deprecated and replaced by the pmp tool, which can do all that and more.

A short time ago in a galaxy nearby, Domas Mituzas wrote about contention profiling with GDB stack traces. Mark Callaghan found the technique useful, and contributed an awk script (in the comments) to aggregate stack traces and identify which things are blocking most threads. I’ve used it myself a time or five. But I’ve found myself wanting it to be fancier, for various reasons. So I wrote a little utility that can aggregate and pretty-print backtraces. It can handle unresolved symbols, and aggregate by only the first N lines of the stack trace. Here’s an example of a mysqld instance that’s really, really frozen up:

bt-aggregate -4 samples/backtrace.txt | head -n12
2396 threads with the following stack trace:
        #0  0x00000035e7c0a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1  0x00000000005f2bd8 in open_table ()
        #2  0x00000000005f3fb4 in open_tables ()
        #3  0x00000000005f4247 in open_and_lock_tables_derived ()

4 threads with the following stack trace:
        #0  0x00000035e7c0a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1  0x0000000000780099 in os_event_wait_low ()
        #2  0x000000000077de42 in os_aio_simulated_handle ()
        #3  0x000000000074a261 in fil_aio_wait ()

I just adore Google Code. But the default wiki view (a list of pages, sorted by last-modified) is lacking something. Fortunately, it’s fixable. Here’s the before:

Default wiki list

And here’s the after:

Wiki with sidebar and default page

Here’s how:

1. Create a wiki page called TableOfContents, or something like that. Using normal wiki syntax, enter links and text for your table of contents. The best way to do this is to use bulleted lists to organize and outline the pages. Keep in mind that we’ll use this same text for the sidebar, so keep it brief.
2. Go to Administer/Wiki and enter that wiki page’s name in the “Wiki Sidebar” box. Save the changes.
3. Go to Administer/Tabs and enter the same page in the Wiki box. Save the changes.

Now both the wiki “homepage” and the sidebar will contain the page you created. No more ugly list-of-pages. And as you navigate through the wiki pages, the sidebar automatically expands and closes the outline to show where you are.

If you want, you can use a different homepage and sidebar, but I’ve found that it works well for me to use the same page for both. It’s a preference, that’s all.

There’s one more trick I’d like to share: you can add the text <wiki:toc /> at the top of any page to create a small table of contents for that page. There are ways to customize it — check the documentation for more options.

If you’ve been holding out for that golden opportunity, now’s a great time to get involved in Maatkit.

Until now I haven’t really made a conscious effort to open-source the decision process and get people involved; Maatkit has been largely driven by so-called “real-world needs,” as perceived through my little lens on the world (and emails from the whole Percona team telling me when something’s wrong). I guess I am likely to remain some kind of benevolent dictator, because I created Maatkit and historically I’m the main hacker. But it doesn’t have to stay that way, and the project and users will be better off if it doesn’t.

So I’ve been trying to break out of the rut of just having some little email exchange with people using the tools, and bring things onto the Maatkit mailing list for discussion and voting. Two mailing lists I’m lurking on (Drizzle and PostgreSQL hackers) have inspired me to do this. And the person who jumped into innotop maintenance set up a whole slew of mailing lists right away, which was also educational for me — it really is beneficial.

Right now if you jump on the mailing list, you can vote on topics such as breaking backwards compatibility with command-line options for the purpose of consistent and easy-to-learn options for the future.

And I have honestly gotten intolerant about the volume of email I deal with, so when someone emails me personally I usually tell them bluntly to take it to the list, so others can learn and contribute, instead of making me the single point of contention in the knowledge-sharing system, and denying others the opportunity to learn.

So — my point is, go participate, and let’s make Maatkit more Open Source, not just Free Software.