A short time ago in a galaxy nearby, Domas Mituzas wrote about contention profiling with GDB stack traces. Mark Callaghan found the technique useful, and contributed an awk script (in the comments) to aggregate stack traces and identify which things are blocking most threads. I’ve used it myself a time or five. But I’ve found myself wanting it to be fancier, for various reasons. So I wrote a little utility that can aggregate and pretty-print backtraces. It can handle unresolved symbols, and aggregate by only the first N lines of the stack trace. Here’s an example of a mysqld instance that’s really, really frozen up:

bt-aggregate -4 samples/backtrace.txt | head -n12
2396 threads with the following stack trace:
        #0  0x00000035e7c0a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1  0x00000000005f2bd8 in open_table ()
        #2  0x00000000005f3fb4 in open_tables ()
        #3  0x00000000005f4247 in open_and_lock_tables_derived ()

4 threads with the following stack trace:
        #0  0x00000035e7c0a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1  0x0000000000780099 in os_event_wait_low ()
        #2  0x000000000077de42 in os_aio_simulated_handle ()
        #3  0x000000000074a261 in fil_aio_wait ()

Further Reading:

• A script snippet to relative-ize numbers embedded in text
• A tweak to column alignment for the mext script
• GnuCash to MySQL export script
• Using Aspersa to capture diagnostic data
• The unexpected consequences of SELinux

Default wiki list

And here’s the after:

Wiki with sidebar and default page

Here’s how:

1. Create a wiki page called TableOfContents, or something like that. Using normal wiki syntax, enter links and text for your table of contents. The best way to do this is to use bulleted lists to organize and outline the pages. Keep in mind that we’ll use this same text for the sidebar, so keep it brief.
2. Go to Administer/Wiki and enter that wiki page’s name in the “Wiki Sidebar” box. Save the changes.
3. Go to Administer/Tabs and enter the same page in the Wiki box. Save the changes.

Now both the wiki “homepage” and the sidebar will contain the page you created. No more ugly list-of-pages. And as you navigate through the wiki pages, the sidebar automatically expands and closes the outline to show where you are.

If you want, you can use a different homepage and sidebar, but I’ve found that it works well for me to use the same page for both. It’s a preference, that’s all.

There’s one more trick I’d like to share: you can add the text <wiki:toc /> at the top of any page to create a small table of contents for that page. There are ways to customize it — check the documentation for more options.

Further Reading:

• Google Code rocks my world
• Javascript formatting and parsing functions now on Google Code
• My favorite wiki is Dokuwiki
• More alternatives to openxml
• How to add paragraph spacing in Google Docs

Until now I haven’t really made a conscious effort to open-source the decision process and get people involved; Maatkit has been largely driven by so-called “real-world needs,” as perceived through my little lens on the world (and emails from the whole Percona team telling me when something’s wrong). I guess I am likely to remain some kind of benevolent dictator, because I created Maatkit and historically I’m the main hacker. But it doesn’t have to stay that way, and the project and users will be better off if it doesn’t.

So I’ve been trying to break out of the rut of just having some little email exchange with people using the tools, and bring things onto the Maatkit mailing list for discussion and voting. Two mailing lists I’m lurking on (Drizzle and PostgreSQL hackers) have inspired me to do this. And the person who jumped into innotop maintenance set up a whole slew of mailing lists right away, which was also educational for me — it really is beneficial.

Right now if you jump on the mailing list, you can vote on topics such as breaking backwards compatibility with command-line options for the purpose of consistent and easy-to-learn options for the future.

And I have honestly gotten intolerant about the volume of email I deal with, so when someone emails me personally I usually tell them bluntly to take it to the list, so others can learn and contribute, instead of making me the single point of contention in the knowledge-sharing system, and denying others the opportunity to learn.

So — my point is, go participate, and let’s make Maatkit more Open Source, not just Free Software.

Further Reading:

• Open-source database developer mailing lists
• Does MySQL really have an open-source business model?
• What does an open source sales model look like?
• MySQL: Free Software but not Open Source
• Measuring open-source success by jobs

Topics: everything and anything, including Maatkit and PostgreSQL.

Further Reading:

• MySQL Conference and Expo 2008, Day One
• MySQL Conference and Expo 2008, Day Two
• MySQL Conference and Expo 2008, Day Three
• My presentations at the 2008 MySQL Conference and Expo
• Like it or not, it is the MySQL Conference and Expo

Inside the PBXT Storage Engine

This session was, as it sounds, a look at the internals of PBXT, a transactional storage engine for MySQL that has some interesting design techniques. I had been looking forward to this session for a while, and Paul McCullagh’s nice explanations with clear diagrams were a welcome aid to understanding how PBXT works. Unlike some of the other storage engines, PBXT is being developed in full daylight, with an emphasis on community involvement and input. (Indeed, I may be contributing to it myself, in order to make its monitoring and tuning capabilities second to none).

PBXT has not only a unique design, but a clear vision for differentiating itself from other transactional storage engines. It’s not trying to clone any particular engine; Paul and friends are planning to add some capabilities that will really set it apart from other engines, including high-availability features and blob streaming.

I left this session with a much better understanding of how PBXT balances various demands to satisfy all sorts of different workload characteristics, how it writes data, how it achieves transactional durability, and so on. I think these capabilities, and its performance, can really be assessed only in the real world (of course), but in principle it sounds good. I love knowing how things work!

There were about 30 people in the talk. I wish there had been more, because I think PBXT is going to be an important part of the open ecosystem going forward. However, I feel pretty confident people will take more notice if it starts to get used in the real world. Someone had a video camera there, so you might check out the video when it’s available. Paul’s explanations are really good.

Helping InnoDB Scale on Servers with Many CPU Cores and Disks

This session was Mark Callaghan’s chance to unveil the work he and others have been doing on InnoDB’s scalability issues, which mostly revolve around mutex contention. Mark’s team has completely solved the problems on their workload and benchmarks. In fact, after the changes, InnoDB exhibited significantly better performance even than MyISAM, which began to be limited by the single mutex that synchronizes access to its key cache. (Yes, in fact MyISAM has scalability problems too).

Google’s workload for MySQL, in case you’re wondering, is pretty traditional (i.e. not web-like; more like an “enterprise” application). Heavily I/O-bound, 24/7 critical systems, and so on.

Mark also wore several community t-shirts at various points in the talk, including one of my Maatkit t-shirts. Mark said Maatkit would be perfect if only it were written in Python (Google’s preferred scripting language). Alas, Mark, it’ll stay in Perl. But thanks for the nice compliment anyway.

The room was packed full.

Scaling Heavy Concurrent Writes In Real Time

Dathan Pattishall, formerly the lead architect at Flickr, explained his techniques for scaling Flickr’s write capacity. He talked about how he’d worked to reduce primary key sizes, queued writes for batching, separated different types of data into different types of tables, and more. Dathan has never been afraid to do what he thinks is a good idea, even if it flies in the face of “best practices,” so I was happy to finally hear him talk.

By the way, Dathan pointed out that distributed locking with memcached and add() isn’t a silver bullet. It works ok until memcached evicts your lock due to the LRU policy. He uses MySQL’s built-in GET_LOCK() function for locking.

Dathan’s blog is a good source of information about his sometimes unorthodox approaches to database design.

The Power of Lucene

This was the only one of Frank (Farhan) Mashraqi’s talks I got to attend. This was pretty technical: how Lucene works, how to configure and install it, how to index documents, how to execute searches. If you were wondering how much work and complexity it would be to install and use Lucene, this talk would have been good for you to attend; I’ve never used it myself, but I’m pretty sure Frank covered everything you need to know.

Further Reading:

• MySQL Conference and Expo 2008, Day Two
• MySQL Conference and Expo 2008, Day One
• My presentations at the 2008 MySQL Conference and Expo
• Baron Schwartz on a podcast at MySQL Conference and Expo 2008
• Sessions of interest at MySQL Conference and Expo 2009