Xaprb

I like to write tools that make hard things easy, when possible. By and large, MySQL is easy and simple. But some simple things are too hard with MySQL. I want to change that, at least for the things that matter the most to me, and which I think I know how to fix.

I will probably write a lot about this. I have already written a number of rants blog posts about the lack of instrumentation in MySQL, and that is where I’ll probably continue to put most of my energy.

To begin with, imagine this simple scenario. You are a remote DBA. Your client says “New Relic is showing periods of slow response time from the database.” You connect to MySQL at the command line and try to troubleshoot. How do you catch the problem in action, from within the database itself? The following are no good:

• It doesn’t count to see the problem two minutes later by observing the application tier, as New Relic does. That’s too late, and it’s not from within the database.
• It’s also not good enough to catch just after the slow queries finish executing, which you can get from the slow query log if you have a realtime log analyzer running; that is a) also after the fact, and b) again not possible from within the database itself.
• Watching TCP traffic or using a proxy is similarly off limits.

This is a database, a complex piece of software — comparable to an operating system in some respects. It should be possible to know that response time is spiking while the slow queries are executing slowly, before they even finish and return their results to the application. This is a simple question that should be easy to answer.

It’s not currently possible inside MySQL, because MySQL doesn’t tell you how much time queries spend executing. It’s that simple. Fortunately the fix is equally simple: measure how long queries spend executing.

We can look at the lowly iostat (or /proc/diskstats in Linux) for an example of how this is possible to solve. We need a counter that shows the sum of execution time, including currently executing queries. I saw that the Facebook patch adds counters similar to this. Search for “Query timing” on that page. I want something slightly more complex, the way Linux’s IO counters work, because they add memory. Simple is really beautiful; a counter that has memory is an amazing thing, and you can apply Little’s Law to derive a surprising amount of information from it.

I might change my opinion about some of the more complex things that are being added to MySQL; much smarter people have, and I’m probably a hold-out because I’m not that smart. But I still see the need for simple combinations of counters and timers for essential metrics, which do not need complex queries or tools to use. I have a few more things on my wishlist, which I’ll write about later.

I’ve decided to start replacing L with GL in acronyms where L supposedly stands for Linux.

I’m not a big user of acronyms, because I think they are exclusionist and they obscure, rather than revealing. (This wouldn’t matter if I wrote for people who already knew what I meant and agreed with me, but that’s a waste of time). However, LAMP is one that I’ve probably used a few times, without thinking that it is supposed to stand for Linux, Apache, MySQL, and PHP/Perl/Python. In fact, it doesn’t refer to Linux, it refers to GNU/Linux. Therefore, it should be GLAMP.

Why does this matter? I try not to say Linux, unless I’m referring to a kernel, because a kernel is not an operating system. I try to be pretty careful about saying GNU/Linux when I’m talking about an operating system. An exception is a recruiting event yesterday at the University of Virginia, where I compromised my principles because of the noise. Trying to explain myself at that decibel level was just beyond my willingness, so I said we use Linux. If the potential recruits hire on with us, they’ll get to hear me say GNU/Linux. And if they don’t, maybe they’ll attend Richard Stallman’s upcoming talk at the engineering school there on March 27th or 28th (sorry, it’s not listed online, so I can’t link to it).

And you’ll see GNU/Linux used conscientiously if you read the book I’m helping to write, too.

GNU matters. A lot. You may not think so, but if it ceased to exist, you’d find out. That applies equally even if you don’t think you are a Free Software user. You have no idea how much you rely on Free Software in your daily life. And the GNU project has been and continues to be a keystone in that arch of freedom.

Thanks to MySQL’s Brian Aker for snapping me out of my LAMP carelessness.