Related posts:

1. Apsersa’s summary tool supports Adaptec and MegaRAID controllers
2. Aspersa, a new opensource toolkit
3. Using Aspersa to capture diagnostic data
4. MySQL Toolkit’s Show Grants tool 0.9.1 released
5. Introducing MySQL Toolkit’s Show Grants tool

I choose to organize by date, simply because most of the time I need access to notes and files about things I’m working on now or recently. If I need to find files by project or subject, there’s a search feature in my file browser, and it works really well! So date-organization is good enough for me.

Inside my home directory, I have a directory per year, and inside that, a directory per month. If I write a note today, it goes into the $HOME/etc/2010/07/03/ directory. The filename starts with today’s date. That’s the simple organizing principle behind my note system. It also lets me eventually move things off my computer into permanent storage, so I don’t have to keep backing things up forever and carrying around infinite amounts of data. I keep the last couple of years; if I need access to notes or projects from 2006, I can go pull a hard drive off the shelf and pop it into my hard drive dock (buy one of those, and you’ll never get ripped off again by external drives with their own enclosures and power supplies).

I still need a quick way to create files and place them there, or move them there after I create them. For creating files, I use Vim. There is nothing better than a plain-text editor for me. My Vim settings are such that if I begin a line with a hyphen, Vim keeps nice indentation for me, making it easy to take notes in bulleted lists with proper indentation. If you’re on a call with me and you hear typing, I’m probably taking notes into Vim.

But it’s a pain to type out the full path to the file including the year, month, and date. So I created some helper scripts and put them into my $PATH. The most important are ‘t’ and ‘c’. ‘t’ simply uses Vim to edit a file. (It also creates any required directories, based on today’s date.) So if I am on a call with Joe, I just type ‘t joe’ into a terminal, and I’m editing /home/baron/etc/2010/07/03-joe.txt.

The ‘c’ tool cats the file’s contents. If I type ‘c joe’, it executes ‘cat /home/baron/etc/2010/07/03-joe.txt’. This makes it easy to grep, copy and paste, and so on.

There are a few more tools: the ‘m’ tool moves any file into the date-based hierarchy, so if I save a PDF of an order-confirmation page, for example, I can then ‘m’ it and it goes into its proper place. And I have a few tools to list files I created today, yesterday, this week, and this month.

I have a very important convention: when I’m taking notes and something becomes my responsibility to follow up on, I type TODO in the notes. After the call ends, I can grep for TODO in the file and quickly transfer the item into the task system I described in the post linked from above. This is how I can be confident that I’m not forgetting anything I’m supposed to do: I take notes and write it out as it happens, and then review the notes afterwards.

All told, this system kind of feels too simple to be a system. Everyone else seems to use complicated online gizmos named after groceries, or whizbang apps created by 37Signals, but I’ve found none of them to meet my needs, and just went back to basics. Basic is good. Basic works. Basic lets me concentrate on what I’m doing.

As I said in my previous post, part of this is based on the GTD book, which I read through a couple of times (with a year in between) and picked the parts that made sense to me. I think it’s a useful book to read, if you’re having trouble organizing yourself. I would just caution against spending all your energy getting organized — leave a little energy for actually doing your work!

Related posts:

1. How I keep track of tasks
2. How to track what owns a MySQL connection
3. JavaScript date chooser
4. How to make file names cross-platform
5. What to do when innotop crashes

But my opinion on this topic is beside the point. I’m the creator, and I’m biased. The Maatkit Wikipedia article should be created by independent people, not the project’s founder. If you think that Maatkit belongs on Wikipedia, I encourage you to help write that article.

Related posts:

1. Where do you use Maatkit in real life?
2. Get Maatkit fast from the command line
3. Maatkit in RHEL and CentOS
4. New Maatkit release policy
5. Maatkit version 3329 released

The main point is not to ensure that I don’t forget, actually. It is to be able to put it out of my mind with confidence that I won’t lose it. I have a crowded mind, and the cleaner I can keep it, the better.

My system has three parts: my pockets, my notepad, and a directory on my computer.

In my front pocket I have a ballpoint pen. Currently it says Holiday Inn on the side. In my back pocket, I have a small piece of paper — usually about half the size of a standard letter paper, folded small enough to fit. It might be a used envelope, or a napkin, or a piece of actual notebook paper. I write down everything that matters to me. If I hear a song on the radio and I think my wife will like it, I write down some key lyrics I can search later, such as “your arms are my castle, your heart is my sky.” I write down anything I feel guilty about not doing, or neat ideas about stuff I could do, or whatever occurs to me. The goal is to write it down and trust that it’s now permanently in the system, then clear my head.

I do much the same thing with my notebook. I tend to pick these things up at conferences. I use two or three pages a week. A small size, like the size of a paperback book, is best. Legal pads are too big. One of the best pads I ever got was an InnoDB pad. I keep one page for random whatever-comes-to-me. At the beginning of each week, I collate these items; some of them I move off to the directory on my computer, others go into a single page, grouped by importance or topic as I see fit, in the notebook. The page needs to fit everything I’ll do that week. There’s no way I can do more than a page’s worth of things in a week. Typically about half the page is carried over to next week. (I just cross things off as I complete them or move them to the clean page.) This ensures that all of the important and/or urgent things are easy for me to reference, without a bunch of other stuff intruding. I also write down things I do that aren’t in my list — if I jump in and help out on a project, for example, I’ll write that down and then cross it off. This is a good record for my weekly report.

I just came back from a conference, so there are pages and pages of thoughts stimulated from conversations, people to follow up with, thank-you notes to send, and so on. A lot of this is going to be easy to take care of: I’ll just do it if it takes only a second, or move it to my computer for later followup. After I collate and organize, I tear out the old pages, feed them into my weekly report, and throw them away. They are redundant.

In my computer’s home directory, I have a directory to hold text files. Here I hold medium-term and long-term items, things that I want to do “someday” or reference material, project notes, and so on. I name each text file by topic, and there are dozens. I keep these as simple and few as possible. There’s one for music, for example. I looked up those lyrics, and then put the artist and album into todo-music.txt in my directory. Next time I decide to order a batch of CDs, I’ll refer to this list. The names of the files aren’t scientific — I just started out with what seemed right, and changed as I saw the need to; the current files have served well for a long time, so I think it’s stable and useful. I organize the lists in two sections: the top N priority items, and everything else. They are separated by a blank line. There is no need to be fancier, I find. Most things go into the everything-else category.

So in the end, my pockets and one page in the notebook are for capturing ideas as they come to me, another page is for what I decided to prioritize for the week, and the computer is the long-term spillover for things that need to get out of the notebook. This is a lot like levels of cache in a computer. I’m keeping the most important stuff in a compact way, easy to work with. And paper is definitely easier to work with than anything with an ON switch. I have no categories, sorting, tagging, hierarchies, or anything else like that. If it’s a single page the size of my hand, there’s no need.

This system was inspired by multiple attempts to use task lists on computers, personal organizers, the Getting Things Done system, the Seven Habits of Highly Effective People method, and many more. Name your favorite app or method — I’ve probably tried it or something like it. These days it’s pen, paper, and Vim. The system has been working for me for a couple of years with excellent results; I rarely to never forget anything. Although I may deprioritize it, which is effectively the same as saying I’ll never do it, I have the peace of mind that comes from knowing I have ten years or so of ideas I’ll never forget, should I ever find myself with ten years to spare and nothing to do with them.

The Getting Things Done system is very valuable in one specific way for me: capture everything and get it out of the head, to keep the head clear. I can’t overstate how important that is to me.

Next I’ll write about how I get things into the system in a way that lets me also have confidence I’m not losing track of something I’m taking on (or being asked to do).

Related posts:

1. How I keep track of notes
2. An easy way to run many tasks in parallel
3. How to track what owns a MySQL connection
4. High Performance MySQL, Second Edition: Query Performance Optimization
5. How to add a wiki homepage, sidebar, and TOC in Google Code

The most important thing is not to guess at what’s happening, but to measure instead. I have seen these problems from DNS, the binary log, failing hardware, the query cache, the table cache, the thread cache, and a variety of InnoDB edge cases. Guessing at the problem is very dangerous; you need diagnostic data. But it is often quite hard to catch a problem in action when you can only observe it in hindsight, and it happens only for a few seconds once or twice a week. This blog post is about how to detect small variations in performance, especially when it is most difficult to observe them.

Sometimes it’s actually quite easy, so let’s look at the easy cases first. Over time I have built up a collection of tricks and tools for catching a problem in action. The process of catching and diagnosing a lightning-fast performance problem looks like the following:

1. Determine the symptoms of the problem.
2. Determine how to observe the symptoms reliably and quickly.
3. Determine how to gather diagnostic data for later.
4. Set up tools or processes to do the above.
5. Sift the collected data and diagnose.

This is 95% about figuring out how to observe the problem and gathering the data, and 5% about actually diagnosing. If you don’t get the 95% right, you’ll gather too little or too much data, or you’ll capture it at the wrong time. Your job is hard enough; you won’t be successful if you simply gather gigs of data for weeks at a time. You need to be as precise as you can. Here are some examples:

1. There are normally very few connections to the server, but sometimes I start getting “Error: max_connections exceeded” or similar. Solution: observe Threads_connected from SHOW GLOBAL STATUS and react when it grows too large, or when you cannot log in to query Threads_connected.
2. There are normally very few queries running, but we have a connection pool (and thus Threads_connected is constant). During the freezes, hundreds of queries show up in SHOW PROCESSLIST. Solution: observe Threads_running from SHOW GLOBAL STATUS and react when it grows too large.

You get the point. Find a simple metric and figure out how to capture it — usually this is possible with a little bit of bash, awk, and grep. You might need to look for something specific in SHOW INNODB STATUS, for example, such as a large number of transactions in LOCK WAIT status.

But sometimes it’s much harder. What if you simply can’t observe the problem internally to MySQL? This does happen, especially when nothing changes except for response time. This was the case in the customer’s system that I discussed in the “predicting performance problems” blog post linked above. Every single metric provided by MySQL itself stayed constant during the mini-freezes. The problem is that you can’t get information on response time from within MySQL.

I ended up writing tools to help with this, of course. I’ll show the results below.

If fluctuations in response time are the problem, then the way to observe it is to measure response time. This requires some care, because you don’t want false positives, and a lot of my ideas were obviously vulnerable to false positives. I could cross them off right away. I can’t trigger on unusually large or small numbers of queries, for example, because those just happen as the workload fluctuates through the day, and random user behavior naturally introduces variations too.

I ended up writing a tool to tail the slow query log file, which I set to zero so it captured all queries with microsecond precision. Once per second, MySQL writes out the current timestamp to the log file, so when I see that marker, I know that a second’s worth of queries has passed by. I aggregate the last second’s worth of queries (count, total time, average time) and print out a line.

This in itself does not provide a good way to know when something unusual is happening, but it gives the foundation for it. I took it slightly further: I kept a sliding window of the last 60 1-second averages, and took the standard deviation of those. If the current second’s average response time deviates significantly from the average response time over the last 60 seconds, then something is wrong. “Significant” is pretty easy to measure with standard deviations, so that’s where the real magic comes in. Let’s see some samples of this.

First, here’s a bit of the slow query log chopped into 1-second segments and aggregated:

           Time        Total  Count          Avg    1-Min Avg  1-Min StDev        Sigma
100519 18:05:17     0.477078   2084     0.000229     0.000795     0.063251     0.008954
100519 18:05:18     0.264729   1823     0.000145     0.000756     0.061334     0.009960
100519 18:05:19     0.287641   1936     0.000149     0.000720     0.059481     0.009599
100519 18:05:20     0.213181   1619     0.000132     0.000691     0.058050     0.009641
100519 18:05:21     0.276063   1520     0.000182     0.000669     0.056806     0.008587
100519 18:05:22     0.289921   1963     0.000148     0.000642     0.055310     0.008936
100519 18:05:23     0.277754   1882     0.000148     0.000618     0.053983     0.008717
100519 18:05:24     0.337821   1900     0.000178     0.000598     0.052745     0.007963
100519 18:05:25     0.236592   1727     0.000137     0.000579     0.051682     0.008556
100519 18:05:26     0.257150   1488     0.000173     0.000566     0.050821     0.007727
100519 18:05:27     0.303697   1672     0.000182     0.000552     0.049908     0.007412
100519 18:05:28     0.182106   1416     0.000129     0.000539     0.049163     0.008346
100519 18:05:29     0.211202   1631     0.000129     0.000525     0.048347     0.008186

The columns mean the following:

• Time is the timestamp of this second’s stats.
• Total is the total response time within the sample, in seconds.
• Count is how many queries were in that sample.
• Avg is just the mean response time (Total / Count).
• 1-Min Avg is the one-minute moving average of response time.
• 1-Min StDev is the standard deviation of the average response times for each of the previous 60 seconds.
• Sigma is the difference between this second’s average response time and the 1-Min Avg, in standard deviations.

As you can see, most of the time the deviation between this second’s average and the last minute’s average is quite low. But when there’s a meaningful fluctuation in performance, that changes pretty clearly. Here’s a sample with a blip at 18:09:30:

           Time        Total  Count          Avg    1-Min Avg  1-Min StDev        Sigma
100519 18:09:26     0.187038   1245     0.000150     0.000159     0.006844     0.001343
100519 18:09:27     0.269272   1697     0.000159     0.000160     0.006862     0.000178
100519 18:09:28     0.329386   1901     0.000173     0.000160     0.006895     0.001865
100519 18:09:29     0.350918   2017     0.000174     0.000161     0.006929     0.001881
100519 18:09:30     0.016610     73     0.000228     0.000161     0.006943     0.009537
100519 18:09:31     0.590175   2905     0.000203     0.000162     0.007074     0.005815
100519 18:09:32     0.384193   1879     0.000204     0.000163     0.007133     0.005783
100519 18:09:33     0.345033   2044     0.000169     0.000163     0.007133     0.000815
100519 18:09:34     0.289663   1793     0.000162     0.000163     0.007148     0.000255

That was a fast one! It flew by too quickly to do much about. But it was also not a very large deviation, and could have been a false positive. In any case, I highly doubt that we would have caught anything meaningful by triggering a stats-collection process just then. Let’s keep looking.

           Time        Total  Count          Avg    1-Min Avg  1-Min StDev        Sigma
100519 18:10:05     0.209619   1578     0.000133     0.000181     0.009542     0.005044
100519 18:10:06     0.279070   1849     0.000151     0.000181     0.009546     0.003167
100519 18:10:07     1.152811   1624     0.000710     0.000189     0.010152     0.051257
100519 18:10:08     0.342763   1450     0.000236     0.000191     0.010188     0.004478
100519 18:10:09     0.200373   1452     0.000138     0.000190     0.010182     0.005155
100519 18:10:10     0.231888   1577     0.000147     0.000191     0.010190     0.004289
100519 18:10:13     0.000459     10     0.000046     0.000191     0.010241     0.014203
100519 18:10:14     0.001999      4     0.000500     0.000189     0.010139     0.030678
100519 18:10:15     0.165705    582     0.000285     0.000189     0.010151     0.009423
100519 18:10:16     7.129640   5104     0.001397     0.000251     0.024463     0.046854
100519 18:10:17     1.140011   2859     0.000399     0.000256     0.024555     0.005817
100519 18:10:18     0.325617   2240     0.000145     0.000255     0.024491     0.004460
100519 18:10:19     0.243101   1538     0.000158     0.000255     0.024510     0.003966

Another relatively short blip but a bit longer. The mean response time really didn’t deviate as much as my client was complaining about — they were showing me New Relic transaction traces with 50-second waits. Maybe I could have caught something here, but I doubt that it’d be enough to separate the signal from the noise. Still, at this point you can clearly see how sensitive this technique is. The deviation in average response varies from a few thousandths of a sigma to a few hundredths. Let’s keep looking for something more dramatic to use as a trigger:

           Time        Total  Count          Avg    1-Min Avg  1-Min StDev        Sigma
100519 18:10:57     0.352336   2038     0.000173     0.000282     0.026701     0.004092
100519 18:10:58     0.260373   1692     0.000154     0.000283     0.026725     0.004817
100519 18:10:59     1.453306   1834     0.000792     0.000294     0.027073     0.018400
100519 18:11:00     0.264517   1658     0.000160     0.000295     0.027084     0.004989
100519 18:11:01     0.093991    953     0.000099     0.000294     0.027148     0.007203
100519 18:11:02     1.119373    469     0.002387     0.000306     0.027513     0.075629
100519 18:11:03     8.609779    291     0.029587     0.000395     0.038954     0.749383
100519 18:11:04     3.474422    103     0.033732     0.000435     0.040854     0.815026
100519 18:11:05     4.095386    111     0.036895     0.000483     0.043286     0.841211
100519 18:11:06    14.951602    131     0.114134     0.000647     0.065451     1.733932
100519 18:11:07     5.954177     52     0.114503     0.000720     0.068831     1.653074
100519 18:11:08    19.979373     53     0.376969     0.000952     0.096110     3.912385
100519 18:11:09     8.056343     16     0.503521     0.001047     0.100590     4.995285
100519 18:11:10    28.035963      7     4.005138     0.001380     0.138789    28.847777
100519 18:11:11     0.017634     69     0.000256     0.001400     0.139897     0.008182
100519 18:11:12    10.516826     20     0.525841     0.001548     0.145732     3.597660
100519 18:11:13    11.889159     50     0.237783     0.001687     0.151319     1.560253
100519 18:11:14     0.032239    138     0.000234     0.001685     0.151199     0.009599
100519 18:11:15    39.607576     38     1.042305     0.002164     0.204041     5.097713
100519 18:11:16    14.577523     40     0.364438     0.002397     0.215261     1.681870
100519 18:11:17    47.602524     34     1.400074     0.003094     0.278776     5.011118
100519 18:11:18     0.016022     84     0.000191     0.003180     0.282795     0.010570

We totally hit pay dirt here. This period in the log corresponded exactly to one of the visible spikes in New Relic. There were extremely long queries in the log, and throughput dropped to the floor — for an extended time. In the far right-hand column, Sigma is in the double digits. More experience showed me that on this particular client’s workload, anything above 0.3 Sigma is a reliable indicator of a real performance problem. If that condition becomes true, then it’s time to gather diagnostic data for a while. This is resistant to false positives from things like the occasional one-off long-running query.

After building this tool — maybe 30 minutes of work or so — I can see that I could have used other metrics instead. The number of queries per second (throughput) varies, just as response time does. And I probably could go back to the database and start watching Handler_ counters, or similar things like Innodb_rows_read, with the same technique. I wasn’t able to see those things as possibilities because of the overwhelming amount of information to sift through before (and I still don’t really know that they’re going to show spikes and notches the same way, I’m just speculating; they might be really noisy and unreliable). However, focusing on response time is an accurate metric, because response time is what actually matters. Handler counters and rows-read counters are secondary effects that can lie, and there is never anything wrong with focusing on primary sources. Looking at secondary things is far too likely to present you with unreliable information, and you end up on wild goose chases that consume huge amounts of your client’s time.

The tool I wrote for this task is crude, and not formally tested, but it’s a great proof of concept. I think the next step is probably going to be something like revamping mk-loadavg (and probably renaming it!) to be able to capture load metrics and variations in a more flexible and meaningful way.

The end result on this case is at least two problems, by the way (we’re still working on it). One was DNS flakiness. The server was not configured with skip_name_resolve, and when DNS stopped working for a short period, everything stopped working. After clearing that up, many but not all of the spikes in response time went away, and permitted me to see that InnoDB is also having trouble. It is actually quite common for multiple things to be going badly on a server, which makes a disciplined approach all the more important. Trial and error is a disaster in cases like these. Peter and I wrote a brief whitepaper about our approach, by the way. You might find it helpful if you are also facing complex performance problems.

Related posts:

1. Response-time optimization in systems that are queued
2. Browser variations in RegExp.exec()
3. A metric for MySQL load average
4. How to find un-indexed queries in MySQL, without using the log
5. mk-query-digest now understands HTTP