Xaprb

Download MySQL Toolkit

I’ve just released changes to two of the tools in MySQL Toolkit. MySQL Table Checksum got some convenient functionality to help you recursively check slaves for bad replicated checksum chunks. MySQL Archiver got statistics-gathering functionality to help you optimize your archiving and purging jobs, plus a few important bug fixes.

Changes in MySQL Archiver:

• Made –time suffix optional.
• Added –statistics option to gather and print timing statistics.
• Added signal handling so mysql-archiver exits cleanly when it can.
• Changed exit status to 0 when –help is given.
• Out-of-column-order primary keys were not ascended correctly.

Changes in MySQL Table Checksum:

• Added –replcheck option to check –replicate results on slaves.
• Added –recursecheck option to do –replcheck recursively.

Download MySQL Toolkit

MySQL Toolkit distribution 620 updates documentation and test suites, includes some major bug fixes and functionality changes, and adds one new tool to the toolkit. This article is mostly a changelog, with some added notes.

Many of the tools have matured and I just needed to make the documentation top-notch, but there’s still a lot to be done on the crucial checksumming and syncing tools. Time is in short supply for me right now, though. In fact, I actually finished this release on June 22, but wasn’t able to release it till just tonight!

Documentation is now maintained online at the MySQL Toolkit website, by the way.

mysql-archiver

Version 0.9.3

Changes

• Added more hooks for plugins before and after archiving.
• Documentation.
• Made –time suffix optional.

Bugs fixed

• MySQL Archiver could crash on a lock wait timeout when –txnsize was not set

mysql-deadlock-logger

Version 1.0.2

Incompatible changes

• Changed the format of the –source and –dest options.

Changes

• Documentation.

mysql-duplicate-key-checker

Version 1.0.4

• Documentation.

mysql-find

Version 0.9.3

• Documentation.

mysql-query-profiler and mysql-profile-compact

Version 1.1.2

• Documentation

mysql-show-grants

Version 1.0.2

• Documentation.

mysql-slave-restart

Version 0.9.2. This is an initial release of a new tool. I found myself in a situation where I needed to do some complex error-skipping on a slave (its relay logs got into an infinite loop). I have written throwaway scripts to skip, restart, check, skip several times in the past, but this situation called for something more complex. Again I realized I was three-quarters of the way to a more flexible, powerful tool many people might find useful, so I went ahead and put the extra effort into it.

It ended up helping me avoid re-snapshotting a slave with a ton of data, so it was worth it.

mysql-table-checksum and mysql-checksum-filter

This version fixes some badly optimized chunking queries. As I have mentioned in the past, the chunking behavior is preliminary and subject to change. This is still true, but this release is much smarter than the previous release! I have also fleshed out some methods of doing chunking on real-valued columns (float, decimal, and even character). I don’t know when I’ll get a chance to code, test, and release that.

Even though much remains to be done, MySQL Table Checksum is still a great way to check that your slaves have the same data as the master. (In fact, it’s the only way I know of — and MySQL employees themselves recommend MySQL Table Checksum).

Version 1.1.8

Changes

• Documentation.
• Support complex host definitions.
• Added –explainhosts option to debug host definitions.
• Added –explain option.
• When exact chunking is impossible, mysql-table-checksum will use approximate.

Incompatible changes

• Added required ‘boundaries’ column to checksum table for –replicate.

Bugs fixed

• Chunking on temporal types defeated indexes.

mysql-table-sync

Version 0.9.5

• Documentation.

Brian Aker was a recent guest on the LinuxCast podcast with Don Marti. Brian has some interesting thoughts in this podcast and elsewhere on his blog, on motivations for writing Free and/or Open Source software. Here’s why I do it myself.

First an overview of the podcast, for context: the topics were storage engines, distributed version control, and motivation for open-source. (You should listen to it, if you haven’t — it’s short and Brian is a great speaker. I listened to it twice.)

I’ve been thinking for a while about why I write the MySQL Toolkit and innotop InnoDB and MySQL monitor for free. Some people have even tried to convince me to sell them. Brian’s comments gave me some things to think about.

The simplest — but incomplete — reason I is I like doing it, I have a lot of unfinished things I can’t finish fast enough to keep up with my new ideas as it is, and I don’t want to divert effort into making it a business. My feeling is that would add a lot to my list of things I need to learn and do.

But there are many more reasons, in fact.

• It helps me avoid commitment and the guilt of not meeting commitments — if I don’t get something done, that’s fine. What I do is more than nothing, and nobody should complain. And I know if I lose interest or for some other reason stop doing this, others can take it over. That’s why I made such an effort to put it on Sourceforge (and yes, it is more work to put it on Sourceforge than to do it myself. They don’t even back up your files for you).
• It builds my personal brand, helps me network, and opens doors for me. People know me through my work who wouldn’t know me otherwise, and vice versa. I get a lot of opportunities I suspect I wouldn’t have if I were trying to make a business out of these tools.
• My employer uses these tools. I build them to solve my own problems. 25% of the work is done anyway; why not release it? Releasing it also gives me the incentive to turn the tools into much more finished products, with real documentation and test suites and decent command-line behavior that conforms to expectations.
• My employer gets community improvements sometimes. Brian mentions the pervasive myth that when you open the source code for something you get a flood of improvements, feedback, and patches. As he says, this doesn’t happen. But it occasionally does, and seldom is better than never. This is probably one of the biggest reasons my employer lets me release things, under my name with my copyright, that I sometimes even work on while I’m at work. That, and we have a great company culture and my boss knows I believe deeply in Freedom, and what’s important to me is important to him too.
• I’m being of service, and that feels good. Brian is probably right that this is a fairly small factor for most people who develop Free Software, in my opinion.
• I’m learning and having fun.
• Brian comments on people who want to fill missing functions in a commercial product, and make money from that. I do provide missing functions, and that’s intentional, but it’s not to make money — it’s because I need it. Providing missing functionality is not an obvious and inevitable reason to write something, by the way. If I wanted to make a business out of some product, I could just as easily try to duplicate someone’s work but compete with them on quality or convenience. Or marketing and packaging, for that matter; we all know which very large company has made a lot of money doing that. There are many good business models.

There’s another element Brian didn’t mention: selling these tools would put me in a totally different frame of mind, one I don’t think I would enjoy. I can’t say for sure, but I think it would become a chore and I’d get burned out and resent it. Sometimes that happens anyway — but when it does I can take a break. I do spend quite a bit of free time on these things, as Brian says. Evenings, weekends, and so on. If you’re not willing to do that, I suggest you do it for a business.

Speaking of all this, just today I got an email offering me a bounty for fixing a bug in my Javascript number-formatting library. This is the first time this has ever happened, I think. If you want to support me too, how about you send me something from my wish list?