Xaprb

A while ago I asked for people and/or organizations to sponsor development on Maatkit (formerly MySQL Toolkit) so I could take a week off work and improve the Table Sync tool. I asked for $2500 USD, but several companies have graciously offered to cover that and then some.

I’m very happy about this, as it will allow me to dedicate a solid week to fixing bugs and adding features. There’s a lot of demand for the tools, and there are a dozen or so bug reports unresolved for the table-sync tool, which I personally want to fix as much as anyone. So I’m very grateful for the support.

Here are the companies who have promised their financial support:

MySQL AB

MySQL AB have offered $3000 USD in support. I had an email conversation with Mårten Mickos, MySQL’s CEO, and he expressed his happiness about the project’s success, and his pleasure in supporting the project:

We have seen you operate in the community and you always have constructive and good ideas. That’s why we want to support you. Our goal with this is to stimulate innovation in the MySQL ecosystem.

I don’t know how the idea to support the project started at MySQL AB, but that quote really tells me “we get it: we have a symbiotic relationship with our community of users.” In a follow-up email, Jay Pipes wrote,

… MySQL wants to make it clear that we very much support and appreciate the work you’ve done on the toolkit. It has proven to be one of, if not the, most popular and successful open source ecosystem projects surrounding MySQL and for good reason. So, for your work and commitment to the project, a big thank you from MySQL. :)

Secondly, we would like to encourage you to be open and public about our support of you. The community team is always looking for opportunities such as the one which presented itself with your toolkit, and we want the outside community to know about our support and encouragement. Therefore, you have our blessing and encouragement to blog about the sponsorship of your development work. Please do let us know if and when you decide to blog about it. Remember also that this sponsorship is no strings attached. There is no expectation of specific work on our end.

Blue Ridge Internetworks

Blue Ridge Internetworks have offered $1000 USD in support. BRIworks, as they’re known locally, is headquartered in the town where I live, Charlottesville, Virginia. They offer networking consulting and services. Jeff Cornejo, who offered the support to me, is a friend and used to work where I used to work, and several other highly respected friends and ex-co-workers work at BRIworks too. BRIworks provides Internet service and hosting for my employer.

Percona

Percona have offered $500 USD in support. Percona does high-performance website consulting, and are perhaps best known for having some of the world’s top MySQL experts, including Peter Zaitsev and Vadim Tkachenko, two of the co-authors on High Performance MySQL, second edition.

The Rimm-Kaufman Group

Last, but absolutely not least, my employer, The Rimm-Kaufman Group, who do paid search marketing and website effectiveness consulting. They have let me spend a significant amount of time writing these tools for use on our own systems, and instead of keeping them in our own Subversion repository, allowed the code to be released as Free Software. The time I’ve spent on the tools has gone well above and beyond what we needed to get our work done. Finally, RKG has blessed my unpaid week off to work on the tools.

A big thanks is due to all of these companies and individuals, as well as other people who have contributed financially and otherwise.

Closing thoughts

I’m grateful for the sponsorship, but I think the real winners are the MySQL community, who have benefited a lot from Maatkit. It has made a lot of hard things easier and impossible things possible. If you’re one of those who benefits from Free Software, I encourage you to patronize the businesses that believe in and support it. Four fine examples are listed above! Not coincidentally, all of them are the creme de la creme in their respective fields.

Finally, a quick journalistic note: I pre-approved this post with representatives from the companies I mentioned, because I respect their right to represent themselves as they wish, but the words are mine, not theirs.

You might also like:

1. I have joined Percona
2. Proposed bounty on MySQL Table Sync features
3. Maatkit bounty begins tomorrow
4. Maatkit t-shirts are here
5. Coming soon: High Performance MySQL, Second Edition

A while ago Peter Zaitsev wrote about his wishes for mysqldump. These included multi-threaded dumps and “safe” dumps that would wait for a server to restart if it crashed, then keep dumping other tables. I’ve had sketches of this done for a while, but during this week I fleshed it out while writing about backup and recovery for our upcoming book. I had my own list of features I wanted to add:

• Support for backup sets, with the backup configuration stored in the database itself.
• Emphasis on tab-delimited dumps.
• Sane defaults, focused on ease of use and ease of recovery.
• Support for compression by default.

The resulting script is satisfactory to me. If you just run it without arguments, it connects to the server mentioned in your .my.cnf file and dumps all databases and tables, one file per table, gzipped, in parallel (at least two, but by default it detects the number of CPUs and runs that many in parallel).

baron@kanga $ mysql-parallel-dump
SET     DATABASE TABLE                      TIME STATUS THREADS
default mysql    columns_priv                  0      0       2
default mysql    db                            0      0       2
default mysql    help_category                 0      0       2
default mysql    func                          0      0       2
default mysql    help_keyword                  0      0       2
...snip...
default test     t1                            0      0       2
default test     t2                            0      0       1

You can tell it to dump elsewhere, and it’s easy to dump all tables in tab-delimited format. Here it’s reading its configuration from the database, writing to /tmp, and not backing up tables that have been dumped in the last 5 minutes:

baron@kanga $ mysql-parallel-dump --basedir /tmp --tab --sets set1 \
    --settable test.backupset --age 5m
Nothing to do for set set1
baron@kanga $ ls -lR /tmp/set1
/tmp/set1:
total 8
-rw-rw-rw- 1 baron baron   40 2007-09-30 21:43 00_master_data.sql
drwxrwxrwx 2 baron baron 4096 2007-09-30 21:43 test

/tmp/set1/test:
total 16
-rw-rw-rw- 1 baron baron 549 2007-09-30 21:43 t1.sql.gz
-rw-rw-rw- 1 baron baron  31 2007-09-30 21:43 t1.txt.gz
-rw-rw-rw- 1 baron baron 550 2007-09-30 21:43 t2.sql.gz
-rw-rw-rw- 1 baron baron  29 2007-09-30 21:43 t2.txt.gz

And as you can see, it knows I’ve dumped those tables recently and didn’t do them again. Pretty handy for scheduling and resuming backups, no? It makes it easy to keep going if something happens in the middle of the backup and you want to restart.

I’m aware of the similar mysqlpdump script, and I generally don’t like duplicating other people’s efforts, but I decided to go ahead and finish what I’d started. To tell you the truth, neither script is complicated. It’s just a matter of providing a sensible wrapper around existing functionality (in my case, that’s mysqldump and SELECT INTO OUTFILE, which I do directly rather than asking mysqldump to do it with -T, which just makes mysqldump into the same kind of wrapper). I also wanted to provide it as part of the MySQL Toolkit, so it’s all in the same place. Frankly, I also built in a lot more functionality than mysqlpdump has, and I consider the defaults to be more useful. I’d love for mysqldump itself to have better defaults — especially for dumping large datasets, which it’s frankly pretty poor at right now. One of these days MySQL AB will make me obsolete, I just know it…

Oh, and in keeping with my tradition, it’s sort of ridiculously sophisticated and overly generic. It has a little macro language that you can use to basically turn it into a loop iterator over the selected databases and tables, and run any command you wish. Here’s an example:

mysql-parallel-dump -- mysqldump --skip-lock-tables '%D' '%N' \| gzip --fast -c - \> '%D.%N.gz'

That basically duplicates the built-in defaults (except the defaults are actually a lot more complicated than that). But it illustrates how you could use this as a shell to select which tables to dump and fork off sub-processes, handling all the locking, error checking, and so forth for them. Here I’m spawning off mysqldump, but it would be just as easy to execute a custom script.

There’s one more wish Peter and I both have, but which is impossible for right now as far as we know. That’s to do parallel SELECT INTO OUTFILE dumps for a bunch of tables in one transaction. This will not be possible until more than one connection can participate in a single transaction. Ask the MySQL guys about that one!

This script is part of MySQL Toolkit and will be released as soon as I have time. There are a few other bug fixes I want to include. In the meantime, if you’re dying to get it, you can grab it from the MySQL Toolkit subversion repository.

You might also like:

1. MySQL Toolkit version 946 released
2. Introducing MySQL Parallel Restore
3. Progress on High Performance MySQL Backup and Recovery chapter
4. MySQL Toolkit version 1254 released
5. Introducing MySQL Deadlock Logger