Xaprb

It’s been a while since I said anything about the progress on the book. That doesn’t mean we are not still working on it, though.

As Peter wrote a while ago, he is basically wearing the hat of a very advanced technical reviewer at this point. We’ve finished writing all the chapters from his detailed outlines. He has worked through about half the chapters, and I’m continuing to spend my evenings and weekends and holidays (yes, nearly all my free time — just ask my wife!) writing some new material (an appendix on EXPLAIN, for example), finishing unfinished things marked with TODO in the text, and revising chapters after Peter reviews them. Vadim is working on benchmarks. For example, he just finished some benchmarks for something I profiled with SHOW STATUS. I thought that would be good enough to assert something about the performance. Sure enough, SHOW STATUS says it does less work, but Vadim’s benchmarks show it’s slower :-) This is why we check each other’s work!

The core chapters on MySQL performance — beginning with Benchmarking and Profiling, and continuing through Optimizing Server Settings — are the ones Andy Oram, our editor, thinks we should put the most effort into, and I agree. We will probably circle back and go through another review/edit cycle before we release them for technical review. Some of the other chapters, such as Replication, are already out for technical review.

Despite the fact that all of the chapters and appendixes are theoretically a “first draft,” as of several weeks ago, there is still a lot of work to do. Depending on the chapter, it takes me a solid weekend to revise a chapter after Peter reviews it. Each little thing anyone points out (does MySQL version X really do Y by default?) requires some research, testing, benchmarks, or even reading the source code.

Some miscellanea:

• The production staff replied to my inquiry to the editor to say that yes, we will be able to have references that point to a specific page number. This was a big relief to me. It requires extra work, but makes the book so much more valuable as a reference work in my opinion. To see why, look at the top of page 151 in the first section, which just refers to chapters and sections by their titles: “See… the “Tools” section…” Now try to find the “Tools” section. If it took you a while… well, the first time I did it, I missed it, and thought it might mean the Tools Chapter. The second edition will say “The X section on page Y” or similar. (Okay, I’ll shut up about this now — everyone has to have a pet peeve, eh?)
• We are currently at 425 pages in OpenOffice.org Writer, which by my calculations puts us around 470 pages in print. As I said before, I think we’ll break 500 pages by the time we finish the rest of the missing material.
• Andrew Aksyonoff has contributed an appendix on the Sphinx full-text search system. If you don’t know anything about it, check it out. It’s an amazing piece of software that does a lot more than just full-text search.

Well, I’ve run out of my allotted thirty minutes of blogging! Back to the salt mines! Just kidding… I’m actually off to the climbing gym soon to get my mind off it.

A while ago I asked for people and/or organizations to sponsor development on Maatkit (formerly MySQL Toolkit) so I could take a week off work and improve the Table Sync tool. I asked for $2500 USD, but several companies have graciously offered to cover that and then some.

I’m very happy about this, as it will allow me to dedicate a solid week to fixing bugs and adding features. There’s a lot of demand for the tools, and there are a dozen or so bug reports unresolved for the table-sync tool, which I personally want to fix as much as anyone. So I’m very grateful for the support.

Here are the companies who have promised their financial support:

MySQL AB

MySQL AB have offered $3000 USD in support. I had an email conversation with MÃ¥rten Mickos, MySQL’s CEO, and he expressed his happiness about the project’s success, and his pleasure in supporting the project:

We have seen you operate in the community and you always have constructive and good ideas. That’s why we want to support you. Our goal with this is to stimulate innovation in the MySQL ecosystem.

I don’t know how the idea to support the project started at MySQL AB, but that quote really tells me “we get it: we have a symbiotic relationship with our community of users.” In a follow-up email, Jay Pipes wrote,

… MySQL wants to make it clear that we very much support and appreciate the work you’ve done on the toolkit. It has proven to be one of, if not the, most popular and successful open source ecosystem projects surrounding MySQL and for good reason. So, for your work and commitment to the project, a big thank you from MySQL. :)

Secondly, we would like to encourage you to be open and public about our support of you. The community team is always looking for opportunities such as the one which presented itself with your toolkit, and we want the outside community to know about our support and encouragement. Therefore, you have our blessing and encouragement to blog about the sponsorship of your development work. Please do let us know if and when you decide to blog about it. Remember also that this sponsorship is no strings attached. There is no expectation of specific work on our end.

Blue Ridge Internetworks

Blue Ridge Internetworks have offered $1000 USD in support. BRIworks, as they’re known locally, is headquartered in the town where I live, Charlottesville, Virginia. They offer networking consulting and services. Jeff Cornejo, who offered the support to me, is a friend and used to work where I used to work, and several other highly respected friends and ex-co-workers work at BRIworks too. BRIworks provides Internet service and hosting for my employer.

Percona

Percona have offered $500 USD in support. Percona does high-performance website consulting, and are perhaps best known for having some of the world’s top MySQL experts, including Peter Zaitsev and Vadim Tkachenko, two of the co-authors on High Performance MySQL, second edition.

The Rimm-Kaufman Group

Last, but absolutely not least, my employer, The Rimm-Kaufman Group, who do paid search marketing and website effectiveness consulting. They have let me spend a significant amount of time writing these tools for use on our own systems, and instead of keeping them in our own Subversion repository, allowed the code to be released as Free Software. The time I’ve spent on the tools has gone well above and beyond what we needed to get our work done. Finally, RKG has blessed my unpaid week off to work on the tools.

A big thanks is due to all of these companies and individuals, as well as other people who have contributed financially and otherwise.

Closing thoughts

I’m grateful for the sponsorship, but I think the real winners are the MySQL community, who have benefited a lot from Maatkit. It has made a lot of hard things easier and impossible things possible. If you’re one of those who benefits from Free Software, I encourage you to patronize the businesses that believe in and support it. Four fine examples are listed above! Not coincidentally, all of them are the creme de la creme in their respective fields.

Finally, a quick journalistic note: I pre-approved this post with representatives from the companies I mentioned, because I respect their right to represent themselves as they wish, but the words are mine, not theirs.

It’s been a while since I’ve written about progress on the book. I actually stopped working on it as much at the beginning of the month, because on October 31st I managed to finish a first draft of the last big chapter (Scaling and High Availability)! Now I’m back to full-time work at my employer, and I’m working on the book in the evenings and weekends only.

This doesn’t mean the book is close to being done, though. The editor is sending out some chapters for technical review, and there’s still a lot more writing and revising to be done.

Last weekend I revised the Security chapter from the first edition, which I think will be the only chapter that we’ll just revise and update, rather than completely rewriting (well, maybe the Architecture chapter could be considered a revision instead of a rewrite, but it’s a stretch; we changed it a lot). I removed a lot of the material that repeated the MySQL manual, and added a lot of information and best practices on grants, new privileges and objects in MySQL 5, common tasks, common mistakes, and so on. The chapter ended up being nearly as long, even though I stripped out all the code listings and so on from the first edition (in fact, I reduced the first edition’s material to a few paragraphs).

Beyond that, though, there are little details to finish out in many of the chapters. Examples that need to be finished, figures that need to be re-drawn, material that doesn’t quite fit and needs to be re-arranged or even moved to another chapter; it’s a lot of work. Peter Zaitsev has been reviewing some of the core chapters on query and schema optimization etc, and I’m revising them in response to his comments. That’s what I spent today doing.

I think the biggest chunks of work that remain are going to be making chapters 3, 4, 5, 6 and 7 (benchmarking, profiling, schema, indexing, query optimization, advanced features, and server tuning) flow together well. The challenge here is how to organize the vast amount of material so it reads well, without too many forward references, and still be useful as a reference work. The detail we’ve gone into is incredible. It makes it very hard to find the single best place to mention each little bit of wisdom, because all of this material is completely inter-related. It’s tough to flatten the graph of knowledge into a one-dimensional narrative.

It’s not just these chapters that have a lot of inter-related material, of course. It’s hard to talk about tuning the server settings (chapter 7) without bringing the OS and hardware (chapter 8) into it, and whenever you do this you also need to think about measuring and monitoring status information (chapter 14). Of course, you need to do that for benchmarking and profiling, too (chapter 3). I’m sure you see the dilemma!

The good news is, if we succeed in doing this well, you will find the book enormously useful. Stay tuned!