Xaprb

As I was driving with a colleague in California a couple of weeks ago during the conference, the topic of conversation turned to the notion that Percona and the rest of the MySQL community really need the presence of a central entity that provides a recognized home for the MySQL server. The conversation went something like “I was talking to so-and-so, and he said, you know, you guys really need Sun/MySQL, because without the mother ship, things will fall apart and your own business will fail.”

I happen to believe this is FUD, and that the reverse might actually be true. (This is one reason why we’re competing head-on with MySQL.) Having a “mother ship” is in the long run, a very complex scenario to fully understand. There are all sorts of causes and effects that play out in subtle ways. I honestly doubt anyone can really know what the true effect is.

So what can we say about the presence or absence of a single controlling organization, then? Simply this: we can look for examples of where power is decentralized, and see how it fares. Let’s see some names:

• The Linux kernel.
• The GNU/Linux operating system in all its variations.
• PostgreSQL.
• Apache.

There are more that I could mention. But these are good enough examples to show that a thriving ecosystem is possible when power is not completely concentrated. In many ways, these projects have been far more successful than MySQL. I invite you to simply take a look at the Postgres hackers mailing list to see the PostgreSQL project’s success first-hand.

I can mention a few effects I’ve seen in the MySQL world that I don’t see elsewhere, and which I attribute to the presence of MySQL the corporation. The single biggest one I see is how the community around MySQL differs from, say, the community around PostgreSQL. As I said in my recent keynote address at the Percona Performance Conference, I think the best way to describe it is “MySQL has a community. PostgreSQL is a community.” (That’s a quote from a PostgreSQL community member.)

PostgreSQL isn’t the only project I can point to — Drizzle, for example, is taking a very good approach in my opinion. But any of these will do as an example. The key point is that there is a fundamentally different outlook on the relationship, both inside and outside MySQL, than there might be. On the outside, I have observed a lot of passivity and negativity. On the inside, I sometimes detect something that might be called paternalism.

Unflattering — yes, I’m being rather unflattering. But let’s be honest. The status quo between MySQL and “the community” is not good, not good at all. It resembles nothing so much as the relationship between users and producers of a proprietary product. That’s not bad for a proprietary product — but really bad for an open-source one. Just the fact that we talk about MySQL and “the community” as though they’re separate reveals a lot.

And I attribute this to the presence of a mother ship, however you might choose to label it. It creates a power relationship that I believe is a roadblock to a lot of good things. I know of examples in the last few weeks where the power relationship caused appalling problems with some of the foremost community members — situations that would never have happened in a relationship of equals. (I’m being discreet for a reason; I don’t want to cause any more heartache for those involved. Let’s just leave it at “the entire community suffered as a result.”) This kind of thing has been happening for years.

I firmly believe those who would truly participate in open source must recognize that there is no true meeting of the minds, unless everyone is an equal. Not just “treated as an equal” — if you hold the power, then you can’t be an equal, no matter how you try to treat others. Anyone who tries to be “more equal” than others is poisoning the well. The only solution I see is to divest oneself of power.

I think there are a lot of opportunities here for everyone involved. And those who are best placed to take those opportunities, but don’t, merely create a vacuum that draws others in to do it in their stead. And that’s why I think that ultimately, the mother ship structure is not only unnecessary and sometimes detrimental, but may even become irrelevant.

I will be very curious to see how things change in the future. MySQL has a lot going for it: an incredible team of talented, dedicated employees; visionary leaders; unprecedented technology landscape; economic conditions — it’s an enviable position. What I’m most interested in is to see how they will nurture the open-source nature of the product, which is its greatest strength — if it is teased out of the closed-source models that have bound it so tightly in recent years. The good news is, I think it’s theirs to lose; they can win if they play it right, but the rest of us win no matter what.

The day we see an announcement like this one for MySQL will be a great day indeed.

Maatkit version 3519 is ready for download. There are a lot of changes in this release, many of which are incompatible with previous releases. There are also a lot of important new features. Read on for the details.

First, thanks to everyone who contributed to this month’s release. A lot of people have jumped into Maatkit and started committing code. I attribute this to deliberately forcing a more open policy with decisions being made on the mailing list, rather than the former policy of “Percona pays for development, so they have more say than you do” — a snobby and ill-advised way to treat an open-source project. If you are interested in contributing to Maatkit, please ask. Subversion commit rights are being handed out willy-nilly. It’s great!

Here’s a synopsis of this month’s most important changes.

• This is a work in progress. We’re making some pretty large changes, and to help us in the process, we’ve changed a lot of code to be more self-checking and help us find errors we introduce during the process. A lot of the other code is kind of involved in this too, so it’s being bundled together with other functionality. As a result, half the tools are done and half the tools aren’t touched yet. The Maatkit wiki has the details. If you’re curious about the reasoning behind the changes, please read the mailing list archives.
• Command-line options have changed. The mailing list members decided that Maatkit’s command-line options are too confusing and inconsistent, and voted to do something about it. As of this month, we’re about halfway through a process of converting all the tools to use consistent, carefully-thought-out command-line options. Next month I expect there will be even more changes. Check your wrapper scripts to make sure they don’t use old, deprecated options.
• Configuration files. We’re adding support for configuration files, as specified and approved on the mailing list. The functionality should be simple for MySQL users to understand.
• Better test coverage. We’re making the tools more testable. Again, a slow process. Very large portions of the code that is bundled together to make each tool is already tested pretty exhaustively, but there are parts of the tools that aren’t, and some of them are very difficult to test; we’re working on that.
• Fully integrated documentation. We’re working on integrating all the documentation into the tools, so there is no possibility of mismatch between behavior and documentation. We do this in general by making the tool derive its behavior from its documentation! This has proven to be very successful, and we have work underway to push that practice even further until no command-line help output is hard-coded anywhere.

Aside from that, we’ve made minor bug fixes and functionality changes this month, with the exception of mk-query-digest, which has some beta functionality and major bug fixes.

The full change log follows.

Changelog for mk-archiver:

2009-05-03: version 1.0.15

   * Added the --config option for issue 231.
   * Added the --help and --verbose options for issue 318.
   * Updates to shared code.

Changelog for mk-audit:

2009-05-03: version 0.9.7

   * Removed the --askpass long option.  Use --ask-pass instead.
   * Removed the --setvars long option.  Use --set-vars instead.
   * Removed the -t short option.  Use --top instead.
   * Added the following options for issue 248:
   *    --charset (-A)
   *    --defaults-file (-F)
   *    --host (-h) (but not implemented yet)
   * Converted script to runnable module (issue 315).
   * Added the --config option for issue 231.
   * Updates to shared code.

Changelog for mk-duplicate-key-checker:

2009-05-03: version 1.2.3

   * Columns with backticks in comments caused a crash (issue 330)
   * Changed the --allstruct option to --all-structs.
   * Changed the --askpass option to --ask-pass.
   * Changed the --engine option to --engines.
   * Changed the --fuction option to --key-types.
   * Changed the --ignoredb option to --ignore-databases.
   * Changed the --ignoreengine option to --ignore-engines.
   * Changed the --ignoreorder option to --ignore-order.
   * Changed the --ignoretbl option to --ignore-tables.
   * Changed the --setvars option to --set-vars.
   * Removed the -a short option.  Use --all-struct instead.
   * Removed the -c short option.  Use --[no]clustered instead.
   * Removed the -f short option.  Use --key-types instead.
   * Removed the -g short option.  Use --ignore-databases instead.
   * Removed the -E short option.  Use --ignore-engines instead.
   * Removed the -n short option.  Use --ignore-tables instead.
   * Added config file handling and --config (issue 231).
   * Converted script to runnable module (issue 315).

Changelog for mk-parallel-dump:

2009-05-03: version 1.0.15

   * Columns with backticks in comments caused a crash (issue 330)

Changelog for mk-query-digest:

2009-05-03: version 0.9.5

   * The query report printed duplicate table names (issue 337).
   * Print a message and exit early if there's an error (issue 190).
   * Added the --config option for issue 231.
   * Added the --log option for issue 241.
   * Added the --help and --verbose options for issue 318.
   * Fixed another crash when sqrt() of a negative number (issue 332).
   * Fixed a division by zero when a query has zero exec time.
   * Added --print to print query events in slow-log format.
   * Added --type to specify the type of log file (default slowlog).
   * Added --tcpdump to permit parsing output of tcpdump (issue 228).
   * The --shorten option was implemented badly and was slow (issue 336).
   * The report's per-class QPS was calculated incorrectly (issue 326).
   * Updates to shared code.

Changelog for mk-query-profiler:

2009-05-03: version 1.1.15

   * Added the --config option for issue 231.
   * Converted script to runnable module (issue 315). 
   * mk-query-profiler only:
   *    Removed the --allowcache long option.  Use --allow-cache instead.
   *    Removed the --askpass long option.  Use --ask-pass instead.
   *    Removed the --setvars long option.  Use --set-vars instead.
   *    Removed the -a short option.  Use --allow-cache instead.
   *    Removed the -c short option.  Use --calibrate instead.
   *    Removed the -e short option.  Use --external instead.
   *    Removed the -f short option.  Use --flush instead.
   *    Removed the -i short option.  Use --innodb instead.
   *    Removed the -n short option.  Use --only instead.
   *    Removed the -s short option.  Use --separate instead.
   *    Removed the -t short option.  Use --tab instead.
   *    Removed the -r short option.  Use --verify instead.
   * mk-profile-compact only:
   *    Removed the -q short option.  Use --queries instead.
   *    Removed the -m short option.  Use --mode instead.
   *    Removed the -h short option.  Use --headers instead.

Changelog for mk-show-grants:

2009-05-03: version 1.0.15

   * The tool crashed when there were no users (issue 359).

Changelog for mk-slave-delay:

2009-05-03: version 1.0.13

   * Removed the --askpass long option.  Use --ask-pass instead.
   * Removed the --setvars long option.  Use --set-vars instead.
   * Removed the --usemaster long option.  Use --use-master instead.
   * Removed the -d short option.  Use --delay instead.
   * Removed the -c short option.  Use --continue instead.
   * Removed the -q short option.  Use --quiet instead.
   * Removed the -t short option.  Use --run-time instead.
   * Removed the --time long option.  Use --run-time instead.
   * Removed the -u short option.  Use --use-master instead.
   * Removed the -i short option.  Use --interval instead.
   * Added the -q short option for --quiet.
   * Added the --config option for issue 231.
   * Added the --log option for issue 241.
   * Added the following options for issue 248:
   *    --charset (-A)
   *    --defaults-file (-F)
   *    --host (-h)
   *    --password (-p)
   *    --port (-P)
   *    --socket (-S)
   *    --user (-u)
   * Converted script to runnable module (issue 315).

Changelog for mk-slave-find:

2009-05-03: version 1.0.6

   * Removed the --print long option; replication hierarchy tree always printed.
   * Removed the --setvars long option.  Use --set-vars instead.
   * Removed the --askpass long option.  Use --ask-pass instead.
   * Removed the -r short option.  Use --recurse instead.
   * Added the --config option for issue 231.
   * Converted script to runnable module (issue 315).
   * Defaults files were not read properly.
   * Added ability to specify master host with DSN.
   * Updated POD to describe what script actually does.
   * Updates to shared code.

Changelog for mk-slave-move:

2009-05-03: version 0.9.7

   * Removed the --setvars long option.  Use --set-vars instead.
   * Removed the --askpass long option.  Use --ask-pass instead.
   * Removed the -m short option.  Use --timeout instead. 
   * Added the --config option for issue 231.
   * Added the following options for issue 248:
   *    --charset (-A)
   *    --defaults-file (-F)
   *    --host (-h)
   *    --password (-p)
   *    --port (-P)
   *    --socket (-S)
   *    --user (-u)
   * Converted script to runnable module (issue 315).
   * Updates to shared code.

Changelog for mk-slave-prefetch:

2009-05-03: version 1.0.7

   * Removed the --askpass long option.  Use --ask-pass instead.
   * Removed the --checkint long option. Use --check-interval instead.
   * Removed the --iolag long option.  Use --io-lag instead.
   * Removed the --maxquerytime option.  Use --max-query-time instead.
   * Removed the --setvars long option.  Use --set-vars instead.
   * Removed the --numprefix long option.  Use --num-prefix instead.
   * Removed the --permitregexp long option.  Use --permit-regexp instead.
   * Removed the --printnonrewritten long option.  Use --print-nonrewritten
     instead.
   * Removed the --querysampsize long option.  Use --query-sample-size instead.
   * Removed the --rejectregexp long option.  Use --reject-regexp instead.
   * Removed the --setvars long option.  Use --set-vars instead.
   * Removed the -i short option.  Use --check-interval instead.
   * Removed the -x short option.  Use --execute instead.
   * Removed the -l short option.  Use --io-lag instead.
   * Removed the -q short option.  Use --max-query-time instead.
   * Removed the -o short option.  Use --offset instead.
   * Removed the -t short option.  Use --run-time instead.
   * Removed the --time long option.  Use --run-time instead.
   * Removed the -w short option.  Use --window instead.
   * Added the --config option for issue 231.
   * Added the --log option for issue 241.
   * --errors did not work properly.
   * Converted script to runnable module (issue 315).
   * --print and --daemonize are no longer mutually exclusive.

Changelog for mk-slave-restart:

2009-05-03: version 1.0.13

   * Added the --log option for issue 241.
   * Added the --config option for issue 231.
   * Added the --help and --verbose options for issue 318.
   * Removed the --setvars long option.  Use --set-vars instead.
   * Removed the --askpass long option.  Use --ask-pass instead.
   * Removed the -L short option.  Use --error-length instead.
   * Removed the -E short option.  Use --error-text instead.
   * Removed the -M short option.  Use --max-sleep instead.
   * Removed the --maxsleep long option.  Use --max-sleep instead.
   * Removed the -m short option.  Use --min-sleep instead.
   * Removed the --minsleep long option.  Use --min-sleep instead.
   * Removed the -r short option.  Use --recurse instead.
   * Removed the -k short option.  Use --skip-count instead.
   * Removed the --skipcount long option.  Use --skip-count instead.
   * Removed the -s short option.  Use --sleep instead.
   * Removed the -t short option.  Use --run-time instead.
   * Removed the --time long option.  Use --run-time instead.
   * Removed the --untilmaster long option.  Use --until-master instead.
   * Removed the --untilrelay long option.  Use --until-relay instead.
   * Removed the -v short option.  Use --verbose instead.
   * Converted script to runnable module (issue 315).

Changelog for mk-table-checksum:

2009-05-03: version 1.2.5

   * Columns with backticks in comments caused a crash (issue 330)

Changelog for mk-table-sync:

2009-05-03: version 1.0.15

   * Columns with backticks in comments caused a crash (issue 330)
   * Added --lock-and-rename (issue 363).

Changelog for mk-visual-explain:

2009-05-03: version 1.0.14

   * Changed the --askpass option to --ask-pass.
   * Changed the --clusterpk option to --clustered-pk.
   * Changed the --setvars option to --set-vars.
   * Removed the -C short option.  Use --clustered-pk instead.
   * Removed the -c short option.  Use --connect instead.
   * Removed the -f short option.  Use --format instead.
   * Added config file handling and --config (issue 231).
   * Converted script to runnable module (issue 315).

At the MySQL conference, a person who used to hold an important position in an important sales organization told us something like the following: “You know, you guys at Percona are great, but you have a big problem. You don’t have any $500,000 customers who only file one support incident per year. Those customers are where you can really make big money.”

We were well aware of the investigations this person did into which customers are the most profitable, and we had decided a long time ago that chasing huge sales without delivering matching services is flawed. I told this person as much: “that model is fundamentally broken because it doesn’t align cost with value delivered.”

Actually, we do have customers who rank in the top 50 of the Fortune 500. I’m thinking of one right now. They filed only one case so far. And they’ve paid us for exactly the amount of hours we spent on that case. And far from being a problem, this is exactly where we’re doing things right.

Some people may see us as shrinking the pie by billing just a few thousand dollars for “huge jobs.” They might think we’re silly for telling the customer that a few hours on the phone and remotely is going to be better for them than a week onsite. They may consider us small peanuts because we don’t have 10-million-dollar deals. What they are missing is what the customers see clearly: it’s good to write win-win contracts and leave something on the table. If it looks like table scraps to Mr. Million Dollar Salesman, that’s okay. Those bloated “profitable” contracts are short-term thinking, and they’re actually a serious weakness. Don’t celebrate our “problem” too much. Keep an eye in the rearview mirror. Is it a problem, or is it our secret sauce?

One problem I think we might actually have is how to get a good sales person with our model. You see, we don’t actually have any sales staff at all right now. I am not sure the math would work out the way a traditional sales person would like. But that’s because a traditional sales person is used to being rewarded for “earning money for the company,” which I think is really broken, at least in the way it’s traditionally implemented.

I’ve said before that I think one of the reasons MySQL was unable to create an open-source business model is that their sales folks pushed the company in the direction of closed source. I’m honestly not sure what the best open-source sales model is. I know that I really feel good about our services delivery model and our pricing model; right now our sales model is nonexistent. We are passive; we just answer inquiries and sign contracts. As things stand now, nobody’s doing any sales.

Still, I think this is a good problem to have, and I trust that the right person will come along and see the opportunity to create a good salary at the same time as really providing a service to Percona and the customers. That person will explain to us how cost (salary) and value (service to us and customers) can be impedance-matched too. And that will probably be an innovation, or at least unconventional. With all the other ways Percona is unconventional, I don’t expect us to create a conventional sales model either.