Xaprb

As I was driving with a colleague in California a couple of weeks ago during the conference, the topic of conversation turned to the notion that Percona and the rest of the MySQL community really need the presence of a central entity that provides a recognized home for the MySQL server. The conversation went something like “I was talking to so-and-so, and he said, you know, you guys really need Sun/MySQL, because without the mother ship, things will fall apart and your own business will fail.”

I happen to believe this is FUD, and that the reverse might actually be true. (This is one reason why we’re competing head-on with MySQL.) Having a “mother ship” is in the long run, a very complex scenario to fully understand. There are all sorts of causes and effects that play out in subtle ways. I honestly doubt anyone can really know what the true effect is.

So what can we say about the presence or absence of a single controlling organization, then? Simply this: we can look for examples of where power is decentralized, and see how it fares. Let’s see some names:

• The Linux kernel.
• The GNU/Linux operating system in all its variations.
• PostgreSQL.
• Apache.

There are more that I could mention. But these are good enough examples to show that a thriving ecosystem is possible when power is not completely concentrated. In many ways, these projects have been far more successful than MySQL. I invite you to simply take a look at the Postgres hackers mailing list to see the PostgreSQL project’s success first-hand.

I can mention a few effects I’ve seen in the MySQL world that I don’t see elsewhere, and which I attribute to the presence of MySQL the corporation. The single biggest one I see is how the community around MySQL differs from, say, the community around PostgreSQL. As I said in my recent keynote address at the Percona Performance Conference, I think the best way to describe it is “MySQL has a community. PostgreSQL is a community.” (That’s a quote from a PostgreSQL community member.)

PostgreSQL isn’t the only project I can point to — Drizzle, for example, is taking a very good approach in my opinion. But any of these will do as an example. The key point is that there is a fundamentally different outlook on the relationship, both inside and outside MySQL, than there might be. On the outside, I have observed a lot of passivity and negativity. On the inside, I sometimes detect something that might be called paternalism.

Unflattering — yes, I’m being rather unflattering. But let’s be honest. The status quo between MySQL and “the community” is not good, not good at all. It resembles nothing so much as the relationship between users and producers of a proprietary product. That’s not bad for a proprietary product — but really bad for an open-source one. Just the fact that we talk about MySQL and “the community” as though they’re separate reveals a lot.

And I attribute this to the presence of a mother ship, however you might choose to label it. It creates a power relationship that I believe is a roadblock to a lot of good things. I know of examples in the last few weeks where the power relationship caused appalling problems with some of the foremost community members — situations that would never have happened in a relationship of equals. (I’m being discreet for a reason; I don’t want to cause any more heartache for those involved. Let’s just leave it at “the entire community suffered as a result.”) This kind of thing has been happening for years.

I firmly believe those who would truly participate in open source must recognize that there is no true meeting of the minds, unless everyone is an equal. Not just “treated as an equal” — if you hold the power, then you can’t be an equal, no matter how you try to treat others. Anyone who tries to be “more equal” than others is poisoning the well. The only solution I see is to divest oneself of power.

I think there are a lot of opportunities here for everyone involved. And those who are best placed to take those opportunities, but don’t, merely create a vacuum that draws others in to do it in their stead. And that’s why I think that ultimately, the mother ship structure is not only unnecessary and sometimes detrimental, but may even become irrelevant.

I will be very curious to see how things change in the future. MySQL has a lot going for it: an incredible team of talented, dedicated employees; visionary leaders; unprecedented technology landscape; economic conditions — it’s an enviable position. What I’m most interested in is to see how they will nurture the open-source nature of the product, which is its greatest strength — if it is teased out of the closed-source models that have bound it so tightly in recent years. The good news is, I think it’s theirs to lose; they can win if they play it right, but the rest of us win no matter what.

The day we see an announcement like this one for MySQL will be a great day indeed.

Cacti is a great tool for collecting information about systems and graphing it. However, it likes to use SNMP, and SNMP is often not desirable. Instead, I often see the need for a method that is:

• Secure. Use trusted, well-known, encrypted communication. Do not open up new ports.
• Zero install on the monitored system.
• As little installation or modification on the monitoring system as possible.

Over the last several years, I’ve slowly created more and more software to create Cacti graphs via standard POSIX command-line utilities over SSH with key-pair authentication. (I’ve also created similar software for Nagios, but that’s another matter.) The major problem with the work I’ve done is that it’s totally un-publicized.

The system works by passing command-line arguments to a local PHP script like any other Cacti script. This script then executes a remote SSH command, such as ssh somehost uptime and extracts statistics from the result.

The requirements are very simple. On the system to be monitored, a public key must be installed. On the monitoring system, the cacti user account must have a private SSH key that it can read and others can’t. This is standard for any SSH key. The cacti user account must also have the SSH key fingerprint of the monitored system in its known_hosts file.

The work I’ve done thus far is available from Subversion in the mysql-cacti-templates project. This project has the scaffolding for creating Cacti templates easily, so I’m using it.

At this point, the following are available:

• Apache
• Operating system (CPU, memory, load average, etc)
• Memcached
• Nginx

Unfinished work includes network, disk, etc. The techniques to monitor something of which there is a variable number (e.g. there can be many disks, each of which needs its own graph) are a little more complex than simple things like monitoring overall CPU usage. So that’s a work in progress. Once it’s done, it’ll make it really easy to discover and monitor multiples of anything — for example, multiple MySQL servers or memcached servers on a single host — without creating a new host for each resource to monitor.

As with the MySQL templates I created, these templates are comprehensive and have lots of nice properties most templates lack. This is something you get free with my scaffolding. If you’ve ever created templates by hand through the web interface, you should give my work a try. You can turn a three-day project into a few minutes and avoid bugs and other hassles. There are instructions for creating Cacti templates on the project wiki.

I have been using Maatkit in a different way since I joined Percona as a consultant. When I’m working on a system now, it’s a new, unfamiliar system — not one where I have already installed my favorite programs. And that means I want to grab my favorite productivity tools fast.

I intentionally wrote the Maatkit tools so they don’t need to be “installed.” You just run them, that’s all. But I never made them easy to download.

I fixed that. Now, at the command line, you can just run this:

wget http://www.maatkit.org/get/mk-table-sync

Now it’s ready to run. Behind the scenes are some Apache mod_rewrite rules, a Perl script or two, and Subversion. When you do this, you’re getting the latest code from Subversion’s trunk.[1][2] (I like to run on the bleeding edge. Releases are for people who want to install stuff.)

Because there’s some Perl magic behind it, I made it even easier — it does pattern-matching on partial names and Does The Right Thing:

baron@kanga:~$ wget http://www.maatkit.org/get/sync
--21:38:50--  http://www.maatkit.org/get/sync
           => `sync'
Resolving www.maatkit.org... 64.130.10.15
Connecting to www.maatkit.org|64.130.10.15|:80... connected.
HTTP request sent, awaiting response... 302 Moved
Location: http://www.maatkit.org/get/mk-table-sync [following]
--21:38:50--  http://www.maatkit.org/get/mk-table-sync
           => `mk-table-sync'
Connecting to www.maatkit.org|64.130.10.15|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-perl]

    [      <=>                            ] 163,259      136.51K/s             

21:38:51 (136.13 KB/s) - `mk-table-sync' saved [163259]

The redirection is there because otherwise wget will save the file under the name ‘sync’ instead of ‘mk-table-sync’.

And if you’ve forgotten which tools exist, you can just click on over to http://www.maatkit.org/get/ and see.

A quick poll: instead of getting the latest trunk, should this give you the code from the last release? I can do that, if you want.

[1] OK, it’s only refreshed every hour. So you’re getting code that’s up to an hour old.

[2] update: now /get/foo gets the latest release, and /trunk/foo gets the latest trunk code.