Xaprb

At the MySQL conference, a person who used to hold an important position in an important sales organization told us something like the following: “You know, you guys at Percona are great, but you have a big problem. You don’t have any $500,000 customers who only file one support incident per year. Those customers are where you can really make big money.”

We were well aware of the investigations this person did into which customers are the most profitable, and we had decided a long time ago that chasing huge sales without delivering matching services is flawed. I told this person as much: “that model is fundamentally broken because it doesn’t align cost with value delivered.”

Actually, we do have customers who rank in the top 50 of the Fortune 500. I’m thinking of one right now. They filed only one case so far. And they’ve paid us for exactly the amount of hours we spent on that case. And far from being a problem, this is exactly where we’re doing things right.

Some people may see us as shrinking the pie by billing just a few thousand dollars for “huge jobs.” They might think we’re silly for telling the customer that a few hours on the phone and remotely is going to be better for them than a week onsite. They may consider us small peanuts because we don’t have 10-million-dollar deals. What they are missing is what the customers see clearly: it’s good to write win-win contracts and leave something on the table. If it looks like table scraps to Mr. Million Dollar Salesman, that’s okay. Those bloated “profitable” contracts are short-term thinking, and they’re actually a serious weakness. Don’t celebrate our “problem” too much. Keep an eye in the rearview mirror. Is it a problem, or is it our secret sauce?

One problem I think we might actually have is how to get a good sales person with our model. You see, we don’t actually have any sales staff at all right now. I am not sure the math would work out the way a traditional sales person would like. But that’s because a traditional sales person is used to being rewarded for “earning money for the company,” which I think is really broken, at least in the way it’s traditionally implemented.

I’ve said before that I think one of the reasons MySQL was unable to create an open-source business model is that their sales folks pushed the company in the direction of closed source. I’m honestly not sure what the best open-source sales model is. I know that I really feel good about our services delivery model and our pricing model; right now our sales model is nonexistent. We are passive; we just answer inquiries and sign contracts. As things stand now, nobody’s doing any sales.

Still, I think this is a good problem to have, and I trust that the right person will come along and see the opportunity to create a good salary at the same time as really providing a service to Percona and the customers. That person will explain to us how cost (salary) and value (service to us and customers) can be impedance-matched too. And that will probably be an innovation, or at least unconventional. With all the other ways Percona is unconventional, I don’t expect us to create a conventional sales model either.

Peter wrote about this recently, but I don’t know if it was really clear what was going on.

Point One: Sphinx can be contacted by the MySQL protocol. Not “as a MySQL storage engine.” Not “from MySQL.” It understands the MySQL protocol itself. So from the protocol point of view, the Sphinx search daemon can look just like a MySQL server.

Point Two: Sphinx understands a SQL-like query language. Don’t be fooled. You’re not writing SQL. It just looks like you are.

Point Three: Because of point One and point Two, you can use the mysql command-line client program to talk directly to Sphinx, with absolutely no MySQL server anywhere in sight. This also means you can connect to Sphinx from your application and query it, exactly like connecting to a MySQL server and querying it.

Go take a look at Peter’s blog post. He’s not writing MySQL queries. He’s writing queries to Sphinx.

Now think about how cool this is — how easy this is to integrate with your code that already communicates with MySQL. Is there any other external full-text search system that masquerades as a MySQL server? I don’t know of one.

Cacti is a great tool for collecting information about systems and graphing it. However, it likes to use SNMP, and SNMP is often not desirable. Instead, I often see the need for a method that is:

• Secure. Use trusted, well-known, encrypted communication. Do not open up new ports.
• Zero install on the monitored system.
• As little installation or modification on the monitoring system as possible.

Over the last several years, I’ve slowly created more and more software to create Cacti graphs via standard POSIX command-line utilities over SSH with key-pair authentication. (I’ve also created similar software for Nagios, but that’s another matter.) The major problem with the work I’ve done is that it’s totally un-publicized.

The system works by passing command-line arguments to a local PHP script like any other Cacti script. This script then executes a remote SSH command, such as ssh somehost uptime and extracts statistics from the result.

The requirements are very simple. On the system to be monitored, a public key must be installed. On the monitoring system, the cacti user account must have a private SSH key that it can read and others can’t. This is standard for any SSH key. The cacti user account must also have the SSH key fingerprint of the monitored system in its known_hosts file.

The work I’ve done thus far is available from Subversion in the mysql-cacti-templates project. This project has the scaffolding for creating Cacti templates easily, so I’m using it.

At this point, the following are available:

• Apache
• Operating system (CPU, memory, load average, etc)
• Memcached
• Nginx

Unfinished work includes network, disk, etc. The techniques to monitor something of which there is a variable number (e.g. there can be many disks, each of which needs its own graph) are a little more complex than simple things like monitoring overall CPU usage. So that’s a work in progress. Once it’s done, it’ll make it really easy to discover and monitor multiples of anything — for example, multiple MySQL servers or memcached servers on a single host — without creating a new host for each resource to monitor.

As with the MySQL templates I created, these templates are comprehensive and have lots of nice properties most templates lack. This is something you get free with my scaffolding. If you’ve ever created templates by hand through the web interface, you should give my work a try. You can turn a three-day project into a few minutes and avoid bugs and other hassles. There are instructions for creating Cacti templates on the project wiki.