Xaprb » memcached

Observations on key-value databases

Xaprb — Sun, 20 Sep 2009 18:57:01 +0000

Key-value databases are catching fire these days. Memcached, Redis, Cassandra, Keyspace, Tokyo Tyrant, and a handful of others are surging in popularity, judging by the contents of my feed reader.

I find a number of things interesting about these tools.

There are many more of them than open-source traditional relational databases. (edit: I mean that there are many options that all seem similar to each other, instead of 3 or 4 standing out as the giants.)
It seems that a lot of people are simultaneously inventing solutions to their problems in private without being aware of each other, then open-sourcing the results. That points to a sudden sea change in architectures. Tipping points tend to be abrupt, which would explain isolated redundant development.
Many of the products are feature-rich with things programmers need: diverse language bindings, APIs, embeddability, and the ability to speak familiar protocols such as memcached protocol.
I think there are more solutions here than the ecosystem will support, and in five years a few will stand out as the most popular.
This process of paring down the gene pool is win-win because they’re open-source, and nothing will be lost.
Choosing which one to use is no easy task even for a highly skilled, technical, up-to-date person. Perhaps the decision-makers will choose on the availability of commercial support and consulting.
Many of them offer built-in, dead-simple, distributed, synchronous replication. This is very difficult to achieve with traditional relational databases. What makes key-value databases different? They don’t have MVCC, for one thing; but I’m not sure of the complete answer to that question, to tell the truth.

We live in interesting times.

Further Reading:

Analyze and optimize memcached usage with Maatkit

Xaprb — Sat, 25 Jul 2009 14:08:14 +0000

Ryan posted an article on the MySQL Performance Blog about how to use mk-query-digest to analyze and understand your memcached usage with the same techniques you use for MySQL query analysis. This is an idea that came to me during the 2009 MySQL Conference, while talking to our friends from Schooner, who sell a memcached appliance.

It suddenly struck me that the science of memcached performance is basically nonexistent, from the standpoint of developers and architects. Everyone treats it as a magical tool that just performs well and doesn’t need to be analyzed, which is demonstrably and self-evidently false. memcached itself is very fast, true, so it doesn’t usually become a performance bottleneck the way a database server does. But that’s not the point. There is a lot to win or lose in the way you use it, which can heavily influence your application’s performance. That’s what the new features in mk-query-digest are designed to analyze.

Here’s an example of the types of problems we’ve seen in production memcached usage, which are very hard to catch without a good tool. What if a “global” value is accidentally stored with a key that includes the user ID? This will cause the value to be duplicated again and again for every user, instead of being stored once. There are really only two ways to catch this: 1) know the application’s source code inside and out, and 2) analyze the memcached traffic scientifically. (Even if you know the source code well, there’s a good chance you can miss a bug like this.) I could go on listing the types of problems you can inadvertently create with a key-value database, but I’ll leave it at that.

The features are only available in trunk, and will be released with this month’s scheduled release.

Further Reading:

Version 1.1.2 of improved Cacti templates released

Xaprb — Thu, 07 May 2009 04:31:51 +0000

I’ve packaged up and released version 1.1.2 of the Cacti templates I’ve written for MySQL, Apache, memcached, nginx etc.

Anyone who would like to help write documentation (or do anything else, for that matter) is welcomed to participate. I’ll give commit access at the drop of a hat.

Changelog:

2009-05-07: version 1.1.2

	* The parsing code did not handle InnoDB plugin / XtraDB (issue 52).
	* The servername was hardcoded in ss_get_by_ssh.php (issue 57).
	* Added Handler_ graphs (issue 47).
	* Config files can be used instead of editing the .php file (issue 39).
	* binary log space is now calculated without a MySQL query (issue 48).
	* There was no easy way to force inputs to be filled (issue 45).
	* Some graphs were partially hidden without --lower-limit (issue 43).
	* Flipped some elements across the Y axis (issue 42).
	* Added Apache, Nginx, and GNU/Linux templates.
	* Unknown output is now -1 instead of 0 to prevent spikes in graphs.
	* If you want to use a script server, you must now explicitly configure it.
	* UNIX sockets weren't permitted for MySQL (issue 38).

Further Reading:

Secure, easy Cacti graphing without SNMP

Xaprb — Sat, 25 Apr 2009 19:01:02 +0000

Cacti is a great tool for collecting information about systems and graphing it. However, it likes to use SNMP, and SNMP is often not desirable. Instead, I often see the need for a method that is:

Secure. Use trusted, well-known, encrypted communication. Do not open up new ports.
Zero install on the monitored system.
As little installation or modification on the monitoring system as possible.

Over the last several years, I’ve slowly created more and more software to create Cacti graphs via standard POSIX command-line utilities over SSH with key-pair authentication. (I’ve also created similar software for Nagios, but that’s another matter.) The major problem with the work I’ve done is that it’s totally un-publicized.

The system works by passing command-line arguments to a local PHP script like any other Cacti script. This script then executes a remote SSH command, such as ssh somehost uptime and extracts statistics from the result.

The requirements are very simple. On the system to be monitored, a public key must be installed. On the monitoring system, the cacti user account must have a private SSH key that it can read and others can’t. This is standard for any SSH key. The cacti user account must also have the SSH key fingerprint of the monitored system in its known_hosts file.

The work I’ve done thus far is available from Subversion in the mysql-cacti-templates project. This project has the scaffolding for creating Cacti templates easily, so I’m using it.

At this point, the following are available:

Apache
Operating system (CPU, memory, load average, etc)
Memcached
Nginx

Unfinished work includes network, disk, etc. The techniques to monitor something of which there is a variable number (e.g. there can be many disks, each of which needs its own graph) are a little more complex than simple things like monitoring overall CPU usage. So that’s a work in progress. Once it’s done, it’ll make it really easy to discover and monitor multiples of anything — for example, multiple MySQL servers or memcached servers on a single host — without creating a new host for each resource to monitor.

As with the MySQL templates I created, these templates are comprehensive and have lots of nice properties most templates lack. This is something you get free with my scaffolding. If you’ve ever created templates by hand through the web interface, you should give my work a try. You can turn a three-day project into a few minutes and avoid bugs and other hassles. There are instructions for creating Cacti templates on the project wiki.

Further Reading:

MySQL Conference and Expo 2008, Day Three

Xaprb — Sat, 19 Apr 2008 05:22:31 +0000

Here’s a rundown of Thursday (day 3) of the MySQL Conference and Expo. This day’s sessions were much more interesting to me than Wednesday’s, and in fact I wanted to go to several of them in a single time slot a couple of times.

Inside the PBXT Storage Engine

This session was, as it sounds, a look at the internals of PBXT, a transactional storage engine for MySQL that has some interesting design techniques. I had been looking forward to this session for a while, and Paul McCullagh’s nice explanations with clear diagrams were a welcome aid to understanding how PBXT works. Unlike some of the other storage engines, PBXT is being developed in full daylight, with an emphasis on community involvement and input. (Indeed, I may be contributing to it myself, in order to make its monitoring and tuning capabilities second to none).

PBXT has not only a unique design, but a clear vision for differentiating itself from other transactional storage engines. It’s not trying to clone any particular engine; Paul and friends are planning to add some capabilities that will really set it apart from other engines, including high-availability features and blob streaming.

I left this session with a much better understanding of how PBXT balances various demands to satisfy all sorts of different workload characteristics, how it writes data, how it achieves transactional durability, and so on. I think these capabilities, and its performance, can really be assessed only in the real world (of course), but in principle it sounds good. I love knowing how things work!

There were about 30 people in the talk. I wish there had been more, because I think PBXT is going to be an important part of the open ecosystem going forward. However, I feel pretty confident people will take more notice if it starts to get used in the real world. Someone had a video camera there, so you might check out the video when it’s available. Paul’s explanations are really good.

Helping InnoDB Scale on Servers with Many CPU Cores and Disks

This session was Mark Callaghan’s chance to unveil the work he and others have been doing on InnoDB’s scalability issues, which mostly revolve around mutex contention. Mark’s team has completely solved the problems on their workload and benchmarks. In fact, after the changes, InnoDB exhibited significantly better performance even than MyISAM, which began to be limited by the single mutex that synchronizes access to its key cache. (Yes, in fact MyISAM has scalability problems too).

Google’s workload for MySQL, in case you’re wondering, is pretty traditional (i.e. not web-like; more like an “enterprise” application). Heavily I/O-bound, 24/7 critical systems, and so on.

Mark also wore several community t-shirts at various points in the talk, including one of my Maatkit t-shirts. Mark said Maatkit would be perfect if only it were written in Python (Google’s preferred scripting language). Alas, Mark, it’ll stay in Perl. But thanks for the nice compliment anyway.

The room was packed full.

Scaling Heavy Concurrent Writes In Real Time

Dathan Pattishall, formerly the lead architect at Flickr, explained his techniques for scaling Flickr’s write capacity. He talked about how he’d worked to reduce primary key sizes, queued writes for batching, separated different types of data into different types of tables, and more. Dathan has never been afraid to do what he thinks is a good idea, even if it flies in the face of “best practices,” so I was happy to finally hear him talk.

By the way, Dathan pointed out that distributed locking with memcached and add() isn’t a silver bullet. It works ok until memcached evicts your lock due to the LRU policy. He uses MySQL’s built-in GET_LOCK() function for locking.

Dathan’s blog is a good source of information about his sometimes unorthodox approaches to database design.

The Power of Lucene

This was the only one of Frank (Farhan) Mashraqi’s talks I got to attend. This was pretty technical: how Lucene works, how to configure and install it, how to index documents, how to execute searches. If you were wondering how much work and complexity it would be to install and use Lucene, this talk would have been good for you to attend; I’ve never used it myself, but I’m pretty sure Frank covered everything you need to know.

Further Reading: