Archive for March, 2008

Going to PostgreSQL Conference East

I’m heading out to PostgreSQL Conference East in a few hours. Alas, I missed the first day but I’ll be there all day tomorrow. I hope to learn, meet people, and generally participate in goodness.

And no, I’m not an expert in PostgreSQL as I am in MySQL, but I have always held it in very high esteem. I am not going to try to spread the MySQL-ness into the PostgreSQL camp :-)

See you there.

Technorati Tags:,

You might also like:

  1. Like it or not, it is the MySQL Conference and Expo

Stock images are too popular

I have an ingrained (possibly even genetic) aversion to stock images. Actually, not all stock: just the vacuous kind. You know what I mean: like the politically-correct, gender-balanced, racially-balanced, age-diverse ones where people are all smiling and pointing at a computer screen you can’t see. Ugh!

Business Group Meeting

(Photo credit: istockphoto.com)

There are many reasons not to use images like this. I guess it’s okay in some situations — for example when you just want a smiling, attractive woman with a customer-service headset to reinforce that you’ve come to the right place for support. However, even these really don’t have to be stock images. One of my former employers used their own employees for such photos, almost exclusively, and it made the site much more real. And there are plenty of examples of companies that use photos of their own employees and get “realness” as a result. If I’m not mistaken, Title Nine does so except for certain things, such as underwear models (for obvious reasons).

However, one great reason to eschew stock: other people will re-use the same image. A famous example from a few years ago: the cover image of Head First Design Patterns was a stock photo that also appeared in a commercial for a feminine hygiene product.

This incident was actually pretty widely linked on the Internet at that time. So no one will ever make that mistake again!

Or will they? Witness: the cover of the MySQL 5.1 Cluster DBA Certification Guide, the xTuple Home Page, and the cover of the MegaRAID Management Suite documentation.

Stock images ad nauseum

Interestingly, I ran across all three over-usages of this image in one day, completely by accident. Are there other places this image is used? I’d bet there are.

Who cares? Well, the images that go on the cover of your book, your brochure, or your website become part of your image. If someone else then uses the same image, they can (accidentally or otherwise) exert some control over what people think of your product or company.

If this matters — and it almost certainly does — you should just get some of your own employees, hire a good photographer, and go into your own server room (or beg a friend to let you into theirs) for a photo.

On the subject of image, I’ve just gone to a photographer for some new portraits of myself, and I’m also hiring someone to design a logo for Maatkit (for a new website, and for t-shirts to give away at the upcoming conference). I’ll post more about that later.

Technorati Tags:, , , , , , , , , , ,

No related posts.

I have joined Percona

Effective April 1, I will join Percona full-time as a consultant. I’ll be helping people build high-performance applications with MySQL, but I’ll also be continuing to develop and improve tools such as Maatkit.

This career change has been a long time in progress. I’m really looking forward to it, but at the same time it’s hard to leave my current employer, The Rimm-Kaufman Group (RKG). Working with them has been the best job I’ve ever had. But ultimately, my dream job is to help as many people as I can, and consulting will be a better way to do that.

At a time like this, I like to reflect on the trail that has led here. It’s a good opportunity to realize how fortunate I really am and fill up my gratitude tank. So I’d like to thank everyone who has helped me reach this point. All the people who have encouraged me, sponsored me, suggested new options… all kinds of help. A special thanks to my wife Lynn, to Alan Rimm-Kaufman and all my colleagues at RKG, to the many fine people at MySQL, and to the MySQL community as a whole. My deep gratitude to all of you. I look forward to working with you even more in the future.

Technorati Tags:, , ,

You might also like:

  1. Four companies to sponsor Maatkit development
  2. innotop 1.4.0 released

What’s the best way to choose graph colors?

I have an issue I hope someone can help me with. I am generating RRDtool graphs (for Cacti monitoring templates for MySQL, which I’ll release soon) that have up to 11 different metrics on them. With that many lines or areas on a graph, it becomes very hard to pick colors that are easy to see and easy to distinguish from each other. What’s a good way to choose such colors? Is there a way to do it automatically — is there a formal method that will produce good results?

I know some color theory and I have read about how you can distinguish colors from each other (hue, value etc). But I am unsure the best way to choose this many colors. Trying by hand produces garish results or graphs that are just hard to read.

My first attempt to solve this with a program was to simply create a list of every possible completely saturated color in a 32-bit space — essentially, the “pure” colors around the rim of the color wheel — and divide it into the desired number of evenly spaced intervals. This produces pure colors, which is not ideal. They are hard to look at. Did I mention garish?

I can shuffle the order so that they’re not adjacent, but that only helps avoid a “rainbow effect” if I’m stacking areas of color on top of each other, like in the following image:

MySQL Command Counters

Ugh, rainbows (I chose those by hand, not with my program). Lines on a white background might be placed in any order, so shuffling doesn’t help with those graphs.

I modified my little script to let me vary the saturation and value. My thinking was that lines on a white background really shouldn’t be full-value, and when I’m drawing areas instead of lines, I should de-saturate them so they become more pleasing pastels. This doesn’t really help as much as I might have hoped for, either. Colors around 80% saturation and 60% value look pretty good, but they’re still ugly colors. And I can’t get over five colors without them starting to run together again. Here’s an example with only four colors that’s already hard to look at:

InnoDB I/O Activity

Part of the problem, I’m currently thinking, is that I’m varying only one dimension. I could be varying the saturation as well as the hue, for example. But that might be another rabbit hole that will waste more time.

Right now I’m thinking that I should ask for help, instead of continuing to work on this myself. So, any ideas are welcome!

By the way, beautiful colors would be nice… a lot of the colors I choose by hand are very pretty and I’m sure my impartial, evenly-distributing script will never choose them in a million years. Also, it’s actually a good thing when graphs each have their own color scheme (as long as it’s attractive) because it becomes easier to identify graphs without having to read the title. Just some extra food for thought.

Technorati Tags:, , , , , ,

You might also like:

  1. Improved Cacti monitoring templates for MySQL
  2. innotop 1.3.5 released
  3. How to monitor MySQL status and variables with innotop
  4. Advanced HTML table features, Part 2

High Performance MySQL 2nd Edition is in production

Just a quick note to say we have reached the production stage of the book project. Production is the process of transforming our OpenOffice.org files into the final page layout using a professional typesetting program.

As you can probably guess, this is later than we would have wished. This also means we won’t have the book for sale at the upcoming MySQL Conference and Expo. We will have a display copy at the O’Reilly booth at the conference, and you will be able to pre-order the book at a discount at that booth. (Several details remain to be worked out — do not trust the Amazon.com information on the book, as it is a weird blend of the first and second editions).

The book is very, very good. You will not be disappointed. I can’t think of a credible way to explain how good this book is — it’s just very, very good. Better than anything else you’ve ever read on the subject. So good that you will not want to share, because you’ll want to have your own copy handy for frequent reference (I currently refer to the OpenOffice.org files several times a week myself, and I wrote them!). But I’ll let you see for yourself. Buy a copy for yourself, your boss, your coworkers, and your mom. And your cat.

Technorati Tags:No Tags

You might also like:

  1. A different angle on the MySQL Conference
  2. High Performance MySQL Second Edition Schedule
  3. Progress on High Performance MySQL Backup and Recovery chapter
  4. What are your favorite MySQL replication filtering rules?
  5. Get a free sample chapter of High Performance MySQL Second Edition

Maatkit version 1877 released

Download Maatkit

Maatkit contains essential command-line utilities for MySQL, such as a table checksum tool and query profiler. It provides missing features such as checking slaves for data consistency, with emphasis on quality and scriptability.

This release contains major bug fixes and new features. Some of the changes are not backwards-compatible. It also contains new tools to help you discover replication slaves and move them around the replication hierarchy.

Changelog for mk-archiver:

2008-03-16: version 1.0.8

   * Added --setvars option (bug #1904689, bug #1911371).
   * Added --charset option (bug #1877548).
   * Changed short form of --analyze to -Z to avoid conflict with --charset.

Changelog for mk-deadlock-logger:

2008-03-16: version 1.0.9

   * Added --setvars option (bug #1904689, bug #1911371).
   * Added 'A' part to DSNs (bug #1877548).

Changelog for mk-duplicate-key-checker:

2008-03-16: version 1.1.5

   * Added --setvars option (bug #1904689, bug #1911371).
   * Added --charset option (bug #1877548).

Changelog for mk-find:

2008-03-16: version 0.9.10

   * Added --setvars option (bug #1904689, bug #1911371).
   * Added --charset option (bug #1877548).

Changelog for mk-heartbeat:

2008-03-16: version 1.0.8

   * Added --setvars option (bug #1904689, bug #1911371).
   * Added --charset option (bug #1877548).

Changelog for mk-parallel-dump:

2008-03-16: version 1.0.7

   * Added --setvars option (bug #1904689, bug #1911371).
   * Added --charset option (bug #1877548).
   * A global database connection was re-used by children, causing a hang.

Changelog for mk-parallel-restore:

2008-03-16: version 1.0.6

   * Added --setvars option (bug #1904689, bug #1911371).
   * Changed --charset to be compatible with other tools (bug #1877548).

Changelog for mk-query-profiler:

2008-03-16: version 1.1.9

   * Added --setvars option (bug #1904689, bug #1911371).
   * Added --charset option (bug #1877548).

Changelog for mk-show-grants:

2008-03-16: version 1.0.9

   * Added --setvars option (bug #1904689, bug #1911371).
   * Added --charset option (bug #1877548).

Changelog for mk-slave-delay:

2008-03-16: version 1.0.6

   * Added --setvars option (bug #1904689, bug #1911371).
   * Added 'A' part to DSNs (bug #1877548).

Changelog for mk-slave-find:

2008-03-16: version 1.0.0

   * Initial release.

Changelog for mk-slave-move:

2008-03-16: version 0.9.0

   * Initial release.

Changelog for mk-slave-prefetch:

2008-03-16: version 1.0.1

   * Added --setvars option (bug #1904689, bug #1911371).
   * Added --charset option (bug #1877548).

Changelog for mk-slave-restart:

2008-03-16: version 1.0.6

   * Added --setvars option (bug #1904689, bug #1911371).
   * Added --charset option (bug #1877548).
   * Added logic to repair tables, and rewrote a lot of code.
   * Added --always option, disabled by default.  Not backwards compatible.
   * --daemonize did not work.
   * --quiet caused an undefined variable error.

Changelog for mk-table-checksum:

2008-03-16: version 1.1.26

   * Added --setvars option (bug #1904689, bug #1911371).
   * Added 'A' part to DSNs (bug #1877548).
   * Added --unique option to mk-checksum-filter.
   * The exit status from mk-checksum-filter was always 0.
   * mk-table-checksum now prefers to discover slaves via SHOW PROCESSLIST.

Changelog for mk-table-sync:

2008-03-16: version 1.0.6

   * --chunksize was not being converted to rowcount (bug #1902341).
   * Added --setvars option (bug #1904689, bug #1911371).
   * Deprecated the --utf8 option in favor of the A part in DSNs.
   * Mixed-case identifiers caused case-sensitivity issues (bug #1910276).
   * Prefer SHOW PROCESSLIST when looking for slaves of a server.

Changelog for mk-visual-explain:

2008-03-16: version 1.0.7

   * Added --setvars option (bug #1904689, bug #1911371).
   * Added --charset option (bug #1877548).
Technorati Tags:, ,

You might also like:

  1. Maatkit version 1314 released
  2. Maatkit version 1709 released
  3. Maatkit version 1674 released
  4. Maatkit version 1753 released
  5. Maatkit version 1508 released

How to install and maintain multiple WordPress blogs easily

My wife has a site that needs two WordPress blog installations. The URLs differ by a subdirectory name. Both blogs need to be (URL-wise) subdirectories of /blog/. They need to be completely independent of each other, yet use the same custom theme. And there used to be just a single blog, which was not in a subdirectory; its permalinks must not break. (It has nice URLs with the date and title in them, not post ID-style URLs). And because I’m the husband, I get to maintain it, so tack “easy to maintain” onto the requirements (it must be easy to upgrade WP in both blogs, for example). In this article I’ll show you how I did it with a single .htaccess file, a single copy of WordPress, two MySQL databases, and a single configuration file.

Fixing URLs

As I mentioned, there used to be a blog at /blog/ which must not break. Suppose this blog was about dogs and my wife has recently started blogging about cats. She wants two completely independent blogs: /blog/dogs/ and /blog/cats/. Now the old permalinks structure, e.g. /blog/2006/03/01/dogs-are-great/, must redirect to /blog/dogs/2006/03/01/dogs-are-great/. How to do this?

I’m not a mod_rewrite wizard, but I figured there must be a way. And indeed there is: if an incoming URL doesn’t contain dogs or cats, it can be rewritten and redirected to the new URL. Here’s the code, which goes in /blog/.htaccess:

RewriteBase /blog/
RewriteCond %{REQUEST_URI} !dogs|cats
RewriteRule ^(.*)$ http://www.furryfriends.org/blog/dogs/$1 [R]

(By the way, the furryfriends thing is just an example, not the real site name).

So far, so good. That works just fine: when I access a URL without dogs or cats in it, it redirects me. But I need to do more: I need rewrite rules to match the date-and-title permalinks both blogs will use. I accomplish that like so:

RewriteCond %{REQUEST_URI} dogs|cats
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule (dogs|cats) /blog/$1/index.php [L]

This is basically the same thing WordPress usually does, but I’ve made it tolerate either dogs or cats and figure out which installation should get the request. The .htaccess file lives in /blog/, not inside /dogs/ or /cats/ where it would be hard to maintain (it would get wiped out with upgrades). I can see different ways of doing this, but this is the way I chose. So here’s the whole file:


RewriteEngine On

# Anything to the old address (e.g. /blog/foo/bar) goes to the new address
# (e.g. /blog/dogs/foo/bar)
RewriteBase /blog/
RewriteCond %{REQUEST_URI} !dogs|cats
RewriteRule ^(.*)$ http://www.furryfriends.org/blog/dogs/$1 [R]

# If that fired, then we didn’t reach this code.  If we did, then this rule
# should do what a normal WP rule does.
RewriteCond %{REQUEST_URI} dogs|cats
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule (dogs|cats) /blog/$1/index.php [L]

Are there any better ways of doing this? I’m curious. Leave a comment if you know of one.

Fixing the maintenance headache

Installing two copies of Wordpress, then customizing both is a pain. And it makes upgrades harder, too. I’d have to upgrade them both, fiddle with plugins (some of them are customized, too) etc etc. Even backups would be more complicated. It would be all too easy to screw up and delete some data. There are just so many ways this is a bad idea.

It occurred to me that I could use a single copy and turn the dogs/ and cats/ subdirectories in the filesystem into symbolic links. (Windows users, you can stop reading now: this won’t work for you).

To make the blogs, the Wordpress installation, and the custom blog theme all independent of each other, I created the following filesystem hierarchy:

blog/
   wordpress/
      2.3.2/
         [The usual WP files are here]
      wp_content/
         plugins/
         uploads/
         themes/
            my_custom_theme/

What I’ve done is separate the custom bits — the parts that don’t ship with WordPress — away from the files I want to upgrade when I upgrade Wordpress. How will this work, though?

I’ll make symbolic links from the dogs/ and cats/ directories to the currently installed version of Wordpress. So, from the root directory of the website, I type the following at the command line:

$ ln -s wordpress/2.3.2/ dogs
$ ln -s wordpress/2.3.2/ cats
$ cd wordpress/2.3.2/
$ rm -rf wp-content/
$ ln -s ../wp-content wp-content

The directory hierarchy now looks like this:

blog/
   cats/ -> wordpress/
   dogs/ -> wordpress/
   wordpress/
      2.3.2/
         [The usual WP files are here]
         wp-content/ -> ../wp-content
      wp_content/
         plugins/
         uploads/
         themes/
            my_custom_theme/

This is looking pretty good! There’s only one minor detail missing: because both blogs are running literally the same code via the magic of symlinks, each blog is trying to access the same database tables. I need to customize the Wordpress configuration file, too. I’ll just give each installation a different table name prefix in wp-config.php:

$table_prefix  = strpos($_SERVER['REQUEST_URI'], 'blog/cats/') ? 'wp_cats_' : 'wp_dogs';

And voila, it works perfectly now. I accessed the two URLs, ran through the installation procedure twice, and have two completely independent blogs running the same code in the same database.

The upgrade procedure

So, this is all a little complicated, right? What if I’ve forgotten how I did it when I upgrade next time, or what if someone else does it instead of me? I wrote myself a little README file to fix this. Here’s what it says:

This is how to upgrade Lynn's blog.

The two blogs are actually using shared files, which are symlinked to make
it so there is only one copy of files.  You can't change the files in one
without changing them in the other.

The wp-content subdirectory is symlinked.

The wp-config file is customized so it will work in either blog:

$table_prefix  = strpos($_SERVER['REQUEST_URI'], 'blog/cats/') ? 'wp_cats_' : 'wp_dogs';

To upgrade, 

 1. Download the latest version and unpack it inside wordpress/ as 2.3.2/
    or whatever version it is.
 2. Then go into that directory.
 3. Remove the wp-content/ directory completely.
 4. Then symlink it like this: ln -s ../wp-content wp-content
 5. Now re-customize wp-config.php
 6. Go back to the blog/ directory.  rm dogs cats
 7. ln -s wordpress/2.3.2/ dogs
 8. ln -s wordpress/2.3.2/ cats

It’s still a manual process, but it should take me all of thirty seconds. I’m okay with that. As long as I remember there’s a README file, that is!

Technorati Tags:, , ,

You might also like:

  1. How to write INSERT IF NOT EXISTS queries in standard SQL
  2. How to exploit an insecure order of access to resources
  3. How to install beautiful X11 cursors
  4. How to make file names cross-platform
  5. Interactive directory merging

A very fast FNV hash function for MySQL

I wrote a User-Defined Function that implements the FNV (Fowler-Voll-No) hash function for MySQL. I’m not the first person to do this — in fact, I was inspired by the Google patches for MySQL. But my implementation is a little bit different from most, in a very important way that leads directly to much higher performance, especially suited for the Maatkit tools.

A bit of background: FNV hashing is a very fast hash algorithm that operates in fixed memory. It is widely used in lots of important areas in computer science. My implementation requires absolutely no malloc() calls, which is a darn good thing because I am not to be trusted with malloc(), having spent too many years programming in managed languages. I made it return a 64-bit integer, which matches the size MySQL uses internally for most integer arithmetic.

The most important thing I did was make my UDF accept 1 to infinity arguments. That means you can hash entire rows with a single function call. And that is very useful for the Maatkit table-checksumming tools, which tend to run about 8-10 times faster when they don’t have to make MySQL do a bunch of string concatenation. That translates directly to less impact on the server, and less slave lag (if that is a problem for you).

Here’s how my implementation works:

SELECT FNV_64(col1, col2, col3, .... colN) FROM ...

Compare this to MD5() hashing that accomplishes the same thing:

SELECT MD5(CONCAT_WS('#', col1, col2, col3, .... colN)) FROM ...

The UDF’s code is distributed with Maatkit, and I plan to eventually build it as a binary that can be installed without requiring you to compile it. However, compiling is very easy; there are instructions in the source code comments. Installing is also easy: just a simple SQL statement.

If you’re using Maatkit to make sure your slaves have the same data as their master, you should install the UDF on all your servers for a significant performance boost. You’ll save your servers a lot of work. You don’t need to do anything extra for Maatkit to take advantage of it. Maatkit will auto-detect it if you have it installed.

I’ve been running it in production for a couple of months now with nothing but good results. And the code is drop-dead simple, so I think the chance of bugs is very slim. But if you have questions, problems, bug reports etc, please use the Maatkit project page to report them.

Technorati Tags:, , , ,

You might also like:

  1. Speed up your MySQL replication slaves

Send your employees to the MySQL Conference

A lot of people contact me asking if I’m looking for a job. (I have an unanswered email in my inbox right now.) People are looking desperately for qualified, knowledgeable MySQL professionals. There’s a critical shortage of people who can admin MySQL moderately well, much less at the guru level.

If you are one of the many who are trying to hire a MySQL DBA, you should send your employees to the MySQL Conference and Expo. Not just this year — every year. Train a smart person instead of trying to hire someone who’s ready to go now.

This is the unfortunate reality: MySQL’s popularity has caused demand to far exceed supply. That’s what happens when a great disruptive innovation takes hold.

What do you do in the meantime?

If you just need a little help, hire a part-time DBA and get some consulting help. Without endorsing them directly, may I suggest Percona, Pythian, Proven Scaling or OpenQuery? You can also get support from MySQL, but the barrier to entry is higher because they’re trying to court larger organizations who need more help. But if your needs are large enough, that can make a lot of sense too.

Another thing you can do is send your employees to training, or get someone to come on-site and train. The companies I just mentioned can do this. So can The Learning Tree. (I’ve taken a Learning Tree course on MS SQL server and found it well worth my time, though I don’t know what it cost because my employer paid for it.)

I don’t get any kickbacks for these suggestions, by the way. And everything I say here is my opinion, not facts.

Make sure you approach your smart, motivated employees now — as in, this week — about going to the conference. As Jay Pipes writes, the sessions are great and many of the tutorials are selling out. I can vouch for what he said about standing-room-only crowds. At some of the sessions last year, it was hard to get out of the rooms, much less into them.

Technorati Tags:, , , , , , , ,

You might also like:

  1. My presentations at the 2008 MySQL Conference and Expo
  2. Sessions I want to see at the MySQL Conference
  3. How to get your session accepted to MySQL Conference 2008
  4. Remember to sign up for MySQL Conference and Expo!
  5. MySQL Conference and Expo 2008, Day One