Xaprb » replication

Blackhole tables and auto-increment keys

Xaprb — Wed, 19 Oct 2011 21:41:32 +0000

Blackhole tables are often used on a so-called “relay slave” where some operation needs to happen but no data needs to exist. This used to have a bug that prevented AUTO_INCREMENT columns from propagating the right values through replication, but that was fixed. It turns out there’s another bug, though, that has the same effect. This one is caused when there is an INSERT into a Blackhole table, where the source data is SELECT-ed from another Blackhole table.

I think it’s wise to keep it simple. MySQL has tons of cool little features that theoretically suit edge-case uses and make ninja tricks possible, but I really trust the core plain-Jane functionality so much more than these edge-case features. That’s precisely because they often have some edge-case bugs, especially with replication.

Something that’s new to MySQL recently is Galera replication. The more I think about it, the more I think it’s fundamentally the right way to replicate. Statement-based replication was brittle; row-based is less so, but still has all kinds of gotchas. The real problem with both is that they are built into the server, not the storage engine. Engine-level replication is the way to go. PBXT has had engine-level replication for a while, although I’ve never used PBXT in production (and kudos to PostgreSQL for adding built-in replication, too). I used to want InnoDB to do replication via streaming the redo logs and applying them, but that actually has a lot of limitations. Galera is InnoDB’s answer to engine-level replication. I think Galera holds a lot of promise for the future.

Further Reading:

Observations on key-value databases

Xaprb — Sun, 20 Sep 2009 18:57:01 +0000

Key-value databases are catching fire these days. Memcached, Redis, Cassandra, Keyspace, Tokyo Tyrant, and a handful of others are surging in popularity, judging by the contents of my feed reader.

I find a number of things interesting about these tools.

There are many more of them than open-source traditional relational databases. (edit: I mean that there are many options that all seem similar to each other, instead of 3 or 4 standing out as the giants.)
It seems that a lot of people are simultaneously inventing solutions to their problems in private without being aware of each other, then open-sourcing the results. That points to a sudden sea change in architectures. Tipping points tend to be abrupt, which would explain isolated redundant development.
Many of the products are feature-rich with things programmers need: diverse language bindings, APIs, embeddability, and the ability to speak familiar protocols such as memcached protocol.
I think there are more solutions here than the ecosystem will support, and in five years a few will stand out as the most popular.
This process of paring down the gene pool is win-win because they’re open-source, and nothing will be lost.
Choosing which one to use is no easy task even for a highly skilled, technical, up-to-date person. Perhaps the decision-makers will choose on the availability of commercial support and consulting.
Many of them offer built-in, dead-simple, distributed, synchronous replication. This is very difficult to achieve with traditional relational databases. What makes key-value databases different? They don’t have MVCC, for one thing; but I’m not sure of the complete answer to that question, to tell the truth.

We live in interesting times.

Further Reading:

Failure scenarios and solutions in master-master replication

Xaprb — Sun, 30 Aug 2009 19:08:20 +0000

I’ve been thinking recently about the failure scenarios of MySQL replication clusters, such as master-master pairs or master-master-with-slaves. There are a few tools that are designed to help manage failover and load balancing in such clusters, by moving virtual IP addresses around. The ones I’m familiar with don’t always do the right thing when an irregularity is detected. I’ve been debating what the best way to do replication clustering with automatic failover really is.

I’d like to hear your thoughts on the following question: what types of scenarios require what kind of response from such a tool?

I can think of a number of failures. Let me give just a few simple examples in a master-master pair:

Problem: Query overload on the writable master makes mysqld unresponsive: Do nothing. Moving the queries to another server will cause cascading failures.
Problem: The writable master is completely unreachable: Fence the writable master and promote the standby master.
Problem: The writable master is reachable but unresponsive due to overload-induced swapping: Do nothing. Moving the load to another server will cause cascading failures.

I don’t want to bias the jury, so I’ll stop there and ask you to contribute your failure scenarios and what you think the correct action should be.

Further Reading:

Don’t forget about SHOW PROFILES

Xaprb — Sun, 31 May 2009 19:24:54 +0000

It seems that a lot of people want to try to improve MySQL performance by focusing on server status counters and configuration variables. Looking at counters, and “tuning the server,” is better than nothing, but only barely. You care first and foremost about how long it takes to execute a query, not about how many of this-and-that the server performs or about how big or small this-and-that buffer is. What you really need is timing information.

You can use the slow query log to find timing information about queries, and then you can examine those queries with SHOW PROFILES to see the timing information about the query’s execution itself.

This concept is very simple and absolutely fundamental: if you care about time (and you do!), then measure and optimize time. But it’s so often overlooked or misunderstood.

The addition of SHOW PROFILES was a major step forward in the ability to optimize server and application performance. (Thanks Jeremy Cole!) As time passes and people upgrade their servers, it’s becoming more common to see it in production, which is an enormous help. Now that the differences between the Community and Enterprise versions of the server have been erased, it will be available in all future server versions, which is great news.

Further Reading:

MySQL replication breaks single-threaded limitation?

Xaprb — Fri, 17 Apr 2009 16:59:59 +0000

It’s a feature preview with many limitations, but this is still good news. This has been a pretty severe performance limitation for replication in MySQL, which has prompted many a workaround.

Interestingly, the feature preview is based on MySQL 5.1, which has recently seemed to be getting some significant changes even though it’s a GA release. Does this signal a change to MySQL’s release cycle, which has sometimes been characterized as too long? More good news?

Further Reading: