What is the scalable replacement for InnoDB?
A while back a Sun engineer posted an article claiming that the best way to scale MySQL is to shard your database in many instances on a single server, each of which runs in threads that individually have low performance. The Sun way has always been to get high throughput with high latency. And that’s fine. Others have commented on the real-world applicability of this technique with MySQL, so I won’t.
But what I’m interested in is something the author says in the comments of his own blog post:
btw; we are working on a scalable replacement of InnoDB. Stay tuned….
What is it?
Surely it’s not Falcon. MySQL and Sun have said many times Falcon isn’t meant to be an InnoDB replacement.
The next question: how should we interpret the word “scalable” given the context? Clearly there’s a difference of opinion between MrBenchmark and others on what that word means. So what will the scaling characteristics of this InnoDB replacement really be?
The following is a joke, not meant to be taken seriously: Maybe it will be MyISAM with a fix to the key buffer scalability problems. Start with your database with 50 tables. Make 1 million databases, each with the same 50 tables, and you can scale up to 1 million rows by putting 1 row in each table. Ta-da! Row-level locking with MyISAM!
But seriously, what is it?
Baron,
What if Sun will use XtraDB ?
VadimTk
13 Jan 09 at 3:31 am
It is “more scalable” but is it really scalable yet? Does it scale linearly to 256 cores?
Xaprb
13 Jan 09 at 10:36 am
Maybe it’s a stoarge engine based on ZFS..
Jeremy Zawodny
13 Jan 09 at 11:16 am
Maybe they were referring to Maria, the crash safe optionally transactional successor to MyIsam engine that Monty was working on?
Either that, or they really are considering falcon to be an innodb replacement, despite their history of stating that wasn’t the case. I really hope they aren’t working on another engine. They haven’t shown much ability to finish Falcon, or Maria. I fear adding another one to the mix would dilute their efforts even further.
William
13 Jan 09 at 12:20 pm
I realize the *official* line from Sun is that Falcon is not meant as an InnoDB replacement, however that is by far the most likely engine that user was talking about. What we have long known is that Sun has many engineers working on Falcon, which is going to replace InnoDB as the default transactional engine for 6.0.
Thus is is no surprise to me that in casual non-official communication such as comments to weblogs, sometimes the official company word can be blurred a bit by those engineers. They probably care more about the technology and less about the spin.
The other alternative is that Sun has given up on Falcon *already*, and is secretly working on a new new transactional engine. If true they have done this without showing any sign in public bug tracker, or comments to Enterprise customers who hav been told not to expect 5.1 schedule delays with 6.0. This alternative seems extraordinarily unlikely to be.
Ryan Thiessen
13 Jan 09 at 12:51 pm
PostgreSQL.
Alex Staubo
13 Jan 09 at 1:46 pm
Another possibility … seriously … is that the “scalable replacement of InnoDB” will be … wait for it … InnoDB!
Although we understand many people are skeptical, we have stated our commitment to enhancing the performance and scalability (and functionality) of InnoDB. And we’ve done work to deliver on that commitment. We intend for InnoDB to remain the leading transactional storage engine for MySQL for years to come. It will be its own “replacement”, We hope the skeptics will be convinced by our actions.
We have already delivered several improvements in the area of performance and scalability. Shortly after
Oracle’s acquisition of Innobase, we released a significant scalability improvement for autoincrement.
At the MySQL User Conference in April 2008, we introduced the InnoDB Plugin, with significant performance improvements in the form of “Fast Index Creation” and data compression (which can improve performance for many, but not all, sets of data and workloads).
We have stated elsewhere our willingness to incorporate third party contributions, under the right circumstances (technical quality/reliability, fit with architectural/design plans and appropriate legal terms). We also said that we have been working with key players like Google and Percona on their proposed patches for improvements. We’ve found (and fixed) some issues in those patches.
It certainly would be nice if we could release performance improvements to you faster. Like any organization, our resources are limited (and some have to be devoted to maintenance/bug fixing). We are (or have been) constrained by the release schedule for MySQL (and the long period when the software was in “Release Candidate” status).
Our focus of development is the InnoDB Plugin. It allows us to release software independently of MySQL’s releases (well, except for a tight version-number compatibility check). Because the new development has gone into the InnoDB Plugin rather than into MySQL 6.0, MySQL users can take advantage of the new functionality now in the 5.1 context.
We released the latest version (1.0.3) last month. We will be releasing new software “soon”. All I can say now is “stay tuned”. You may be (pleasantly) surprised!
Ken Jacobs
13 Jan 09 at 8:53 pm
Ken,
You said
“key players like Google and Percona on their proposed patches for improvements. We’ve found (and fixed) some issues in those patches.”
Of course our patches are not bug-free. But this is a place where we see power of OpenSource. You can download and just look source code. And report bugs to our system. Unfortunately I did not see any bugreports, so I even can’t comment what issues you found.
VadimTk
13 Jan 09 at 10:52 pm
Firstly, many of the folks using MySQL don’t *need* a scalable replacement, as they are running it fine on their own (or with the help of companies like Pythian, Percona and Proven Scaling).
Secondly, it depends what you’re using it for, and what you mean by scalable. For some applications, the scalable replacement for InnoDB is MySQL Cluster. If you want something that uses multiple processors, or is feature-heavy, the replacement might be Oracle or Postgres. If you’re a minimalist and an experimentalist, Drizzle.
So, really….it’s very difficult unless you define your terms, and remember that what you see is a very, very small percentage of the MySQL InnoDB users out there.
Sheeri Cabral
13 Jan 09 at 11:11 pm
Sheeri,
“What is scalable” is one of the primary questions I had in the blog post. Who defines scalable? Again I don’t think MrBenchmark talks the same language the rest of us on this thread so far talk. But it’s not just him. Look at the video on the front page of http://www.sun.com/systems/solutions/mysql/ and notice what Mikael Ronstrom says about Cluster:
“MySQL has been focusing its efforts on dual-core systems for a long time, and it’s only recently that we’ve seen this massive development of multi-core systems in the world. So customers care very much about scalability, I mean they, they buy systems with 16 cores and they expect to be able to use all those 16 cores.”
OK, so far so good, although I take exception to “only recently.” But then, they cut the video and immediately show him saying
“So we have actually achieved linear scalability of MySQL Cluster…”
Those two have nothing to do with each other. And, right now Cluster is single-threaded, so it’s *less* scalable — you can use one core per instance, period. (I know that is supposed to change, but… I’ll save my predictions about that for later). Any way you look at it, it’s a bait and switch.
So, my rhetorical question remains: by what criterion does this wait-and-see storage engine scale?
Xaprb
13 Jan 09 at 11:59 pm
Sorry, Vadim, we’ve already fixed the issues we found in the (Google, not Percona) patch. The people who know InnoDB best are the InnoDB group at Innobase. We respect what you guys do, but there is not much value (to us, admittedly) in showing you what those issues were at this point.
This isn’t about the “power of open source”. It’s about doing what’s right for InnoDB users. That includes making sure any patches we do include in InnoDB are completely solid and reliable. The best way to do that is for the InnoDB team to be careful and thoroughly test and evaluate potential solutions to possible problems.
Again, stay tuned.
Ken Jacobs
14 Jan 09 at 12:19 am
Yes, InnoDB has provided valuable feedback on the rw-mutex patch and not yet published changes that reduce mutex contention on the transaction log mutex and buffer cache mutex. Valuable == fixed a few bugs, improved code quality and portability. They also have published 5.1 as a plugin so that it is easy for anyone to publish updated versions of it separate from the MySql release cycle.
Mark Callaghan
14 Jan 09 at 2:40 pm
[...] and a couple questions pertaining to InnoDB. The first comes from xaprb’s Baron Schwartz: What is the scalable replacement for InnoDB? — a question provoked by a statement by a Sun engineer that Sun is working on this [...]
Log Buffer #131: a Carnival of the Vanities for DBAs
16 Jan 09 at 1:21 pm
[...] while ago I posted about a comment a Sun performance engineer made about a scalable replacement for InnoDB. At the time, I did not believe it referred to Falcon. In hindsight, it seems even clearer that the [...]
Hindsight on a scalable replacement for InnoDB at Xaprb
9 May 09 at 4:40 pm