Hindsight on a scalable replacement for InnoDB
A while ago I posted about a comment a Sun performance engineer made about a scalable replacement for InnoDB. At the time, I did not believe it referred to Falcon. In hindsight, it seems even clearer that the Sun performance experts were already working hard on InnoDB itself.
Sun’s engineers have shown that they can produce great results when they really take the problems seriously. And I’m sure that InnoDB’s performance has untapped potential we don’t see right now. However, it does not follow that their work on InnoDB is what was meant by a scalable replacement for InnoDB. Or does it?
General-purpose MVCC transactional storage engines with row-level locking, whatever their performance and scaling characteristics in edge cases, fall into a category together. A person assembling a MySQL server for general-purpose use might choose a different storage engine for various uses — MyISAM here, Memory there… and use “one of those transactional engines” for the bulk of the work. PBXT, InnoDB, Falcon — I don’t see a justification for running more than one of those side by side. The operational costs alone (backups, training the users, etc) would be too high. It is also not at all clear that MySQL itself is ready for multiple transactional storage engines working together (e.g. cross-engine transactions) in the real world.
So what’s left for Falcon? I think they are asking themselves the same question (brilliant gallows humor, by the way). I think Falcon’s ideas and techniques are very interesting, but a storage engine — especially one with such lofty goals — is always a show-me undertaking that will require years to mature and prove itself even after the code is “ready.” With or without the Oracle acquisition, this question has loomed for years: where’s the justification for Falcon politically, functionally, economically? A third party engine such as PBXT, with eyes on replication at the storage engine level and other add-on functionality, has always seemed more likely to really add value than a straight-up InnoDB replacement.
But from my point of view, the biggest win in the short term would still be to drive InnoDB development forward at a consistent and accelerating pace to meet the needs of users and the advances in hardware. Of course, that’s what XtraDB set out to do, and I think the XtraDB project has helped snap InnoDB out of their Percheron-like plod towards improvement. This is nothing but good; when it comes to competition among storage engines, no one should be resting on their laurels. I also see that Sun’s team has more good things in the works, which is great. I’d love for InnoDB to stop being a work horse and start being a quarter horse. We need it to be both scalable and high-performance.
I had a good chat with Ann at the MySQL Expo and Conference last month.
I think Falcon can deliver something special which can be useful for an important niche of users when they have resolved their scalability and stability issues.
I simply hope that my discussion with Ann will bear fruit and at least help make Falcon stable.
Antony Curtis
9 May 09 at 9:45 pm
Yeah, I haven’t talked to any of the Falcon team for a while, and of course it’s not like I’m close friends with any of them — so I don’t mean to sound like I’m the authority on it. I have just never understood what’s going on with the project, or what options are really viable for it. I probably shouldn’t spend much time thinking about any of this until the acquisition deal stabilizes a little anyway.
If someone who knows wants to write a brief and frank assessment of what really differentiates Falcon and what needs it will really fill, without any marketing pitch or fluff, I would find it very helpful.
Xaprb
9 May 09 at 10:34 pm
I would also like to know what it will do to improve on InnoDB. An ETA for when it will do this would also be good. That it has a different implementation than InnoDB is not interesting to me.
Mark Callaghan
9 May 09 at 11:59 pm
in a nutshell, the use case for which Falcon is targeting is for large numbers of small transactions, where typically with Innodb, the bottleneck would be the commit queue where committed transactions must have all row updates written to disk.
Falcon is different insofar as a transaction is considered committed when the commit log entry is written to disk and the in-memory record data is committed. It will lazily update the disk version of the row data safely in separate worker threads.
Theoretically, the design tries to provide a trade-off between the traditional random-access engines such as InnoDB and compliment it with fast commit speeds as in log-structured engines by taking advantage of plentiful RAM.
Of course, for a more accurate and complete description of Falcon, you’d have to ask the Falcon Team, Ann Harrison or Jim Starkey.
Antony Curtis
10 May 09 at 2:11 am
I doubt that Falcon provides for faster commit speeds. Innodb uses a WAL. Commit only requires that entries are written to the transaction log on disk.
Mark Callaghan
10 May 09 at 10:13 am
If innoDB isn’t using a separate process for row data commits then Falcon will be faster because it won’t have to wait for all data to commit to disk – a very slow process. Of course I don’t see why innoDB couldn’t do the same.
Mike Stevens
11 May 09 at 10:45 am