Why no TPC benchmarks for MySQL?
Someone asked me yesterday why there are no TPC benchmarks for MySQL. By “benchmarks” I don’t mean “benchmark tools” but rather “official benchmark results.” For example, see the current TPC-C benchmarks. Any true TPC benchmark has to be certified by the TPC and is an arduous and expensive process. Is it time for MySQL to play in this theater? Or does it simply not matter?
And it is ESPENSIVE! I don’t think TPC matters. I think they are an out of date standard. You can’t expect to get their results with your code and your hardware will not match theirs. (to many BIOS and patch level)
I do think it’s time to discuss what kinds of standards we should keep. Is MySQL really getting faster or is it just the hardware? (one systems many version of MySQL)
Mark Grennan
7 Mar 12 at 5:57 pm
Could it be that MySQL performance isn’t good enough to be competitive? Almost all results are published by hardware vendors. MySQL doesn’t have a DeWitt-clause, so any hardware vendor is free to use it. I doubt there’s a lack of interest. And there’s an obvious price advantage to using MySQL. And yet, the hardware companies predominately choose to use SQL Server for TPC benchmarks (and Oracle and DB2 to a much lesser extent). By Occam’s razor, the answer is that MySQL isn’t fast enough to win TPC benchmarks.
Charles Levine
8 Mar 12 at 6:38 pm
After trying to run TPC-H benchmarks on MySQL, it seems MySQL isn’t quite ready. Some of the queries wouldn’t even run as written (this is after fixing minor syntactic differences). I recall one query running through the optimizer for a couple hours before even starting execution.
Michael Mior
13 Mar 12 at 11:31 pm
Right, MySQL isn’t at all qualified for TPC-H. Percona ran TPC-H for Kickfire back in the days when they wanted external validation and testing. It ran well on Kickfire (though that was essentially the only thing that would run at all on it) but of course it would never complete on vanilla MySQL.
Xaprb
14 Mar 12 at 7:14 am