More details about SchoonerSQL performance, please!
Schooner has a blog post showing that one node of their product beats 9 nodes of Clustrix’s in throughput. But this reduces everything to a single number, and that’s not everything that matters. If you’ve looked at Vadim’s white paper about Clustrix’s (paid-for) performance evaluation with Percona, you see there is a lot of detail about how consistent the throughput and response time are.
I’d love to see that level of details in any product comparison. A single number often isn’t enough to judge how good the performance is — fast is not the only thing that matters.
I have absolutely no doubts that a single node of Schooner’s product can run like a deer. It isn’t doing any cross-node communication, after all, so it had better be faster than something that blends multiple nodes together into a virtual “single database server.” And I think if the full story were told, it would be a great knock-down drag-out fight. Give us more details, Schooner!
Further Reading:
It should include all the necessary information needed to repeat the tests independently.
My $.02
G
Gerry
2 Feb 12 at 5:20 pm
As I said, I have no doubt that SchoonerSQL is going to beat Clustrix in a raw performance comparison, node for node. I don’t dispute the results, I just want to know more about them.
Xaprb
2 Feb 12 at 6:15 pm
Baron,
At the same time Schooner has published the benchmark
with the result where async MySQL replication is slower than semi-sync MySQL replication.
So there is doubt about credibility of their methodology.
VadimTk
2 Feb 12 at 7:41 pm
There’s not enough information to doubt :)
Xaprb
2 Feb 12 at 9:31 pm
Hello Vadim,
Long time no chat :)
We measured MySQL v5.5 and shared results at the April 2011 MySQL User Conference comparing async, semi-sync and Schooner-sync replication between two identical nodes in Master/Slave configuration. For MySQL 5.5 async and semi-sync runs, we ensured slave lag ~< 1 sec by tweaking DBT2 think-time. If this were not done, Slave can get hours behind (slave lag) for every hour of benchmark run.
I believe you are referring to the results in the slides: http://www.slideshare.net/darpandinker/performance-comparisons-and-trade-offs-for-various-my-sql-replication-schemes-presentation
Notice the choppy throughput with async Master and Slave vs more stable throughput with semi-sync for MySQL 5.5. I was hoping you have similar runs and can explain it if you see the same.
If you think MySQL 5.5 async vs semi-sync data doesn't look right, we can run it again for you and share the results. Given the same benchmark conditions, I would still expect to see 4X throughput gains with SchoonerSQL in reruns.
Darpan Dinker
3 Feb 12 at 2:50 am
Hello Baron,
No question Percona did a thorough job writing the white-paper for Clustrix. Having done performance benchmarking, I can surely understand the time it would have taken to execute all those runs, validating data (rerunning benchmarks if necessary) and then plotting all the results.
I think its easy to get carried away with the title of Pavan’s blog. 9 Clustrix nodes with 48GB DRAM each = 432GB, incidentally the size of the 5000 warehouse tpcc-mysql database. The core question is if the scalability of Clustrix efficient? I picked one data-point to pursue that question. To make that point, I blogged about it here: http://darpanetwork.blogspot.com/2012/01/how-scalable-is-clustrix-scalability.html
and then put in tpcc-mysql details:
http://darpanetwork.blogspot.com/2012/01/schoonersql-throughput-with-perconas.html
BTW I did want to ensure stable environment and results, so I did have a 4 hour long run.
If you provide a cheat-sheet, in addition to the CPU util chart already posted, I will add throughput and latency charts for the run.
So let me ask this: lets say you validated my results. Will you ask the question: for what workloads and price/performance will a Clustrix or similar building-block (with a fraction of throughput capability) be better suited for?
If someone pays me, I will love to do 128 runs for comparison – any takers? ;-)
Darpan Dinker
3 Feb 12 at 3:23 am
I remember some years back that SolidDB (now discontinued) published a benchmark ‘proving’ that it handled many times more transactions per second than InnoDB. It could not be verified independently unless a specific testcase was used.
So I can only agree that all details (configurations, data, queries/scripts) should be available. As long as this is not the case any postulate is as good as another.
Peter Laursen
3 Feb 12 at 4:49 am
Peter,
Good point.
Let’s see if I can move you along this process:
Doubt -> Verify -> Trust.
Darpan
Darpan Dinker
3 Feb 12 at 10:29 am
Working with Baron and Vadim (thanks), I was able to decipher the output of tpcc-mysql (it can be definitely improved :) ).
I have updated my blog with a throughput charts – for folks who want to look at the stability of the same SchoonerSQL run:
http://darpanetwork.blogspot.com/2012/01/schoonersql-throughput-with-perconas.html
Yes, I will add more details after understanding the output some more with Vadim’s help.
Darpan Dinker
3 Feb 12 at 6:07 pm