More details about SchoonerSQL performance, please!
Schooner has a blog post showing that one node of their product beats 9 nodes of Clustrix’s in throughput. But this reduces everything to a single number, and that’s not everything that matters. If you’ve looked at Vadim’s white paper about Clustrix’s (paid-for) performance evaluation with Percona, you see there is a lot of detail about how consistent the throughput and response time are.
I’d love to see that level of details in any product comparison. A single number often isn’t enough to judge how good the performance is — fast is not the only thing that matters.
I have absolutely no doubts that a single node of Schooner’s product can run like a deer. It isn’t doing any cross-node communication, after all, so it had better be faster than something that blends multiple nodes together into a virtual “single database server.” And I think if the full story were told, it would be a great knock-down drag-out fight. Give us more details, Schooner!
It should include all the necessary information needed to repeat the tests independently.
My $.02
G
Gerry
2 Feb 12 at 5:20 pm
As I said, I have no doubt that SchoonerSQL is going to beat Clustrix in a raw performance comparison, node for node. I don’t dispute the results, I just want to know more about them.
Xaprb
2 Feb 12 at 6:15 pm
Baron,
At the same time Schooner has published the benchmark
with the result where async MySQL replication is slower than semi-sync MySQL replication.
So there is doubt about credibility of their methodology.
VadimTk
2 Feb 12 at 7:41 pm
There’s not enough information to doubt :)
Xaprb
2 Feb 12 at 9:31 pm
Hello Vadim,
Long time no chat :)
We measured MySQL v5.5 and shared results at the April 2011 MySQL User Conference comparing async, semi-sync and Schooner-sync replication between two identical nodes in Master/Slave configuration. For MySQL 5.5 async and semi-sync runs, we ensured slave lag ~< 1 sec by tweaking DBT2 think-time. If this were not done, Slave can get hours behind (slave lag) for every hour of benchmark run.
I believe you are referring to the results in the slides: http://www.slideshare.net/darpandinker/performance-comparisons-and-trade-offs-for-various-my-sql-replication-schemes-presentation
Notice the choppy throughput with async Master and Slave vs more stable throughput with semi-sync for MySQL 5.5. I was hoping you have similar runs and can explain it if you see the same.
If you think MySQL 5.5 async vs semi-sync data doesn't look right, we can run it again for you and share the results. Given the same benchmark conditions, I would still expect to see 4X throughput gains with SchoonerSQL in reruns.
Darpan Dinker
3 Feb 12 at 2:50 am
Hello Baron,
No question Percona did a thorough job writing the white-paper for Clustrix. Having done performance benchmarking, I can surely understand the time it would have taken to execute all those runs, validating data (rerunning benchmarks if necessary) and then plotting all the results.
I think its easy to get carried away with the title of Pavan’s blog. 9 Clustrix nodes with 48GB DRAM each = 432GB, incidentally the size of the 5000 warehouse tpcc-mysql database. The core question is if the scalability of Clustrix efficient? I picked one data-point to pursue that question. To make that point, I blogged about it here: http://darpanetwork.blogspot.com/2012/01/how-scalable-is-clustrix-scalability.html
and then put in tpcc-mysql details:
http://darpanetwork.blogspot.com/2012/01/schoonersql-throughput-with-perconas.html
BTW I did want to ensure stable environment and results, so I did have a 4 hour long run.
If you provide a cheat-sheet, in addition to the CPU util chart already posted, I will add throughput and latency charts for the run.
So let me ask this: lets say you validated my results. Will you ask the question: for what workloads and price/performance will a Clustrix or similar building-block (with a fraction of throughput capability) be better suited for?
If someone pays me, I will love to do 128 runs for comparison – any takers? ;-)
Darpan Dinker
3 Feb 12 at 3:23 am
I remember some years back that SolidDB (now discontinued) published a benchmark ‘proving’ that it handled many times more transactions per second than InnoDB. It could not be verified independently unless a specific testcase was used.
So I can only agree that all details (configurations, data, queries/scripts) should be available. As long as this is not the case any postulate is as good as another.
Peter Laursen
3 Feb 12 at 4:49 am
Peter,
Good point.
Let’s see if I can move you along this process:
Doubt -> Verify -> Trust.
Darpan
Darpan Dinker
3 Feb 12 at 10:29 am
Working with Baron and Vadim (thanks), I was able to decipher the output of tpcc-mysql (it can be definitely improved :) ).
I have updated my blog with a throughput charts – for folks who want to look at the stability of the same SchoonerSQL run:
http://darpanetwork.blogspot.com/2012/01/schoonersql-throughput-with-perconas.html
Yes, I will add more details after understanding the output some more with Vadim’s help.
Darpan Dinker
3 Feb 12 at 6:07 pm
Baron,
Here is a report that you requested. Please let me know if you’d like to see other details.
http://www.schoonerinfotech.com/downloads/benchmarks/Schooner_Benchmark_SchoonerSQL-Percona-TPCC-MySQL_details.pdf
Best regards,
Darpan
PS You wrote “I have absolutely no doubts that a single node of Schooner’s product can run like a deer”.
I’d rather like my product be called a deer (as opposed to a pig).
Thanks ;-)
Darpan Dinker
15 Mar 12 at 7:58 pm
I’m glad that Schooner published full results that anyone can reproduce. Now the door is open for people to say the comparison was apples-to-oranges because Schooner’s system has no redundancy (single node) and Clustrix’s had 2x redundancy. But that is another topic, and at least that conversation can be legitimate now.
Xaprb
15 Mar 12 at 8:21 pm