Comments on: What’s a good benchmark? http://www.xaprb.com/blog/2011/05/10/whats-a-good-benchmark/ Stay curious! Thu, 02 May 2013 12:36:53 +0000 hourly 1 http://wordpress.org/?v=3.5.1 By: Xaprb http://www.xaprb.com/blog/2011/05/10/whats-a-good-benchmark/#comment-19365 Xaprb Thu, 12 May 2011 09:58:57 +0000 http://www.xaprb.com/blog/?p=2316#comment-19365 Jeff,

I think “what is the maximum capacity of two very different systems” is a useful and valid question to ask. Not everything has to be apples-to-apples.

]]> By: Morgan TOcker http://www.xaprb.com/blog/2011/05/10/whats-a-good-benchmark/#comment-19363 Morgan TOcker Thu, 12 May 2011 02:48:35 +0000 http://www.xaprb.com/blog/?p=2316#comment-19363 One of my favourite comments on benchmarking comes from Larry McVoy:

http://lwn.net/Articles/368908/

]]> By: Jeff Kibler http://www.xaprb.com/blog/2011/05/10/whats-a-good-benchmark/#comment-19361 Jeff Kibler Wed, 11 May 2011 19:56:33 +0000 http://www.xaprb.com/blog/?p=2316#comment-19361 Hi Baron,

When it comes to benchmarking, do you believe it is important that structure between the two alternatives be exactly the same -or- the setup be optimized for the respective application? I’d like to get your opinion.

I would argue that each system should be calibrated to its optimal performance (assuming you know what optimal for each application is). As long as the raw dataset is the exact same, the organization/structure/layout of the data within the application can be different.

Some tend to argue that benchmarks should consist entirely of controls and exactly 1 test.

Thanks,

Jeff

]]>