Here are two publicly available cases that appear to demonstrate this phenomenon: Robert Haas’s recent blog post on PostgreSQL, titled Scalability, in Graphical Form, Analyzed and Mikael Ronstrom’s post from May on MySQL (NDB) Cluster, titled Better than Linear Scaling is Possible.

Dr. Ronstrom’s post discusses the mechanics of the phenomenon, and speculates (I’m not sure it’s conclusive) that it is from a combination of partitioning and better use of CPU caches. Now someone needs to do the math to figure out how to include this factor into the equation.

The good thing about the Universal Scalability Law is how simple and applicable it is for many systems. It’s nice that this economy-of-scale factor seems to be unusual and the simpler model remains easy to apply for a large variety of tasks.

Further Reading:

• Products that scale linearly to hundreds of servers
• Response-time optimization in systems that are queued
• How to scale writes with master-master replication in MySQL
• Subtleties in the Universal Scalability Law
• A review of Guerrilla Capacity Planning by Neil Gunther

Instrumentation can be a lot of work to add to a server, and it can add overhead to the server too. The bits of instrumentation I’ll advocate in this post are few and trivial, but disproportionately powerful.

If all server software shipped with these metrics as the basic starting point, it would change the world forever:

1. Time elapsed, in high resolution (preferably microseconds; milliseconds is okay; one-second is mostly useless). When I ask for this counter, it simply tells me either the time of day, or the server’s uptime, or something like that. It can be used to determine the boundaries of an observation interval, defined by two measurements. It needs to be consistent with the other metrics that I’ll explain next.
2. The number of queries (statements) that have completed.
3. The current number of queries being executed.
4. The total execution time of all queries, including the in-progress time of currently executing queries, in high resolution. That is, if two queries executed with 1 second of response time each, the result is 2 seconds, no matter whether the queries executed concurrently or serially. If one query started executing .5 seconds ago and is still executing, it should contribute .5 second to the counter.
5. The server’s total busy time, in high resolution. This is different from the previous point in that it only shows the portion of the observation interval during which queries were executing, regardless of whether they were concurrent or not. If two queries with 1-second response time executed serially, the counter is 2. If they executed concurrently, the counter is something less than 2, because the overlapping time isn’t double-counted.

In practice, these can be maintained as follows, in pseudo-code:

global timestamp;
global concurrency;
global busytime;
global totaltime;
global queries;

function run_query() {
  local now = time();
  if ( concurrency ) {
    busytime += now - timestamp;
    totaltime += (now - timestamp) * concurrency;
  }
  concurrency++;
  timestamp = now;

  // Execute the query, and when it completes...

  now = time();
  busytime += now - timestamp;
  totaltime += (now - timestamp) * concurrency;
  concurrency--;
  timestamp = now;
  queries++;
}

I may have missed something there; I’m writing this off the cuff. If I’ve messed up, let me know and I’ll fix it. In any case, these metrics can be used to derive all sorts of powerful things through applications of Little’s Law and queueing theory, as well as providing the inputs to the Universal Scalability Law. They should be reported by simply reading from the variables marked as “global” above, to provide a consistent view of the metrics.

Further Reading:

• 4 ways that instrumentation is like sex
• Subtleties in the Universal Scalability Law
• Using BASE instead of ACID for scalability
• High Performance MySQL, Second Edition: Query Performance Optimization
• A review of PostgreSQL 9.0 High Performance by Gregory Smith

I see that Selena has already announced that there’ll be a 2012 event, which is great. I intend to support it, and I’ve already marked the date on my calendar.

A few people asked me what instrumentation to support scalability and performance analysis would be valuable inside PostgreSQL. The answer I gave in my talk was somewhat sidestepping the question, in hindsight, and I agreed afterwards to follow up with a blog post about it. I’ll write that when I have a chance.

Further Reading:

• I’ll be presenting at Postgres Open 2011
• Postgres folks, consider the 2011 MySQL conference
• Awesome Postgres/MySQL cross-pollination
• mk-query-digest now supports Postgres logs
• Open-source database developer mailing lists

Further Reading:

• Postgres folks, consider the 2011 MySQL conference
• Well done, Postgres Open
• O’Reilly MySQL 2011 conference CfP is open
• My sessions at the O’Reilly MySQL Conference 2011
• I’m a Postgres user, as it turns out

PostgreSQL 9 Administration Cookbook. By Simon Riggs and Hannu Krosing, Packt, 2010. Approximately 330 pages. (Link to publisher’s site.)

This is a good book for PostgreSQL database administrators to pick up, especially if you’re new to PostgreSQL but familiar with another system such as Oracle, SQL Server, or MySQL. It has more than 80 “recipes” that range from quick tips to moderately detailed discussion of how to accomplish specific tasks. The chapters and recipes are well organized, and you can either read the book from start to finish or jump to a specific recipe for quick help.

The major topics are introductory/overview, configuration, managing tables and data, security, managing the server itself, monitoring, maintenance, performance, backup and recovery, replication, and upgrades. The chapter list mimics that list pretty well, though I lumped some chapters together in my topic list.

Some of the same topics are covered in much greater detail in Greg Smith’s excellent PostgreSQL 9 High Performance, which I reviewed previously.

On the negative side, I can only remark that the cookbook format in general isn’t my favorite; each “recipe” is quite formulaic, with little sub-headings titled “getting ready, how to do it, how it works, there’s more, see also.” But it works pretty well nonetheless as a quick-reference guide.

Overall worth picking up, unless you’re quite knowledgeable about PostgreSQL already, in which case I wouldn’t expect you to learn much from this book.

Further Reading:

• A review of PostgreSQL 9.0 High Performance by Gregory Smith
• An invitation to PostgreSQL community members
• Going to PostgreSQL Conference East
• What are your favorite PostgreSQL performance resources?
• PostgreSQL Conference East 2009, Day Two