Comments on: Would you trust a more advanced MySQL optimizer?

By: Xaprb

Xaprb — Thu, 23 Feb 2012 13:33:51 +0000

The union and intersection operations are usually not expensive in my experience, although sometimes they can be. The problem sometimes comes afterwards, where we get a lot of random I/O to look up the rows that were selected from the indexes.

MySQL 5.6 is making important improvements in this area too.

By: Jaimie Sirovich

Jaimie Sirovich — Fri, 30 Dec 2011 05:26:55 +0000

@Xaprb:

You said “it is a fair guess that there are no good indexes for the query, and it’s making the best of a bad situation.”

What you say regarding index merge might be true if you are running say, a few types of queries that all could use a few indices to accelerate lookups. However, you’re leaving out the ad-hoc queries. In that case you’d need a ridiculous amount of composite queries (a subset of n!). So saying “your indexes are wrong” is not quite fair.

The cost model may be screwy, and I’m not sure how exactly an index merge in MySQL works, but I’d guess it builds N bitmap indices as it goes through each index, then does some cheap bit arithmetic to compute the final answer. If the table type is fixed length in MyISAM that’s some basic arithmetic. That could be pretty fast, and it’s why I wish MySQL had bitmap and bitmap join indices :)

But yeah, if you’re asking a database something ad hoc based on a subset of any or all 10 criteria, you couldn’t possibly catch all the cases with composite indices. In some cases, 1 good almost unique index (what b-trees do best) + a table scan would be great — but in some other cases an index merge would be much much better, esp. if you had a few indexes with so-so cardinality. Of course using bitmap indices itself might be better in that case.

Of course if the cost model is screwy, it will make the wrong decisions. I agree.

Am I missing something? Is this how index merge works? It’s the only way I’ve seen index merges implemented in the wild.

I’ve learned a lot more from your blog than you could learn from me, so if the above is wrong, please feel free to correct me :)

By: Xaprb

Xaprb — Fri, 02 Dec 2011 14:11:46 +0000

I think my opinions in April 2010 were influenced a lot by what was going on in the larger MySQL world at that time, and the initially buggy 5.1 releases in 2008-2009. I feel a lot better about the improvements Oracle is putting into MySQL 5.6 (and 5.5) now than I would have a couple of years ago.

By: Erik Jones

Erik Jones — Fri, 02 Dec 2011 03:47:16 +0000

I think that the way you’ve phrased the question is akin to asking if we’d be down to swap out the little piglet that is the current optimizer with some other whole hog. There are a number of improvements that could be made incrementally such as including more information about how the optimizer will actually carry out it’s query plan (nested loop you say?), flattening subqueries where possible, and something akin to Postgres’s EXPLAIN ANALYZE. Two of those three examples I just listed support the fact that I’d be a lot more likely to “trust” a “smarter” optimizer if I had a better view into what it actually does rather than replacing the current black box (when compared to other database planners out there) with another, bigger & fancier black box. You’re right about the index_merge “feature” actually being a bad idea in most cases; but the biggest problem isn’t that it’s there, it’s that we aren’t given enough information by EXPLAIN to know that without just running the query to see what happens.

I’ve spent the last 3.5 years banging my head against MySQL servers. I’ve got a lot more respect for it now than I did at the beginning of that stretch (largely due to the work you guys @ Percona have put into it and Oracle’s “let’s fix & improve what we have and slow down with the tacking on more new features onto existing broken features) but I still consider MySQL’s optimizer to be one of the server’s biggest weaknesses when viewed in the context of the greater RDBMS ecosystem so thanks for bringing this up and giving me the chance to chime in.

By: Xaprb

Xaprb — Fri, 23 Apr 2010 14:50:37 +0000

I’m aware that I’m getting in over my head here, but I think that for index_merge to work well, the optimizer needs a much more sophisticated cost model. It’s not really a bug per se, just an overly simplistic view of the world. “All models are wrong, some models are useful.” Statistics aren’t the problem; the optimizer assumes that buffering and merging the results found from multiple passes through indexes is free. In fact, broadly speaking, the optimizer assumes that *everything* is free except for a disk read, and that’s not true.