My DBA had a really hard time believing my solution in the first email in this thread should be faster. So after digging some, he ran an ANALYZE on the tables in question.

This helps the PostgreSQL planner out to create more optimal query plans.

It turns out that after analyze the original query that uses an “IN” takes 1ms.

So we went from:
IN Clause: ~4500ms
Inner Join: ~300ms
Analyze with IN Clause: ~1ms

Holy cow.

The lesson is: use explain, use analyze, and consult your DBAs. They are your friends.

In our case, we went from ~4500ms for the query to ~300ms

An order of magnitude improvement!

Thanks for the great post!!

search 1000 recs via:

IN: 270ms
OR: 270ms
JOIN: 200ms (excluded INSERT time)

BASH script:

#!/bin/bash

M=1000
SQLB="SELECT AVG(id) FROM x ";
s=$[1000000/$M]
v=0

#
# IN test
#
SQL="$SQLB WHERE id IN("
for((i=1;i<$M;i++)); do
  SQL="$SQL$v,"
  v=$[$v+$s]
done

SQL="$SQL$v);"
time /usr/bin/psql -d test -c "$SQL"

#
# OR test
#
v=0
SQL="$SQLB WHERE "

for((i=1;i<$M;i++)); do
  SQL="$SQL id=$v OR"
  v=$[$v+$s]
done

SQL=$SQL" id=$v;"
time /usr/bin/psql -d test -c "$SQL"

#
# JOIN table test
#
v=0
SQL="DROP TABLE tx;CREATE TABLE tx (idFK INTEGER);"
SQL=$SQL"BEGIN; "

for((i=1;i<$M;i++)); do
  SQL=$SQL"INSERT INTO tx VALUES($v);"
  v=$[$v+$s]
done

SQL=$SQL"INSERT INTO tx VALUES($v);COMMIT;"
psql -d test -c "$SQL"
SQL="$SQLB INNER JOIN tx ON tx.idFK=x.id"
time /usr/bin/psql -d test -c "$SQL"

Roberto Colmegna

ERROR 1064 (42000): parser stack overflow near '21053740 ) as aids on a.id = aids.aid
  JOIN kategorie k JOIN hersteller h JOIN ' at line 5334
Segmentation fault

Now, I couldn’t reproduce the segfault, but the parse-error occurred reliably.

My mysql-version is 5.0.22.

It would be great if you can post an example of the kind of join you mention.

I love your posts. However, as others have pointed out, you’re offbase wrt INs and mysql. I’ll go one further and say that I have quite often been able to optimize painful joins away with INs at my consulting clients. However, my other finding is that the gain is reversely proportional to the clue coefficient of the engineers who designed the db schema ;]

Some of my experience is with Microsoft SQL Server, which definitely, at least in SQL Server 2000, did NOT optimize IN well. I also realize now that I didn’t write the article that clearly, distinguishing between all the different types of things that can go into an IN. There can be lists of numbers, a subselect, lists of columns, and maybe more. Subselects are not that well optimized in MySQL, as discussed ad nauseum elsewhere. But it looks like lists of numbers are very well optimized, which is great.