As Xaprb said, your second insert is slow because you are inserting in a random order. Before loading, sort your data in primary key order and the two inserts should run in similar time.

If insert speed is your priority, I’d stick with the first table design. But now that you’ve got the data into the second table, test how fast the queries run against it. You’ve just artificially unique-ified a non-unique value, and it might be a good trade-off.

Use SHOW TABLE STATUS to see how big the table and index data is. Compare the two designs. The second will probably be quite a bit bigger.

I picked up on clustered indexes from your “case study in profiling queries in MySQL”.

Then I read the manual. Then I came up with this idea. Instead of:

CREATE TABLE impressions (
  id int NOT NULL auto_increment,
  epoch int NOT NULL,
  count int NOT NULL,
  PRIMARY KEY (id),
  INDEX epoch_idx (epoch)
) ENGINE=InnoDB;

I could instead do:

CREATE TABLE impressions (
  id int NOT NULL auto_increment,
  epoch int NOT NULL,
  count int NOT NULL,
  PRIMARY KEY (epoch, id),
  UNIQUE INDEX id_idx (id)
) ENGINE=InnoDB;

I need the id column to join with other tables, and there can be multiple rows with the same epoch, so epoch cannot be unique.

The vast majority of my queries are operating on a subset of the impressions table and the vast majority of those use epoch to pick the subset.

I then created some sample data using the following perl:

my $time = time;
for (my $i = 1; $i < 250000 + 1; $i++) {
    my $r1 = $time - int(rand(100000));
    my $r2 = int(rand(10000));
    print "$i\t$r1\t$r2\n";
}

And then used:

LOAD DATA INFILE 'sample.sql' INTO TABLE impressions;

Note that I’m using exactly the same data on both tables. On my Macbook Pro, the first table loaded in about 6.19 second. The second took 70.16 seconds. Worse yet, as the number of rows increased, the times for the second table appeared to be growing exponentially!

My current working hypothesis is that the extra size used for the clustered index in the second table is causing the problems. Aka your “exceptions to the rule.” But I’m frankly not sure and a little curious. If I shifted the epoch to be seconds since 1st Jan 2000 would that help? Or if I carefully managed my insert process so that I appended a dot followed by an counter to multiple rows with the same epoch - would that help?

How big is too big for the primary key in an InnoDB table?