Stay Curious!

How to generate sequences and surrogate keys in generic SQL

This article demonstrates how to “roll your own” surrogate keys and sequences in a platform-independent way, using standard SQL.

Surrogate keys

Relational theory talks about something called a “candidate key.” In SQL terms, a candidate key is any combination of columns that uniquely identifies a row (SQL and the relational model aren’t the same thing, but I’ll put that aside for this article). The data’s primary key is the minimal candidate key. Many people think a primary key is something the DBA defines, but that’s not true. The primary key is a property of the data, not the table that holds the data.

Unfortunately, the minimal candidate key is sometimes not a good primary key in the real world. For example, if the primary key is 6 columns wide and I need to refer to a row from another table, it’s impractical to make a 6-column wide foreign key. For this reason, database designers sometimes introduce a surrogate key, which uniquely identifies every row in the table and is “more minimal” than the inherently unique aspect of the data. The usual choice is a monotonically increasing integer, which is small and easy to use in foreign keys.

Every RDBMS of which I’m aware offers a feature to make surrogate keys easier by automatically generating the next larger value upon insert. In SQL Server, it’s called an IDENTITY column. In MySQL, it’s called AUTO_INCREMENT. It’s possible to generate the value in SQL, but it’s easier and generally safer to let the RDBMS do it instead. This does lead to some issues itself, such as the need to find out the value that was generated by the last insertion, but those are usually not hard to solve (LAST_INSERT_ID() and similar functions, for example).

It’s sometimes desirable not to use the provided feature. For instance, I might want to be sure I always use the next available number. In that case, I can’t use the built-in features, because they don’t generate the next available number under some circumstances. For example, SQL Server doesn’t decrement the internal counter when transactions are rolled back, leaving holes in the data (see my article on finding missing numbers in a sequence). Neither MySQL nor SQL Server decrements the counter when rows are deleted.

In these cases, it’s possible to generate the next value in the insert statement. Suppose my table looks like this:

create table t1 (
    c1 int not null primary key,
    c2 int not null
);

The next value for c1 is simply the maximum value + 1. If there is no maximum value, it is 1, which is the same as 0 + 1.

select 1 + coalesce(max(c1), 0) as next
from t1;

There are platform-dependent ways to write that statement as well, such as using SQL Server’s ISNULL function or MySQL’s IFNULL. This code can be combined into an INSERT statement, such as the following statement to insert 3 into the second column:

insert into t1 (c1, c2)
    select 1 + coalesce(max(c1), 0), 3
    from t1;

The code above is a single atomic statement and will prevent any two concurrent inserts from getting the same value for c1. It is not safe to find the next value in one statement and use it in another, unless both statements are in a transaction. I would consider that a bad idea, though. There’s no need for a transaction in the statement above.

Downsides to this approach are inability to find the value of c1 immediately after inserting, and inability to insert multiple rows at once. The first problem is inherently caused by inserting meaningless data, and is always a problem, even with the built-in surrogate keys where the RDBMS provides a mechanism to retrieve the value.

Sequences: a better surrogate key

Surrogate keys are often considered very bad practice, for a variety of good reasons I won’t discuss here. Sometimes, though, there is just nothing for it but to artificially unique-ify the data. In these cases, a sequence number can often be a less evil approach. A sequence is just a surrogate key that restarts at 1 for each group of related records. For example, consider a table of log entries related to records in my t1 table:

create table t1log(
    t1 int not null references t1(c1),
    seq int not null,
    message varchar(50) not null,
    primary key(t1, seq)
);

At this point I might want to enter some more records (0, 11) into t1:

insert into t1 (c1, c2)
    select 1 + coalesce(max(c1), 0), 0 from t1;
insert into t1 (c1, c2)
    select 1 + coalesce(max(c1), 0), 11 from t1;
select * from t1;
+----+----+
| c1 | c2 |
+----+----+
| 1  | 3  |
| 2  | 0  |
| 3  | 11 |
+----+----+

Now suppose I want the following three log entries for the first row in t1:

+----+------------------+
| t1 | message          |
+----+------------------+
| 1  | Sent to customer |
| 1  | Received reply   |
| 1  | Responded        |
+----+------------------+

There’s no good primary key in this data. I will have to add a surrogate key. It might seem I could add a date-time column instead, but that’s a dangerous design. It breaks as soon as two records are inserted within a timespan less than the maximum resolution of the data type. It also breaks if two records are inserted in a single transaction where the time is consistent from the first to the last statement. I’m much happier with a sequence column. The following statement will insert the log records as desired:

insert into t1log(t1, seq,message)
    select 1, 1 + coalesce(max(seq), 0), "Sent to customer"
    from t1log
    where t1 = 1;
insert into t1log(t1, seq,message)
    select 1, 1 + coalesce(max(seq), 0), "Received reply"
    from t1log
    where t1 = 1;
insert into t1log(t1, seq,message)
    select 1, 1 + coalesce(max(seq), 0), "Responded"
    from t1log
    where t1 = 1;
 select * from t1log;
+----+-----+------------------+
| t1 | seq | message          |
+----+-----+------------------+
| 1  | 1   | Sent to customer |
| 1  | 2   | Received reply   |
| 1  | 3   | Responded        |
+----+-----+------------------+

If I want to enter a log record on another record in t1, the sequence will start at 1 for it:

insert into t1log(t1, seq,message)
    select 11, 1 + coalesce(max(seq), 0), "Deleted"
    from t1log
    where t1 = 11;
 select * from t1log;
+----+-----+------------------+
| t1 | seq | message          |
+----+-----+------------------+
| 1  | 1   | Sent to customer |
| 1  | 2   | Received reply   |
| 1  | 3   | Responded        |
| 11 | 1   | Deleted          |
+----+-----+------------------+

MySQL actually allows an AUTO_INCREMENT value to serve as a sequence for certain table types (MyISAM and BDB). To do tihs, just make the column the last column in a multi-column primary key. I’m not aware of any other RDBMS that does this.

Posted on Thu, Apr 20, 2006. Approximately 1200 Words.

Databases