Comments on: How to implement a queue in SQL http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/ Stay curious! Thu, 24 Jul 2008 00:19:51 +0000 http://wordpress.org/?v=2.2.2 By: Rdb Notes · How to implement a queue in SQL at Xaprb http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-13353 Rdb Notes · How to implement a queue in SQL at Xaprb Thu, 30 Aug 2007 15:03:28 +0000 http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-13353 [...] How to implement a queue in SQL at Xaprb This article explains how to create a fixed-size FIFO (first-in, first-out) queue in SQL, where rows added after a threshold will cause the oldest row to be deleted. There are several ways to do this, but MERGE on Oracle and DB2, and MySQL’s non-standard extensions to SQL, make an elegant solution easy. [...] […] How to implement a queue in SQL at Xaprb This article explains how to create a fixed-size FIFO (first-in, first-out) queue in SQL, where rows added after a threshold will cause the oldest row to be deleted. There are several ways to do this, but MERGE on Oracle and DB2, and MySQL’s non-standard extensions to SQL, make an elegant solution easy. […]

]]> By: Anish http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-12506 Anish Tue, 24 Jul 2007 06:23:26 +0000 http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-12506 Great article! How do I ensure unique value in fruit column, say no other rows should have 'peaches' if it is already in queue Great article! How do I ensure unique value in fruit column, say no other rows should have ‘peaches’ if it is already in queue

]]> By: Xaprb http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-5410 Xaprb Fri, 30 Mar 2007 13:19:45 +0000 http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-5410 <p>It might work to use a calculation like <code>TO_DAYS(CURRENT_DATE) % 60</code>, or some variation thereof.</p> It might work to use a calculation like TO_DAYS(CURRENT_DATE) % 60, or some variation thereof.

]]> By: Ivo http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-5401 Ivo Fri, 30 Mar 2007 08:27:31 +0000 http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-5401 <p>What about using a certain timeframe for the fifo mechanism. E.g. when the timestamp between the insert of a new item is more than 60 days later than the timestamp of the first item, it replaces the first item. What would this mean to the insert query?</p> What about using a certain timeframe for the fifo mechanism. E.g. when the timestamp between the insert of a new item is more than 60 days later than the timestamp of the first item, it replaces the first item.

What would this mean to the insert query?

]]> By: Xaprb http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-4797 Xaprb Mon, 05 Mar 2007 15:25:28 +0000 http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-4797 <p>Note, this idea of a 'queue' is actually pretty similar to a round-robin database (RRD). Most of you are probably familiar with <a href="http://oss.oetiker.ch/rrdtool/" rel="nofollow">RRDTool</a>. Here's another article on the topic: <a href="http://www.shinguz.ch/MySQL/mysql_20070223.html" rel="nofollow">Round-Robin Database</a> in MySQL.</p> Note, this idea of a ‘queue’ is actually pretty similar to a round-robin database (RRD). Most of you are probably familiar with RRDTool. Here’s another article on the topic: Round-Robin Database in MySQL.

]]> By: Xaprb http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-3925 Xaprb Tue, 06 Feb 2007 13:10:10 +0000 http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-3925 <p>Thanks for mentioning that BIGINT and UNSIGNED. I assume that in 29 million years, MySQL will have a HUGEINT type, too ;-)</p> Thanks for mentioning that BIGINT and UNSIGNED. I assume that in 29 million years, MySQL will have a HUGEINT type, too ;-)

]]> By: Anonymous http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-3910 Anonymous Mon, 05 Feb 2007 20:15:00 +0000 http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-3910 <p>Another edge case -- reaching the limit of the id field.</p> <p>If your application does 100 of these per second, an UNSIGNED INT will last 1.36 years -- not so great. You should use an UNSIGNED INT because you're never going to have a negative number, and a signed int would only last just over 8 months if there were 100 REPLACEs per second.</p> <p>(60 seconds/min, 60 min/hr, 24 hrs/day, 365 days/yr) 4294967295/60/60/24/365/100=1.3619251950</p> <p>So, for scaling/known high traffic, use a BIGINT. However, in this case, do NOT use UNSIGNED, as all MySQL arithmetic is done with signed bigints or doubles. Not that it matters in this case; at 100 REPLACEs per second, you will wrap at 2.9 billion years:</p> <p>9223372036854775807/60/60/24/365/100=2,924,712,086.7753601074</p> <p>Let's say your system does 10,000 of these REPLACEs per second, for eternity (our main database system, where we're about to use this, average 6,000 qps, not all writes, but it's a good figure to use for our own numbers) -- move the decimal places a few spots over and you're down to running out of numbers in 29 million years.</p> <p>That's an OK limit for us. :)</p> Another edge case — reaching the limit of the id field.

If your application does 100 of these per second, an UNSIGNED INT will last 1.36 years — not so great. You should use an UNSIGNED INT because you’re never going to have a negative number, and a signed int would only last just over 8 months if there were 100 REPLACEs per second.

(60 seconds/min, 60 min/hr, 24 hrs/day, 365 days/yr)
4294967295/60/60/24/365/100=1.3619251950

So, for scaling/known high traffic, use a BIGINT. However, in this case, do NOT use UNSIGNED, as all MySQL arithmetic is done with signed bigints or doubles. Not that it matters in this case; at 100 REPLACEs per second, you will wrap at 2.9 billion years:

9223372036854775807/60/60/24/365/100=2,924,712,086.7753601074

Let’s say your system does 10,000 of these REPLACEs per second, for eternity (our main database system, where we’re about to use this, average 6,000 qps, not all writes, but it’s a good figure to use for our own numbers) — move the decimal places a few spots over and you’re down to running out of numbers in 29 million years.

That’s an OK limit for us. :)

]]> By: Xaprb http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-3558 Xaprb Mon, 29 Jan 2007 12:43:34 +0000 http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-3558 <p>If I were worried that people would query the table wrong, I'd restrict access via a stored procedure instead. But in real life, there is no such thing as a table that only allows you to insert things the right way. You can always insert something that violates business rules or makes something break in an application of any complexity.</p> If I were worried that people would query the table wrong, I’d restrict access via a stored procedure instead. But in real life, there is no such thing as a table that only allows you to insert things the right way. You can always insert something that violates business rules or makes something break in an application of any complexity.

]]> By: Thanos http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-3556 Thanos Mon, 29 Jan 2007 09:52:45 +0000 http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-3556 <p>A nice article. I think the most interesting use case for a query like this would be to implement a RRD/MRTG kind of database, where (network performance) data comes in every x seconds, and you only need a weeks worth of data. I don't know though why people say it is an "elegant" solution. A really elegant solution would not change the way the database is accessed. Why should I change the way I insert things in a database? I should just make standard queries and the database should take care of the rest, as happens with RRDtool. If more than one person is interfacing with a database like this, what is the probability that somebody forgets/gets this query wrong and messes up things? I guess mysql was not made for stuff like this. I nice blog nevertheless!!</p> A nice article. I think the most interesting use case for a query like this would be to implement a RRD/MRTG kind of database, where (network performance) data comes in every x seconds, and you only need a weeks worth of data.

I don’t know though why people say it is an “elegant” solution. A really elegant solution would not change the way the database is accessed. Why should I change the way I insert things in a database? I should just make standard queries and the database should take care of the rest, as happens with RRDtool.

If more than one person is interfacing with a database like this, what is the probability that somebody forgets/gets this query wrong and messes up things? I guess mysql was not made for stuff like this.

I nice blog nevertheless!!

]]> By: Eric Ryan Harrison http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-3199 Eric Ryan Harrison Tue, 16 Jan 2007 14:39:12 +0000 http://www.xaprb.com/blog/2007/01/11/how-to-implement-a-queue-in-sql/#comment-3199 <p> As the initial commenter who requested this article to be written, I see that you guys are having a hard time understanding the actual reason that we want this solution. </p> <p> We are using this solution to track and store query times for our users for each application we provide (mostly large database query applications). We are storing the ten most recent query times and then displaying the average query time (for each tool) when a query is made. We are doing this just to give the user a heads up regarding how long the queries are taking. Hence why we want to have a circular table like this. </p> <p> Originally, we WERE merely doing a dual insert/delete operation, but I posted the question here to Baron to see if MySQL had a more efficient way to do it in one line (which he beautifully provided). </p> <p> Does that clear this up for you guys? </p> <p> -E </p> As the initial commenter who requested this article to be written, I see that you guys are having a hard time understanding the actual reason that we want this solution.

We are using this solution to track and store query times for our users for each application we provide (mostly large database query applications). We are storing the ten most recent query times and then displaying the average query time (for each tool) when a query is made. We are doing this just to give the user a heads up regarding how long the queries are taking. Hence why we want to have a circular table like this.

Originally, we WERE merely doing a dual insert/delete operation, but I posted the question here to Baron to see if MySQL had a more efficient way to do it in one line (which he beautifully provided).

Does that clear this up for you guys?

-E

]]>