Comments on: Kickfire: stream-processing SQL queries http://www.xaprb.com/blog/2008/04/04/kickfire-stream-processing-sql-queries/ Stay curious! Thu, 24 Jul 2008 00:20:23 +0000 http://wordpress.org/?v=2.2.2 By: zillablog http://www.xaprb.com/blog/2008/04/04/kickfire-stream-processing-sql-queries/#comment-14437 zillablog Tue, 15 Apr 2008 04:34:10 +0000 http://www.xaprb.com/blog/2008/04/04/kickfire-stream-processing-sql-queries/#comment-14437 <strong>Where there's smoke, there's Kickfire...</strong> Over the weekend I had a chance to catch up on some of my favorite database oriented blogs, and I noticed a number of them were mentioning the upcoming Kickfire engine/appliance based around MySQL. Always a sucker to read about Yet Another MySQL Engine... Where there’s smoke, there’s Kickfire…

Over the weekend I had a chance to catch up on some of my favorite database oriented blogs, and I noticed a number of them were mentioning the upcoming Kickfire engine/appliance based around MySQL. Always a sucker to read about Yet Another MySQL Engine…

]]> By: Dan http://www.xaprb.com/blog/2008/04/04/kickfire-stream-processing-sql-queries/#comment-14429 Dan Sun, 13 Apr 2008 12:30:15 +0000 http://www.xaprb.com/blog/2008/04/04/kickfire-stream-processing-sql-queries/#comment-14429 I suspect along with their own "stream processor" there will be at least one other general purpose processor in the device. Looking at other industries where the same sort of thing is done (again, going with the graphics analogies that are being spoken about at the moment), stream processors all on their own are non-existent to my knowledge. Most database engines that specialize in OLAP will use column-stored data. Kickfire sounds like it analyses queries with a general-purpose processor which also co-ordinates standard file handling/locking, figures out which column(s) it will need access to, initiate the data transfer from disk to the steam processor, and then push the data direct to the dedicated processor, letting it work it's highly-optimised magic on the data that comes down the pipe (I'm guessing basic match/discard/logic operations). It sounds like a pretty simple concept, honestly, and one I'm surprised nobody has implemented to date. Data store will be the final bottleneck, and even then if you can get a disk to push 250MB/s (with something like SAN/fibrechannel), you wouldn't need a stream processor to be all that fast (speaking in clockspeed) to deal with the data being pushed into it. Again, think realtime video or audio effects on data streams, only SQL functions instead of pixel shaders or audio distortions. Again, I'm surprised it hasn't been done before. Particularly given the money big vendors like Oracle and IBM sink into their DB development every year. I suspect along with their own “stream processor” there will be at least one other general purpose processor in the device. Looking at other industries where the same sort of thing is done (again, going with the graphics analogies that are being spoken about at the moment), stream processors all on their own are non-existent to my knowledge.

Most database engines that specialize in OLAP will use column-stored data. Kickfire sounds like it analyses queries with a general-purpose processor which also co-ordinates standard file handling/locking, figures out which column(s) it will need access to, initiate the data transfer from disk to the steam processor, and then push the data direct to the dedicated processor, letting it work it’s highly-optimised magic on the data that comes down the pipe (I’m guessing basic match/discard/logic operations).

It sounds like a pretty simple concept, honestly, and one I’m surprised nobody has implemented to date. Data store will be the final bottleneck, and even then if you can get a disk to push 250MB/s (with something like SAN/fibrechannel), you wouldn’t need a stream processor to be all that fast (speaking in clockspeed) to deal with the data being pushed into it. Again, think realtime video or audio effects on data streams, only SQL functions instead of pixel shaders or audio distortions.

Again, I’m surprised it hasn’t been done before. Particularly given the money big vendors like Oracle and IBM sink into their DB development every year.

]]> By: Nathaniel http://www.xaprb.com/blog/2008/04/04/kickfire-stream-processing-sql-queries/#comment-14415 Nathaniel Tue, 08 Apr 2008 04:21:34 +0000 http://www.xaprb.com/blog/2008/04/04/kickfire-stream-processing-sql-queries/#comment-14415 Good article, even for the non-SQL savvy people out here. I like how you keep pointing out the environmental impacts of computing. Many of us feel that working with computers is one profession that is immune to affecting the environment. Unfortunately it is expensive and wasteful - we just don't see it all. I guess I'm feeling a bit sore lately because my flatscreen monitor just went bad and will cost as much to fix as to buy a replacement. I digress... Good article, even for the non-SQL savvy people out here. I like how you keep pointing out the environmental impacts of computing. Many of us feel that working with computers is one profession that is immune to affecting the environment. Unfortunately it is expensive and wasteful - we just don’t see it all. I guess I’m feeling a bit sore lately because my flatscreen monitor just went bad and will cost as much to fix as to buy a replacement. I digress…

]]> By: Xaprb http://www.xaprb.com/blog/2008/04/04/kickfire-stream-processing-sql-queries/#comment-14408 Xaprb Sat, 05 Apr 2008 13:31:30 +0000 http://www.xaprb.com/blog/2008/04/04/kickfire-stream-processing-sql-queries/#comment-14408 vsmi, sure. I will be expanding on this in an upcoming post. I have a crude understanding of how they've really implemented this stream-processing in hardware, but I should know for real in a week. I'm refraining from writing more until then :-) vsmi, sure. I will be expanding on this in an upcoming post. I have a crude understanding of how they’ve really implemented this stream-processing in hardware, but I should know for real in a week. I’m refraining from writing more until then :-)

]]> By: vsmi http://www.xaprb.com/blog/2008/04/04/kickfire-stream-processing-sql-queries/#comment-14407 vsmi Fri, 04 Apr 2008 21:33:06 +0000 http://www.xaprb.com/blog/2008/04/04/kickfire-stream-processing-sql-queries/#comment-14407 I might be wrong, but this sounds similar SIMD, and very close to how I know most DSPs work. And yes, this is kind of hardware that functional languages are designed for. I might be wrong, but this sounds similar SIMD, and very close to how I know most DSPs work. And yes, this is kind of hardware that functional languages are designed for.

]]> By: Xaprb http://www.xaprb.com/blog/2008/04/04/kickfire-stream-processing-sql-queries/#comment-14406 Xaprb Fri, 04 Apr 2008 18:33:45 +0000 http://www.xaprb.com/blog/2008/04/04/kickfire-stream-processing-sql-queries/#comment-14406 It is supposed to excel at this type of query, but how it does that is another question entirely. I'll definitely be asking this. It is supposed to excel at this type of query, but how it does that is another question entirely. I’ll definitely be asking this.

]]> By: Dewey http://www.xaprb.com/blog/2008/04/04/kickfire-stream-processing-sql-queries/#comment-14405 Dewey Fri, 04 Apr 2008 17:39:05 +0000 http://www.xaprb.com/blog/2008/04/04/kickfire-stream-processing-sql-queries/#comment-14405 Baron....this is way over my head, but I'm curious if it would also have advantages during correlated sub queries?? In that case, you don't know what row (or even table if there is case stmt branching) of data you need until you get there and so that data can't be "pushed" in advance (unless they push all tables (in total?) through the bus as the default)......would you ask them about that? Even if they tried to move that much data, seems like there would be timing issues regarding when rows needed together, pass through at different times... Very interesting stuff.. Thanks---Dewey Baron….this is way over my head, but I’m curious if it would also have advantages during correlated sub queries?? In that case, you don’t know what row (or even table if there is case stmt branching) of data you need until you get there and so that data can’t be “pushed” in advance (unless they push all tables (in total?) through the bus as the default)……would you ask them about that? Even if they tried to move that much data, seems like there would be timing issues regarding when rows needed together, pass through at different times…

Very interesting stuff..
Thanks—Dewey

]]> By: Xaprb http://www.xaprb.com/blog/2008/04/04/kickfire-stream-processing-sql-queries/#comment-14404 Xaprb Fri, 04 Apr 2008 14:29:02 +0000 http://www.xaprb.com/blog/2008/04/04/kickfire-stream-processing-sql-queries/#comment-14404 Honestly I studied the Cell processor in university, but I've forgotten everything I learned. And I have not gotten back into chip architecture since then so I'm very rusty. But yes, that Wikipedia article is about the same concept, <strong>at least as much as Kickfire has explained to me so far</strong>. The article also points out that stream processing isn't as suited for small amounts of data, which is why I said I'm curious about overall performance. Does it perform well for SELECT * FROM TBL WHERE id=500? I have no idea yet. Honestly I studied the Cell processor in university, but I’ve forgotten everything I learned. And I have not gotten back into chip architecture since then so I’m very rusty. But yes, that Wikipedia article is about the same concept, at least as much as Kickfire has explained to me so far.

The article also points out that stream processing isn’t as suited for small amounts of data, which is why I said I’m curious about overall performance. Does it perform well for SELECT * FROM TBL WHERE id=500? I have no idea yet.

]]> By: Johan http://www.xaprb.com/blog/2008/04/04/kickfire-stream-processing-sql-queries/#comment-14403 Johan Fri, 04 Apr 2008 14:11:29 +0000 http://www.xaprb.com/blog/2008/04/04/kickfire-stream-processing-sql-queries/#comment-14403 This makes me think of Cell, with it's "stream processors" Specifically mentioned on http://en.wikipedia.org/wiki/Stream_processing Is this similar? :) This makes me think of Cell, with it’s “stream processors”
Specifically mentioned on http://en.wikipedia.org/wiki/Stream_processing

Is this similar? :)

]]>