Comments on: How to re-index a large database table http://www.xaprb.com/blog/2006/06/14/how-to-re-index-a-large-database-table/ Stay curious! Sun, 20 Jul 2008 22:43:23 +0000 http://wordpress.org/?v=2.2.2 By: Xaprb http://www.xaprb.com/blog/2006/06/14/how-to-re-index-a-large-database-table/#comment-1327 Xaprb Thu, 03 Aug 2006 19:48:21 +0000 http://www.xaprb.com/blog/2006/06/14/how-to-re-index-a-large-database-table/#comment-1327 <p>The reason the bulk inserts were taking days was <em>probably</em> that the rows weren't inserted in primary key order. It may have gone much faster if they had been. I'm convinced it's still better to do it a bit at a time though.</p> The reason the bulk inserts were taking days was probably that the rows weren’t inserted in primary key order. It may have gone much faster if they had been. I’m convinced it’s still better to do it a bit at a time though.

]]> By: Xaprb http://www.xaprb.com/blog/2006/06/14/how-to-re-index-a-large-database-table/#comment-1236 Xaprb Wed, 26 Jul 2006 13:09:34 +0000 http://www.xaprb.com/blog/2006/06/14/how-to-re-index-a-large-database-table/#comment-1236 <p>Siri,</p> <p>I don't think the update/insert should take a terribly long time on that data size, if you're only doing it once a day, but you may want to use <code>REPLACE</code> or <code>INSERT... ON DUPLICATE KEY UPDATE</code> instead. Deleting and re-inserting is hard on the server. If you must delete, at least try to use <code>TRUNCATE</code> instead; it has much lower cost. I wrote an article on <a href="http://www.xaprb.com/blog/2006/02/21/flexible-insert-and-update-in-mysql/">MySQL's various statements for inserting and updating data in different ways</a>.</p> Siri,

I don’t think the update/insert should take a terribly long time on that data size, if you’re only doing it once a day, but you may want to use REPLACE or INSERT... ON DUPLICATE KEY UPDATE instead. Deleting and re-inserting is hard on the server. If you must delete, at least try to use TRUNCATE instead; it has much lower cost. I wrote an article on MySQL’s various statements for inserting and updating data in different ways.

]]> By: Siri http://www.xaprb.com/blog/2006/06/14/how-to-re-index-a-large-database-table/#comment-1218 Siri Tue, 25 Jul 2006 11:15:28 +0000 http://www.xaprb.com/blog/2006/06/14/how-to-re-index-a-large-database-table/#comment-1218 <p>I have a issue which seems to have some relation with your analysis</p> <p>1)I have a fact table of size 950MB</p> <p>2)I refresh it daily by droping the indexes and truncating the table and then loading the whole data again</p> <p>Now i need to do the same refresh with out truncating the table. To be more specific i need to insert/update the required records while keeping the old data in the table itself. Here i am confused as the table is very huge my update or insert is gonna take very long. Can you please help me out with any suggestions?</p> <p>Thanks</p> <p>Siri</p> I have a issue which seems to have some relation with your analysis

1)I have a fact table of size 950MB

2)I refresh it daily by droping the indexes and truncating the table and then loading the whole data again

Now i need to do the same refresh with out truncating the table. To be more specific i need to insert/update the required records while keeping the old data in the table itself. Here i am confused as the table is very huge my update or insert is gonna take very long. Can you please help me out with any suggestions?

Thanks

Siri

]]> By: Xaprb http://www.xaprb.com/blog/2006/06/14/how-to-re-index-a-large-database-table/#comment-992 Xaprb Wed, 28 Jun 2006 16:04:10 +0000 http://www.xaprb.com/blog/2006/06/14/how-to-re-index-a-large-database-table/#comment-992 <p>That is in the neighborhood of the data sizes we had too, so the method we used could work for you. Please feel free to add more comments as your work proceeds! I'm curious to know how it goes. Best wishes.</p> That is in the neighborhood of the data sizes we had too, so the method we used could work for you. Please feel free to add more comments as your work proceeds! I’m curious to know how it goes. Best wishes.

]]> By: David Vickers http://www.xaprb.com/blog/2006/06/14/how-to-re-index-a-large-database-table/#comment-991 David Vickers Wed, 28 Jun 2006 12:44:57 +0000 http://www.xaprb.com/blog/2006/06/14/how-to-re-index-a-large-database-table/#comment-991 <p>This sounds very similar to our system. The larger files were up to 10Gig in size, which we've reduced by making the main column smaller. We are now in the re-indexing stages, but will eventually have to reduce the 220Gig files, by removing the older data, deleting it etc. Your idea seems sound. We probably will produce a new table, and then remove the old data to archive and then eventually add the present 90 days into the new log, with all the indexing we can find.</p> This sounds very similar to our system. The larger files were up to 10Gig in size, which we’ve reduced by making the main column smaller. We are now in the re-indexing stages, but will eventually have to reduce the 220Gig files, by removing the older data, deleting it etc. Your idea seems sound. We probably will produce a new table, and then remove the old data to archive and then eventually add the present 90 days into the new log, with all the indexing we can find.

]]>