Comments on: An algorithm to find and resolve data differences between MySQL tables http://www.xaprb.com/blog/2007/03/05/an-algorithm-to-find-and-resolve-data-differences-between-mysql-tables/ Stay curious! Sun, 20 Jul 2008 22:54:46 +0000 http://wordpress.org/?v=2.2.2 By: Xaprb http://www.xaprb.com/blog/2007/03/05/an-algorithm-to-find-and-resolve-data-differences-between-mysql-tables/#comment-13502 Xaprb Tue, 09 Oct 2007 12:52:04 +0000 http://www.xaprb.com/blog/2007/03/05/an-algorithm-to-find-and-resolve-data-differences-between-mysql-tables/#comment-13502 But by all means, explore your algorithm too! I don't mean to say you shouldn't. It may be a much better way. But by all means, explore your algorithm too! I don’t mean to say you shouldn’t. It may be a much better way.

]]> By: Xaprb http://www.xaprb.com/blog/2007/03/05/an-algorithm-to-find-and-resolve-data-differences-between-mysql-tables/#comment-13501 Xaprb Tue, 09 Oct 2007 12:50:36 +0000 http://www.xaprb.com/blog/2007/03/05/an-algorithm-to-find-and-resolve-data-differences-between-mysql-tables/#comment-13501 Please take a look at MySQL Table Sync in the MySQL Toolkit (http:mysqltoolkit.sourceforge.net). It may save you a lot of work. I've implemented both algorithms there. Please take a look at MySQL Table Sync in the MySQL Toolkit (http:mysqltoolkit.sourceforge.net). It may save you a lot of work. I’ve implemented both algorithms there.

]]> By: Negruzzi Cristian http://www.xaprb.com/blog/2007/03/05/an-algorithm-to-find-and-resolve-data-differences-between-mysql-tables/#comment-13500 Negruzzi Cristian Tue, 09 Oct 2007 12:38:21 +0000 http://www.xaprb.com/blog/2007/03/05/an-algorithm-to-find-and-resolve-data-differences-between-mysql-tables/#comment-13500 Nice article. I've studied the algorithm before, but wasn't so clearly. Actually I met he same need for keeping my databases synchronized, and for some days I'm trying to build a suitable algorithm for that. The Top-Down algorithm is good, but, as you mentioned, is too hard to find a good grouping. And how what is to be done if there is only one indexed column as Primary Key? So I find the Bottom-Up more suitable for that purpose, but with some differences, I'm going to build a B*-Tree based on row checksums for each table, keep it saved locally (I think a XML structure is a good way) and, for saving time and traffic, do the comparison locally too. I'm not sure it's the best way for that, but i want to try. Best regards. Nice article. I’ve studied the algorithm before, but wasn’t so clearly.
Actually I met he same need for keeping my databases synchronized, and for some days I’m trying to build a suitable algorithm for that. The Top-Down algorithm is good, but, as you mentioned, is too hard to find a good grouping. And how what is to be done if there is only one indexed column as Primary Key? So I find the Bottom-Up more suitable for that purpose, but with some differences, I’m going to build a B*-Tree based on row checksums for each table, keep it saved locally (I think a XML structure is a good way) and, for saving time and traffic, do the comparison locally too. I’m not sure it’s the best way for that, but i want to try.
Best regards.

]]> By: chhivhorng http://www.xaprb.com/blog/2007/03/05/an-algorithm-to-find-and-resolve-data-differences-between-mysql-tables/#comment-11909 chhivhorng Wed, 27 Jun 2007 09:57:07 +0000 http://www.xaprb.com/blog/2007/03/05/an-algorithm-to-find-and-resolve-data-differences-between-mysql-tables/#comment-11909 I have some problem with query sql. I want you correct for me. this is my query : $queryselect="select Distinct student.id_student,first_name,last_name,promotion,class from student,applicationrecieve where student.id_student applicationrecieve.id_student"; the query that I write to you, I want to select data that it don't have in the table applicationrecieve from table student. thanks, regards, chhivhorng I have some problem with query sql.
I want you correct for me.
this is my query :
$queryselect=”select Distinct student.id_student,first_name,last_name,promotion,class
from student,applicationrecieve where student.id_student applicationrecieve.id_student”;

the query that I write to you, I want to select data that it don’t have in the table applicationrecieve from table student.

thanks,
regards,
chhivhorng

]]> By: Xaprb http://www.xaprb.com/blog/2007/03/05/an-algorithm-to-find-and-resolve-data-differences-between-mysql-tables/#comment-6793 Xaprb Tue, 15 May 2007 11:57:22 +0000 http://www.xaprb.com/blog/2007/03/05/an-algorithm-to-find-and-resolve-data-differences-between-mysql-tables/#comment-6793 <p>Hello Fabien. The issue with the indexing is not scans, but lookups from a child table to its parent tables, including the group-by queries. These happen potentially many times. I could benchmark with and without indexes fairly easily and see for sure, but after writing all the queries I'm satisfied the index is important.</p> <p>The WHERE clause has proven to be very important, as you guessed.</p> <p>I did some testing with real data, and the results are here: <a href="http://www.xaprb.com/blog/2007/03/30/comparison-of-table-sync-algorithms/" rel="nofollow">Comparison of Table Sync Algorithms</a>.</p> Hello Fabien. The issue with the indexing is not scans, but lookups from a child table to its parent tables, including the group-by queries. These happen potentially many times. I could benchmark with and without indexes fairly easily and see for sure, but after writing all the queries I’m satisfied the index is important.

The WHERE clause has proven to be very important, as you guessed.

I did some testing with real data, and the results are here: Comparison of Table Sync Algorithms.

]]> By: Fabien Coelho http://www.xaprb.com/blog/2007/03/05/an-algorithm-to-find-and-resolve-data-differences-between-mysql-tables/#comment-6764 Fabien Coelho Tue, 15 May 2007 09:06:13 +0000 http://www.xaprb.com/blog/2007/03/05/an-algorithm-to-find-and-resolve-data-differences-between-mysql-tables/#comment-6764 I finally read the stuff (this article and the source code). The various discussions are very interesting. I'm not sure that I understand fully the "index" issue with the bottom-up approach. Basically each summary table is build (once) and then it is scanned just once, so having an index built on some attribute would not be amortized. The only exception may be for bulk deletes or inserts, but that should not happen. On the "exploit append-only tables" idea, the bottom-up approach can have a "where" clause on the initial table so that the comparison is only performed on part of the data. Moreover, if the candidate tuples are somehow an identifiable fraction of the table, it might be simpler to just download them directly for comparison, that would be a third algorithm:-) Do you have performance figures with your tool in different settings? I finally read the stuff (this article and the source code). The various discussions are very interesting.

I’m not sure that I understand fully the “index” issue with the bottom-up approach. Basically each summary table is build (once) and then it is scanned just once, so having an index built on some attribute would not be amortized. The only exception may be for bulk deletes or inserts, but that should not happen.

On the “exploit append-only tables” idea, the bottom-up approach can have a “where” clause on the initial table so that the comparison is only performed on part of the data. Moreover, if the candidate tuples are somehow an identifiable fraction of the table, it might be simpler to just
download them directly for comparison, that would be a third algorithm:-)

Do you have performance figures with your tool in different settings?

]]> By: Xaprb http://www.xaprb.com/blog/2007/03/05/an-algorithm-to-find-and-resolve-data-differences-between-mysql-tables/#comment-4824 Xaprb Tue, 06 Mar 2007 12:57:43 +0000 http://www.xaprb.com/blog/2007/03/05/an-algorithm-to-find-and-resolve-data-differences-between-mysql-tables/#comment-4824 <p>Rohit, if I read your comment right, you're subtly saying I'm going a good direction, which is encouraging :-)</p> <p>James, you've reinforced my belief that lots of people need a tool that doesn't disrupt replication.</p> Rohit, if I read your comment right, you’re subtly saying I’m going a good direction, which is encouraging :-)

James, you’ve reinforced my belief that lots of people need a tool that doesn’t disrupt replication.

]]> By: James Holden http://www.xaprb.com/blog/2007/03/05/an-algorithm-to-find-and-resolve-data-differences-between-mysql-tables/#comment-4822 James Holden Tue, 06 Mar 2007 11:48:19 +0000 http://www.xaprb.com/blog/2007/03/05/an-algorithm-to-find-and-resolve-data-differences-between-mysql-tables/#comment-4822 <p>Very interesting. I've already developed a simple tool to do just that, albeit based on a simple row-by-row comparison, with the correcting actions being inserts or updates directly on the slave. It can recurse all the databases and tables to repair an entire database, or just operate on a single table. I'd not considered doing the corrective action on the master, but it's an interesting idea. You're absolutely correct that tables that lack a primary key are barely worth attempting, and so far my script ignores them. One thing that I've found in practice, is that you must perform the synchronisation while replication is actually running. If you don't, you will inevitably end up replicating ahead of the normal replication process and breaking it. I've found that this tool is useful for bringing up a new replica where it's impossible for the master to be affected in any ways, such as through table locking. If you do develop a working implementation of your own, do let me know! James</p> Very interesting. I’ve already developed a simple tool to do just that, albeit based on a simple row-by-row comparison, with the correcting actions being inserts or updates directly on the slave.

It can recurse all the databases and tables to repair an entire database, or just operate on a single table.

I’d not considered doing the corrective action on the master, but it’s an interesting idea.

You’re absolutely correct that tables that lack a primary key are barely worth attempting, and so far my script ignores them.

One thing that I’ve found in practice, is that you must perform the synchronisation while replication is actually running. If you don’t, you will inevitably end up replicating ahead of the normal replication process and breaking it.

I’ve found that this tool is useful for bringing up a new replica where it’s impossible for the master to be affected in any ways, such as through table locking.

If you do develop a working implementation of your own, do let me know!

James

]]> By: Rohit http://www.xaprb.com/blog/2007/03/05/an-algorithm-to-find-and-resolve-data-differences-between-mysql-tables/#comment-4814 Rohit Tue, 06 Mar 2007 06:32:50 +0000 http://www.xaprb.com/blog/2007/03/05/an-algorithm-to-find-and-resolve-data-differences-between-mysql-tables/#comment-4814 <p>Nice article! We have a similar tool called SQLyog Job Agent which incorporates most of what you have discussed in this article. Unfortunately, it is not open-source. We are always trying to improve the algorithm and look forward to more articles on this topic!</p> Nice article!

We have a similar tool called SQLyog Job Agent which incorporates most of what you have discussed in this article. Unfortunately, it is not open-source.

We are always trying to improve the algorithm and look forward to more articles on this topic!

]]>