Comments on: Introducing MySQL Parallel Dump http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/ Stay curious! Fri, 16 May 2008 03:40:22 +0000 http://wordpress.org/?v=2.2.2 By: Xaprb http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-14257 Xaprb Thu, 28 Feb 2008 13:12:28 +0000 http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-14257 Hi, There's a current forum conversation on the same topic. It would be good to move this there so it's all in the same place: http://sourceforge.net/forum/forum.php?thread_id=1952777&forum_id=664350 Hi,

There’s a current forum conversation on the same topic. It would be good to move this there so it’s all in the same place:

http://sourceforge.net/forum/forum.php?thread_id=1952777&forum_id=664350

]]> By: RV http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-14256 RV Thu, 28 Feb 2008 10:02:33 +0000 http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-14256 If you had multiple schema's, within one MySQL instance, and each one had no dependency/connection to the other, could one not spawn multiple (single-transaction, master-data) mysqldumps against each one and therefore reduce the time to perform full consistent InnoDB hot backups from a master? If not, then I believe this type of tool is only usable for large InnoDB instances on slaves. I know that performing dumps on masters is not recommended, but our replication confidence is low, so at the moment it's a must. Hopefully that's where Maatkit will again come to the rescue. If you had multiple schema’s, within one MySQL instance, and each one had no dependency/connection to the other, could one not spawn multiple (single-transaction, master-data) mysqldumps against each one and therefore reduce the time to perform full consistent InnoDB hot backups from a master? If not, then I believe this type of tool is only usable for large InnoDB instances on slaves. I know that performing dumps on masters is not recommended, but our replication confidence is low, so at the moment it’s a must. Hopefully that’s where Maatkit will again come to the rescue.

]]> By: Xaprb http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-14254 Xaprb Wed, 27 Feb 2008 15:42:22 +0000 http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-14254 It's not possible. Locking tables is the only way to do it. A single transaction would require many connections to be able to participate in the transaction, which is something I wouldn't assume MySQL will build anytime soon :-) I suppose another way would be to lock tables briefly and allow all child processes to do BEGIN TRANSACTION, but that isn't the way it's built at the moment. It’s not possible. Locking tables is the only way to do it. A single transaction would require many connections to be able to participate in the transaction, which is something I wouldn’t assume MySQL will build anytime soon :-)

I suppose another way would be to lock tables briefly and allow all child processes to do BEGIN TRANSACTION, but that isn’t the way it’s built at the moment.

]]> By: RV http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-14253 RV Wed, 27 Feb 2008 14:05:43 +0000 http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-14253 Baron, Great work, but would you mind confirming if this does a consistent snapshot as in single-transaction for mysqldump for InnoDB tables or is this simply not possible via a parallel method? Cheers Baron,

Great work, but would you mind confirming if this does a consistent snapshot as in single-transaction for mysqldump for InnoDB tables or is this simply not possible via a parallel method?

Cheers

]]> By: Xaprb http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-13448 Xaprb Thu, 04 Oct 2007 02:01:11 +0000 http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-13448 Just for information, the new version is released, and can do chunked dumps as promised. Testing and benchmarks welcome! Just for information, the new version is released, and can do chunked dumps as promised. Testing and benchmarks welcome!

]]> By: Xaprb http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-13446 Xaprb Wed, 03 Oct 2007 14:32:34 +0000 http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-13446 By default, it does FLUSH TABLES WITH READ LOCK once at the beginning of the whole process, then passes the --skip-lock-tables option to mysqldump so it doesn't try to get its own lock. If you give the --locktables option, it locks only tables it uses. This is enabled by default when --sets is given. Each set gets and releases locks on the tables it uses, so tables in each set will be consistent, but may not be consistent with tables from other sets. You can disable the global lock with --noflushlock. By default, it does FLUSH TABLES WITH READ LOCK once at the beginning of the whole process, then passes the –skip-lock-tables option to mysqldump so it doesn’t try to get its own lock.

If you give the –locktables option, it locks only tables it uses. This is enabled by default when –sets is given. Each set gets and releases locks on the tables it uses, so tables in each set will be consistent, but may not be consistent with tables from other sets.

You can disable the global lock with –noflushlock.

]]> By: Dan http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-13445 Dan Wed, 03 Oct 2007 14:18:41 +0000 http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-13445 Baron - Could you be a little more explicit about how you lock tables? Do you lock all tables in a database at the start of the dump and only release the locks when all the threads are done with every table? Or are you locking tables as needed and releasing the lock when the dump for that table is done? Thanks again, Dan Baron -

Could you be a little more explicit about how you lock tables? Do you lock all tables in a database at the start of the dump and only release the locks when all the threads are done with every table? Or are you locking tables as needed and releasing the lock when the dump for that table is done?

Thanks again,

Dan

]]> By: Xaprb http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-13444 Xaprb Tue, 02 Oct 2007 20:27:20 +0000 http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-13444 Thanks, that's great. The next version will perform even better on the single big table: it'll dump it into chunks in parallel (this is to help address the issue of loading in parallel and/or loading with commits between chunks). Thanks, that’s great.

The next version will perform even better on the single big table: it’ll dump it into chunks in parallel (this is to help address the issue of loading in parallel and/or loading with commits between chunks).

]]> By: Dan http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-13443 Dan Tue, 02 Oct 2007 20:24:50 +0000 http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-13443 Here are some benchmarks: Performed on a 3.6GHz Xeon with 4 processors. The filesystem is a RAID 10 array of 15k speed disks, directly attached to the server with a fibre channel. <pre>Size of DB: #of tables: mysqldump: mysql-parallel-dump: 1.4GB 22 5m 45sec 4m 29sec 117MB 232 37sec 7sec</pre> Note that the larger db had most of the space allocated for a single large table, so a lot of the time was spent on dumping that table. Parallel threads were not that helpful in that case. They were very helpful where there were lots of tables of roughly equivalent sizes. I can probably run some more thorough tests, if you'd like, since my methods were pretty informal. Thanks, Dan Here are some benchmarks:

Performed on a 3.6GHz Xeon with 4 processors. The filesystem is a RAID 10 array of 15k speed disks, directly attached to the server with a fibre channel.

Size of DB:   #of tables:     mysqldump:        mysql-parallel-dump:
1.4GB            22           5m 45sec          4m 29sec
117MB            232          37sec             7sec

Note that the larger db had most of the space allocated for a single large table, so a lot of the time was spent on dumping that table. Parallel threads were not that helpful in that case. They were very helpful where there were lots of tables of roughly equivalent sizes.

I can probably run some more thorough tests, if you’d like, since my methods were pretty informal.

Thanks,

Dan

]]> By: Xaprb http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-13442 Xaprb Tue, 02 Oct 2007 19:45:18 +0000 http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/#comment-13442 Hi Dan, How large a speedup? If you can share your benchmarks (data size, speed of mysqldump, speed with mysql-parallel-dump, number/speed of CPUs, number/speed/RAID configuration of disks) it would be great to have some representative numbers for the documentation. I'm trying to give some general ideas in the documentation of when and how much this is likely to be helpful. I've found one system where it's not all that helpful. Multiple transactions shouldn't be a problem. Good old LOCK TABLES keeps it consistent. Hi Dan,

How large a speedup? If you can share your benchmarks (data size, speed of mysqldump, speed with mysql-parallel-dump, number/speed of CPUs, number/speed/RAID configuration of disks) it would be great to have some representative numbers for the documentation. I’m trying to give some general ideas in the documentation of when and how much this is likely to be helpful. I’ve found one system where it’s not all that helpful.

Multiple transactions shouldn’t be a problem. Good old LOCK TABLES keeps it consistent.

]]>