Thanks for sharing your experiences regarding RHCS and crossover cables. Very useful to know, you seem to have at least some experience in testing various systems.

1. keeping ib_logfileN files on the fastest storage available for the budget and on a separate array (not just partition) when possible. In regard to filesystems for best recovery timings on this specific process I’ve found the following to be in order of fast->slow time: Reiser > XFS > Ext2 > Ext3. Haven’t seen anyone else posting test results for this issue, perhaps I’ll put some graphs up on my site later.
2. setting innodb_log_files_in_group to a value > 2 where applicable to allow faster I/O processing of the crash related log data.
3. setting innodb_log_file_size to a minimum sane value (as required by the size of the buffer pool and workload type)
4. setting innodb_support_xa=1 on master servers to ensure binlog contents are in proper order (more of an assurance than speed thing though as I have not tested the speed difference on/off)

Of course LVS, Pacemaker, RHCS are not perfect. In an ideal world InnoDB would just work in an active/active/active/etc role like NDB. Then we wouldn’t have to worry about the current issues with active node failover and slave (or passive node) promotion. Likewise, replication as it is currently implemented is not perfect either. Semi-Synchronous is a nice addition but still does not solve the performance requirement of statement processing in heavy traffic environments.

Back to the Active/Passive setups: I have noticed less false positive failovers and less split brain scenarios when using RHCS vs LVS. I suppose it has to do with the fencing method to prevent split brain and the method in which RHCS monitors the MySQL process. In regard to missed heartbeats causing failovers when not needed I find that direct (no switch layer) crossover cables utilizing bonded interfaces prevents that issue almost entirely, to the point that I have never had false failovers when using this method of heartbeat setup. I’ve seen a lot of people asking for help with their H/A pair where the issue can be traced back to not using bonded interfaces for heartbeat (and their VIP on a separate set of interfaces as well). My minimum requirement for all clusters I admin is bond0 for the DB traffic VIP and bond1 for the heartbeat interfaces. Works well.

As far as solutions, Baron seems to be on the hunt for a better than MMM/Heartbeat/RHCS/… whereas I have brought up the concept of not really needing to have a decision between master vs slave. Systems that provide such feature seem to be either synchronous replication (MySQL Cluster, Galera) or NoSQL CAP theorem based solutions (Voldemort, Dynamo, etc…).

There are a number of types of architectures for HA systems. There are good solutions for sync replication, block level replication, proxies, etc. But there isn’t a good tool for moving virtual IP addresses and promoting a slave to replace a failed master. Yves is trying to solve that right now. It’s too early to say anything definitive but I’m really optimistic. This model of HA tool is the only really useful one that isn’t really provided well, IMO, among the major types of HA solutions that people want/need.

Personally I haven’t seen any RHCS+SAN deployments. A lot of our customers are running in Rackspace/Softlayer/etc and that’s why they want plain old replication and VIPs. That’s why I’ve been exposed to that more.