The Ma.gnolia data might not be permanently lost
I keep reading that Ma.gnolia’s data is permanently lost because “a specialist had been unable to recover any data from the corrupted hard drive.” This is not in itself a reason to consider data completely lost.
It is not clear to me whether the hard drive itself is unusable, e.g. the spindle won’t spin and the head won’t read the ones and zeroes, or whether the filesystem is corrupted. It sounds to me, from reading Larry Hallf’s comments, like it’s a simple matter of filesystem corruption. And even if the disk is dead, there is apparently a backup made from the corrupted filesystem, so there should be more than one way to try to recover this data: “Ma.gnolia’s database server suffered from file system corruption, which also corrupted it’s database backup, even though it was on a separate system.”
You don’t need to recover your filesystem to recover your MySQL data. Shameless plug: Percona can do it for you. We can get the raw data off a block device without even trying to mount it as a filesystem. Recovering MySQL data is not the same as recovering other types of data. If the disk spins, it might be possible to recover data from it.
Whether it’s worth it or not is another matter. Percona data recovery isn’t cheap, but it’s worth it for at least some people. I cannot name names, but you are using services from companies that have retained Percona to recover from worse cases of data loss than this appears to be, going by the limited available information. The original reason we built our data recovery toolset was to help one of the world’s largest corporations.
But cost and time may not have been the driving factor here. Whoever the unnamed data recovery specialist was, they took a long time and got no results. And now Ma.gnolia has given up and declared it a lost cause, which is sad for their users. I hope Larry Halff didn’t pay for the results he didn’t get. And I hope he didn’t wipe out his corrupted backup yet.
In the meantime, at least this incident is shining a bright light on the need for tested, verified backups. I’ve had two clients ask me how they can avoid ending up the same way as Ma.gnolia.
It sounds like file corruption at either the software or hardware level, with the “backup” consisting of either bit-level or file-level copying. If they’d been doing a DB-level copy, there shouldn’t be corruption on the target machine. (Invalid data, sure, but not an unrecoverable filesystem.)
Tim McCormack
19 Feb 09 at 7:59 pm
[...] The Ma.gnolia data might not be permanently lost at Xaprb Really important thing to remember. "We can get the raw data off a block device without even trying to mount it as a filesystem. Recovering MySQL data is not the same as recovering other types of data. If the disk spins, it might be possible to recover data from it." (tags: data information harddrive mysql datarecovery) [...]
links for 2009-02-20 | Yostivanich.com
20 Feb 09 at 10:14 am
Why don’t you write to Larry and ask him yourself? The email larry@ma.gnolia.com still works, I’m sure, and there is a thread at Get Satisfaction where he’s been responding to member questions.
Todd Sieling
20 Feb 09 at 11:06 am
I posted to that thread and didn’t get a response.
Xaprb
20 Feb 09 at 11:10 am
Really? I just looked and didn’t see Xaprb in the threads, but if you say so. Maybe a pointer to the question?
Todd Sieling
20 Feb 09 at 11:21 am
Look for “Percona”.
Xaprb
20 Feb 09 at 11:25 am
There we go. I found: “Call Percona. You don’t just need filesystem recovery, you need MySQL-specific data recovery, and we are the only ones who can do that well.”
When I read the blog post here, it sounds like you have some questions, which is totally fair. But the posted comment on GS does come off like a command to call you as the only option, and given the commercial and commanding tone I’m not really surprised you didn’t hear back. I guess my comment in both cases is really about tone; if you wonder who the recovery company is, ask who it is privately or publicly. If you think they took too long (I think it was like a week?), then how long should it take? I guess I’m looking for less of what feels like baiting and more of what feels like conversation. It’s what I’d want from any vendor, regardless of the circumstances.
Todd Sieling
20 Feb 09 at 11:35 am
I don’t think it’s germane who the data recovery specialist was; there’s no need to know or name names when someone doesn’t succeed. And my comment on the thread is very brief, so you can read it any number of ways. I am not particularly attached to this issue one way or another. I think it’s sad for the users that the service is shut down when there might still be a chance, but I didn’t use it myself. And as for being commercial — you can see it that way, but honestly we are not pushing anyone for more work to do. We are trying to hire more people to do the work we have already. Data recovery is long, hard, tedious work.
I believe the company they tried took something between two and three weeks, which may or may not be reasonable, depending on the complexity of the problem, which is unknown.
My point here is this: there is another option, I’m not attached to the results myself, I informed about the option (too tersely, you’re right), and now in this blog post I’m actually trying to fill a knowledge gap in a broader sense:
“hey, if you lose your MySQL data, you should know that generic data recovery isn’t necessarily the best option. For this specific case, there’s a specific solution with a higher probability of success.”
Xaprb
20 Feb 09 at 11:56 am
I’ve posted a followup comment on the GS thread with more information, thanks for the prompting.
Xaprb
20 Feb 09 at 12:04 pm
I think we’re on the same page, and I appreciate the dialog. I worked with Larry extensively on Ma.gnolia as product manager until Feb 08 and on redesign work after that, so I’ve been in touch with him through all of this. Drive recovery analysis took about 10 calendar days, and was started after the cache recovery tools were in place, since the cache viability was a race against the clock.
I don’t know the details of the drive recovery work, just the outcome, but it’s good to know that there are other options for MySQL problems. As I finish this comment I see the followup on the GS thread, and really appreciate that.
Todd Sieling
20 Feb 09 at 12:08 pm
And I appreciate you setting me straight, and helping me have a chance to present this more clearly and appropriately!
Xaprb
20 Feb 09 at 12:12 pm
[...] I don’t know if Baron can rescue Ma.gnolia, per se, but I think the problem was largely: Doing a file sync over the [...]
On Ma.gnolia, and data recovery | Colin Charles Agenda
24 Feb 09 at 11:09 am
Good post, Baron! When I read about Ma.gnolia’s disaster, I wanted to write a somewhat similar post. Of course, you put it all in a nice way.
Reminds me of the time when I was left with a 1TB severe corruption. Every consultant we tried to bring on board (MySQL included) told us that the data cannot be recovered. Yet, after a very painful process, we were able to recover 99%.
I believe Ma.gnolia shouldn’t have given up so easily.
Frank
Frank Mashraqi
24 Feb 09 at 3:09 pm