Xaprb

I am currently finishing some features to make a program highly resilient to occasional crashing bugs. A particular function was found to crash on queries of the form WHERE x IN(NULL), and that crashed the entire program. Now we have a framework for intelligently recovering from arbitrary crashes. I will write more on this in the future, because I think it’s a very interesting thing to share.

In this episode, I want to focus on a related topic: how do you test a program that is supposed to be resilient to bugs you can’t predict? Many new problems are caused by writing clever code that is supposed to detect, avoid, or recover from problems, even known problems. Unknown problems are even riskier.

The approach that has given me a great deal of confidence in the “resilience framework” is to write a “crash injection framework.” (If it’s not a framework, it can’t be any good, right?) Crash injection can be activated at runtime to cause crashes in arbitrary source code locations. A fairly simple analysis of the source code showed me some places I’m particularly interested in crashing to see how the program reacts.

In the first iteration, it turned out that I hadn’t actually solved the problem, even though I was sure I had. (I had already discovered previous mistakes, so this was actually about the 3rd time through, and I was sure I’d found all the bits I’d previously missed.) My crash injection framework allowed me to quickly dispel the notion that my program was resilient to the kinds of bugs we’d already found.

The fix was simple, but the lesson was well learned. I don’t remember doing this before, but I’ve studied and admired other pieces of software that are built to be resilient. For example, CouchDB has a crash-only design. If you’re not familiar with that phrase, it’s well worth reading more about that. Stewart Smith has also written software that helps reveal wrong assumptions, such as libeatmydata, and I think he also wrote something that randomly broke malloc(), but I can’t find that easily. Assumptions like “allocating memory will always succeed” are great places to end up with horrible bugs.

In the MySQL world, it would be great if we had more crash-injection testing of InnoDB as well. If you agree, you might consider subscribing to http://bugs.mysql.com/bug.php to keep up to date with effort on that front.

One of the most rewarding things you can do is help someone get a great job or hire a great person for the position they need to fill. I have traveled a lot, written books, done a bunch of consulting, and spoken widely on MySQL, other databases, open source, and so forth. I’ve gotten to know a lot of people, some I’d call good friends, and many of them are leading large organizations. I think this is both a privilege and a serious responsibility.

It’s a privilege because I can ask some of these people for help or introductions or advice sometimes. It’s a responsibility because I need to be ready to do something for them, too. In many cases it’s a pay-it-forward kind of readiness.

Many, many people contact me looking for people to hire. I keep a list. When someone tells me they are on the job market, I try to match them with openings I’m aware of, if any are appropriate. (Many fewer people tell me they’re looking for jobs than tell me they’re trying to recruit.)

These are my friends, on both sides. I don’t introduce people unless I think everyone is really genuinely bringing something great to the table. A great position — a great talent — the right circumstances and attitudes.

When I do introduce people, I usually do it one-sidedly, depending on the situation. For example, I’ll forward the candidate’s resume and contact information sometimes, with a few words of introduction. At other times I’ll just BCC the candidate. Or I might send the recruiting party’s information to the candidate.

People usually take this very seriously, but occasionally I’ve referred several people to a company only to find out that the company didn’t follow up on the introduction. That’s pretty much the last chance that company will get from me. If I made a special personal effort to send you someone fantastic who’s really qualified for the job, and you didn’t contact that candidate, there’s really no excuse for such rudeness. You’re telling me you don’t think my experience and judgment about who’s good for the job is valid. I won’t offer it again, then.

Recruiters are an interesting lot. They have a bad reputation, and I think that’s because a few aggressive, pushy people poison the well for the rest of the recruiters. Most recruiters are really good people doing a hard and extremely valuable job, with a lot of care and personal attention to their client as well as their candidates. If you’ve ever built a team, you know how hard it can be. Many recruiters are used to being treated very badly because of their profession’s reputation. I try to make up for that by being extra nice to them. Recruiters who contact me by email often get a personal call in return, and always a personal email reply with a thank you. In almost all cases, recruiters will tell me who their client is after I’ve introduced myself. I only refer my friends to recruiters when I know who they’re recruiting for, and I never make an end run around recruiters. They are doing a mostly thankless job, and they should not be cheated out of their commission.

Over the years I’ve been lucky enough to help a half dozen friends find jobs, and there’s pretty much no better feeling. If you’re looking, let me know. I know really good people who are looking for you. I’m never too busy to help make introductions.

I was a guest on the Food Fight Show last week, along with a bevy of really smart people asking and answering tough questions on fault detection. We didn’t talk a lot about MySQL, but given that VividCortex is focusing on MySQL initially, pretty much all of my experience with zero-threshold, zero-configuration fault detection is MySQL-based.

It’s a fun conversation with a lot of insights into the industry, what’s wrong with current monitoring tools, and where monitoring is going. Also, it’s sold out now, but Monitorama is a conference you might be interested in if you’re doing monitoring (and who isn’t?)