Crash injection for writing resilient software

I am currently finishing some features to make a program highly resilient to occasional crashing bugs. A particular function was found to crash on queries of the form WHERE x IN(NULL), and that crashed the entire program. Now we have a framework for intelligently recovering from arbitrary crashes. I will write more on this in the future, because I think it’s a very interesting thing to share. In this episode, I want to focus on a related topic: how do you test a program that is supposed to be resilient to bugs you can’t predict?

» Continue Reading (about 500 words)

The right way to do social

The technology industry moves incredibly fast, from one bubble to another. Web 2.0. Online auctions. (Remember when the Internet was filled with hundreds of eBay clones?) Social. Mobile. Location-based. Big Data. Whatever. I don’t think anyone will call me insightful for observing that the general idea of “social” had a peak in its hype cycle some time ago. I’d say three years ago was really the peak. At some point, lots of people were excited about applying social-ness to everything.

» Continue Reading (about 700 words)

Adaptive Fault Detection food fight

I was a guest on the Food Fight Show last week, along with a bevy of really smart people asking and answering tough questions on fault detection. We didn’t talk a lot about MySQL, but given that VividCortex is focusing on MySQL initially, pretty much all of my experience with zero-threshold, zero-configuration fault detection is MySQL-based. It’s a fun conversation with a lot of insights into the industry, what’s wrong with current monitoring tools, and where monitoring is going.

» Continue Reading (about 200 words)

Playing matchmaker for job seekers and recruiters

One of the most rewarding things you can do is help someone get a great job or hire a great person for the position they need to fill. I have traveled a lot, written books, done a bunch of consulting, and spoken widely on MySQL, other databases, open source, and so forth. I’ve gotten to know a lot of people, some I’d call good friends, and many of them are leading large organizations.

» Continue Reading (about 600 words)

Upcoming events of interest for MySQLers

Here is a collection of upcoming events that are interesting to me as a MySQL user (in some cases because I’m speaking). I think some of them are must-see events. I am sure I am missing a lot of events, but some of these are only publicized in specific channels, and I wanted to mention them here to help spread the word. Upcoming Meetups This coming Thursday, Charlottesville’s Neon Guild tech group will co-host a Meetup with a noted Kanban expert.

» Continue Reading (about 300 words)

What's the value of a Computer Science education?

I’ve been increasingly questioning the current model of university education in the US. Not only the value for the money, but just the entire notion that it’s a good way to learn. I got my Bachelor’s in Computer Science from UVA, which has been going through utter facepalm-worthy madness recently. It may be biasing my point of view. A friend recently sent me this: This shop was written up in the WSJ last week: My daughter is looking at it.

» Continue Reading (about 400 words)

Seeing things from the user's point of view

I was discussing how to avoid surprising users and someone pointed out that what seems intuitive and rational to one person is often complete insanity for others. The mental gap between a developer and a user can often be a chasm far too wide to cross. Of all the bug reports I’ve filed against MySQL, here is my all-time favorite: select * from t where a >= 1.0order by a; Does not cause an error.

» Continue Reading (about 400 words)

How scalable is your database?

Most of the time, when people say “scalability” they mean any of dozens of things. Most of the time, when I say it I mean exactly one precisely defined thing. However, I don’t claim that’s the only correct use of “scalability.” There is another, in particular, that I think is very important to understand: the inherent limitations of the system. This second one doesn’t have a single mathematical definition, but it’s vital nonetheless.

» Continue Reading (about 400 words)

Updated spam controls

I’m not using my previous combination of Bad Behaviour + WP-Gatekeeper anymore. I’ve removed them in favor of ordinary reCAPTCHA. Although it lets a lot of spammers through (almost a thousand spam comments in the day I’ve had it enabled) and Bad Behaviour let only 1 or 2 a month through, it became clear that a lot of people were getting blocked wrongly by Bad Behaviour. This means that I can’t practically moderate the spam queue.

» Continue Reading (about 200 words)

Setting up a Go build system in Sublime Text 2

Sublime Text 2 is really nice, but the documentation is truly awful. There are at least 3 versions of it, all of them wrong and incomplete. This blog post shows how I set up a Go build system. I am using GoSublime, but that is completely independent of the build system. The goal of GoSublime is nice auto-completion and snippets and so on; the build system is just supposed to invoke a command with Control-B (or Command-B), capture the output, and recognize file/line in the output so you can navigate between build errors easily.

» Continue Reading (about 200 words)

Sessions I'd like to see at Percona Live in April

I’m really looking forward to this year’s Percona Live MySQL Conference. This is always THE event of the year for me in the MySQL conference circuit. It’s also the first year I haven’t been a speaker! I’ve been a speaker since 2007 but this year things were too uncertain for me to submit a proposal in time. As usual, the real highlight of the conference is seeing and talking to everyone.

» Continue Reading (about 500 words)

Why does MySQL's version comment change when logging is enabled?

I wonder if the MySQL archaeologists out there would be willing to unearth some (presumably ancient) history for me. Why does the logging configuration merit special mention in the version_comment variable? The more I think about this, the more bizarre it seems. I enabled logging. The version reported by the server changed. No, really, is my server somehow a different version of MySQL now? 130203 15:39:55 [Note] ./bin/mysqld: ready for connections.

» Continue Reading (about 200 words)

If yoda you were, SQL you would invent

SQL is such a bizarre language. It is all backwards. Consider: SELECT item_id, COUNT(*) AS qty, SUM(qty * price) AS total FROM line_item WHERE invoice_id = 18743 GROUP BY item_id HAVING SUM(discount) 0 ORDER BY item_id; What is the logical flow of this statement? If SQL were written in logical-flow order, it might look something like this: FROM line_item WHERE invoice_id = 18743 RETRIEVE item_id, qty, price, discount GROUP BY item_id COMPUTE item_id, COUNT(*) AS qty, SUM(qty * price) AS total, SUM(discount) HAVING SUM(discount) 0 PROJECT item_id, qty, total ORDER BY item_id; SQL can get really hard to read and understand, really fast.

» Continue Reading (about 300 words)

Thinking clearly about fitting a model to data

I have often seen people fitting curves to sets of data without first understanding whether that is appropriate. I once even used this blog to criticize someone for doing that. I was trying to explain that it’s wrong to fit a model to a set of measurements, unless the model actually describes the process that produced the measurements. All of my explanations (and rants) have fallen far short of the clarity and simplicity of this curve-fitting guide.

» Continue Reading (about 400 words)

Deleting millions of rows in small chunks with common_schema

I wrote pt-archiver for jobs like deleting or archiving rows from a big table in small chunks. These days, that’s the kind of task I like doing inside the database, and Shlomi’s magical common_schema feels a lot more suited for this than an external Perl script. When I say it’s magical, it really does feel magical. It’s amazing how he’s created an entire expressive scripting language that runs in MySQL and feels just right for the job.

» Continue Reading (about 200 words)

How scalable is Riak?

I’m reading a little bit about Riak, and was curious about performance and scalability. The only benchmark I found that allowed me to assess scalability was this one from Joyent. Of course, they say scalability is linear (everyone says that without knowing what it means) but the results are clearly not a straight line. So how scalable is it, really? The Universal Scalability Law is such a powerful tool for thinking about scalability.

» Continue Reading (about 200 words)

MySQL 5.6 adds connection attributes

I enjoyed being able to add metadata to a connection in Microsoft SQL Server. I’d annotate my connections so that a DBA could learn a little bit by inspecting it. For example, what was its purpose, and from which application did it originate? The employer where I did this wasn’t perfect at managing their database user accounts and so forth, and there were many servers with hundreds of databases on each server, so this was a good way to provide some extra hints.

» Continue Reading (about 200 words)

Bold predictions on which NoSQL databases will survive

In case you’ve been living under a rock for the last 5 years, the NoSQL movement has changed. There was a time when everyone – EVERYONE – was dumping on relational databases, and MySQL in particular. Nonsense like “SQL itself is inherently unscalable” routinely came out of the mouths of otherwise usually sensible people. But that’s cooled off a little bit, thank heavens. And what’s the new hotness? Well, Big Data, of course!

» Continue Reading (about 300 words)

A close look at New Relic's scalability chart

I’ve written a lot about modeling MySQL with the USL, and I like it best of all the scalability models I’ve seen, but it’s not the only way to think about scalability. I was aware that New Relic supports a scalability chart, so I decided to take a peek at that. Here’s a screenshot of the chart, from their blog: Here’s how it works. It plots response time (or database time, or CPU) as the dependent variable, versus throughput as the independent variable.

» Continue Reading (about 800 words)

Modeling scalability with the USL at concurrencies less than 1

Last time I said that you can set a starting value for the USL’s coefficient of performance and let your modeling software (R, gnuplot, etc) manipulate this as part of the regression to find the best fit. However, there is a subtlety in the USL model that you need to be aware of. Here is a picture of the low-end of the curve: The graph shows the USL model as the blue curve and linear scalability as the black line.

» Continue Reading (about 500 words)