Why Deployment Freezes Don't Prevent Outages

I have $10 that says you’ve experienced this before: there’s a holiday, trade show, or other important event coming up. Management is worried about the risk of an outage during this all-important time, and restricts deployments from the week prior through the end of the event.

What really happens, of course, is that the system in question becomes booby-trapped with extra risk. As a result, problems are more likely, and when there there is even a slight issue, it has the potential to escalate into a major crisis.

Why does this happen? As usual, there’s no single root cause, but a variety of problems combine to create a brittle, risky situation.

freeze

Assumptions

When managers declare a freeze, they’re not being malicious. They’re doing something that seems to make sense. That’s why it’s important to understand the reasoning.

» Continue Reading (about 2100 words)

Can You Bring A Guitar To Velocity?

I need help. I’m giving an Ignite talk at Velocity EU that involves a guitar. I don’t want to bring a guitar all the way from America just for this. Would you please loan me one?

guitar

» Continue Reading (about 100 words)

How To Print All The Way To The Edge In Microsoft Word

You’re creating a document with Word that you want to turn into a nice full-page PDF. It has a gorgeous background color that will look great. But every time you convert it to a PDF, it ends up with ugly white borders at the edges, and Word warns you about printing beyond the printable margins. Dragging the margins and changing the Page Setup options does no good. How can you fix this?

Magic Margin

» Continue Reading (about 300 words)

Win a Free Pass to Velocity

O’Reilly Velocity is November 17-19 in Barcelona. O’Reilly gave me a 2-day pass to give away, and I decided to have some fun with it. We’re also giving away a pass on the VividCortex blog, so you can double your odds of winning. For your chance to win a 2-day pass, do one of the following: Answer any of the following questions; or Write a haiku that’s somehow relevant to Velocity Tweet your answers to @xaprb with #velocitytrivia.

» Continue Reading (about 300 words)

The Root Cause Fallacy

Wouldn’t you like to find the root cause of that downtime incident? Many people would. But experience has taught me that there is no such thing as a single root cause. Instead, there’s a chain of interrelated causes, each of which is necessary but none of which is sufficient to cause the overall problem.

Turtles

» Continue Reading (about 600 words)

A Review Of The Docker Book

The Docker Book is a newly published book from James Turnbull, whose name you will recognize if you’re at all familiar with DevOps, Puppet, or Docker itself. It’s a nice introduction to what Docker is and how to get started using it. It’s like Goldilocks — not too detailed, not too superficial, just right. The book starts from the basics, assuming no prior knowledge with Docker, or even most of the core concepts of virtualization, but moves quickly through these topics into installing Docker and getting started with it.

» Continue Reading (about 300 words)

Mac's Time Machine and Symlinks

I use Mac OSX’s built-in Time Machine for backups, and a couple of times I’ve noticed my backups failed and couldn’t be completed successfully. I was unable to fix the problem until I reformatted the backup drive. Today I think I stumbled on the solution.

Time Machine

» Continue Reading (about 200 words)

On Focus

Focus is perhaps the most important attribute in an organization. In fact, my dictionary defines an organization as “an organized body of people with a particular purpose…” A focused organization recognizes and cleaves to its purpose.

Likewise, the ability to create and sustain focus is perhaps the most valuable skill of the organization’s members, including both individual contributors and leaders.

Rainbokeh

What Is Focus, And Why Is It Hard?

My dictionary says focus is “an act of concentrating.” Consider the root words: concentrate literally means to bring to a common center, to be centered together.

» Continue Reading (about 2600 words)

Meditate With Me At Velocity

Join me and other Velocity attendees during the Wednesday afternoon 2:40pm break for a 10-15 minute guided meditation session appropriate for people of any faith or of none.

Meditation has a host of scientifically proven immediate and long-term benefits. If you get an extra 10% of clarity and effectiveness for the rest of the afternoon, you’ll end up learning more and making your conference experience more worthwhile.

meditation

» Continue Reading (about 300 words)

Unfixable Code

Over the years I’ve come to believe something that I’m not sure others will agree with. I would like to hear your point of view on it.

I posit that some code can become literally unfixable. Programmers can paint themselves into a corner with the code and it becomes impossible to get out again.

humpty

The scenario arises when a specific set of conditions exists:

» Continue Reading (about 300 words)

Two of My Favorite Conferences: Velocity and Surge

Two of my favorite conferences are coming up. One’s just next week, and another’s in the fall.

Velocity

Velocity is such a great event. I wanted to go for years, and when I finally did it was honestly one of the highlights of my professional career. I still don’t know what I did get get invited to speak that first year. It was a golden horseshoe falling out of the sky and landing right in front of me.

Velocity

» Continue Reading (about 700 words)

Time-Series Database Requirements

I’ve had conversations about time-series databases with many people over the last couple of years. I wrote previously about some of the open-source technologies that people commonly use for time-series storage.

Time Series

Because I have my own ideas about what constitutes a good time-series database, and because a few people have asked me to describe my requirements, I have decided to publish my thoughts here. All opinions that follow are my own, and as you read you should mentally add “in my opinion” to every sentence.

» Continue Reading (about 1500 words)

Can Anomaly Detection Solve Alert Spam?

Anomaly detection is all the buzz these days in the “#monitoringlove” community. The conversation usually goes something like the following: Alerts are spammy and often generate false positives. What you really want to know is when something anomalous is happening. Anomaly detection can replace static thresholds and heuristics. The result will be better accuracy and lower noise. I’m going to give a webinar about the science of statistical anomaly detection on June 17th.

» Continue Reading (about 100 words)

The Goal

Once upon a time I managed several teams of consultants. At a certain stage of the organization’s growth, we wanted to achieve a higher billable-time utilization more easily, and we wanted more structure and process.

Cary Millsap, about whom I have written quite a bit elsewhere on this blog, suggested that I might profit from reading The Goal by Eliyahu Goldratt. I will let history be the judge of the outcome, but from my perspective, this was revolutionary for me. It is a clear watershed moment in my memory: I lived life one way and saw things through one lens before, and afterwards everything was different.

Horse Race

» Continue Reading (about 1500 words)

Can MySQL be a 12-factor service?

A while ago I wrote about some of the things that can make MySQL unreliable or hard to operate. Some time after that, in a completely unrelated topic, someone made me aware of a set of principles called 12-factor that I believe originated from experiences building Heroku.

Dodecahedron

That’s been over a year, and I’ve come to increasingly agree with the 12-factor principles. I guess I’m extremely late to the party, but making applications behave in 12-factor-compliant ways has solved a lot of problems for me.

This experience has repeatedly reminded me of one of the applications that continues to cause a lot of the kinds of pain that the 12-factor principles have solved for me: MySQL.

» Continue Reading (about 500 words)

Monitorama 2014: This One Weird Time-Series Math Trick

Monitorama 2014 Portland has been a great show. I’ve enjoyed the technical nature of the talks, the diversity of the speakers, the topics from hilarious to thought-provoking, and the stage in a theater, set up for a Shakespearean tragedy. I have also taken a lot of notes. For example, Toufic from Metafor Software suggested that the audience look into the Kolmogorov-Smirnov test. I am proud of the slide that made its way into my talk as a result: My slides are embedded below.

» Continue Reading (about 200 words)

GopherCon 2014

I spoke at Gophercon last week in Denver, and it was one of the best conferences I’ve attended. I can’t remember learning so much and meeting so many great people in years. I have page after page of notes in my notebook, many of which I’ve yet to follow up on. The conference prompted a burst of learning and a flurry of creativity for me, as well as a huge list of things to study further.

Gophercon

In no particular order, here are some of the many highlights for me:

» Continue Reading (about 600 words)

Go MySQL Drivers

If you’re interested in Google’s Go programming language, perhaps you aren’t sure what drivers to use for MySQL. The good news is there are excellent drivers for MySQL. There are several opensource ones on GitHub and elsewhere, but the driver I recommend is https://github.com/go-sql-driver/mysql/. Why? It is pure Go, not a wrapper around a C library, and is liberally licensed. It is high performance. A lot of work has gone into making it avoid allocations and consume minimal CPU.

» Continue Reading (about 300 words)

JOIN Versus Key-Value Stores

I was listening to a conversation recently and heard an experienced engineer express an interesting point of view on joins and key-value databases. I don’t entirely agree with it. Here’s why.

Library Of Congress

First, the opinion. If I may paraphrase, the discussion was something like this:

  • With experience in building distributed systems, one learns to avoid JOIN.
  • Therefore, much of the work of JOIN is done in the application instead of the database.
  • Access to the database is usually reduced to simple primary-key lookups.
  • Therefore, a key-value store is as good a choice as a relational database.

I’m simplifying, because the speaker actually suggested that MySQL makes a really good database for primary-key lookups as well.

The place I would differ slightly is on the last bullet point.

» Continue Reading (about 700 words)

Ultima Online and the History of Sharding

Have you heard of sharding a database? Of course you have. Do you know where the term comes from? Someone asked me this at a cocktail party recently. I gave it my best shot.

“The earliest I remember was Google engineers using it to describe the architecture of some things,” I said. “That would have been about 2006.”

“Nope. Much earlier than that,” said my new friend.

ultima

I pondered. “Well, I guess there was the famous LiveJournal architecture article about MySQL. That was, I dunno, 2003?”

The person then told me the following history. I can neither confirm nor deny it; what do you know about it?

» Continue Reading (about 400 words)