Xaprb

Web Operations

Web Operations. By John Allspaw and Jesse Robbins, O’Reilly 2010, with a chapter by myself. (Here’s a link to the publisher’s site).

I wrote a chapter for this book, and it’s now on shelves in bookstores near you. I got my dead-tree copy today and read everyone else’s contributions to it. It’s a good book. A group effort such as this one is necessarily going to have some differences in style and even overlapping content, but overall it works very well. It includes chapters from some really smart people, some of whom I was not previously familiar with. John and Jesse obviously have good connections. A lot of the folks are from Flickr.

Here are the highlights in my opinion.

• Theo Schlossnagle, who has a place on my list of essential books, opens things with an overview of what web operations really is, and why it’s hard. Don’t skip this. Theo’s introduction is concise and thoughtful.
• Eric Ries discusses the benefits of continuous deployment. He is right on the money. Right out of college I spent 3 years as a developer at a company with very little engineering discipline, and then left for another company built by a small ace team practicing extreme programming. Eric nails the benefits of continuous deployment — he really gets it. I hadn’t heard of Eric before, but now I’ve subscribed to his blog.
• John Allspaw (whose book on capacity planning is also on my list of essentials) and Richard Cook discuss how complex systems fail. This chapter appeared in part as a whitepaper and blog post on John’s blog, and is expanded in this book. I have spent a lot of time examining failures for clients, and as VP of Consulting, also a lot of time examining Percona’s own mistakes. I fully agree with the conclusions in this chapter. A few key points: there is never a single root cause; our desire to find one blinds us and keeps us from learning; true failures are inherently unpredictable and happen only when a series of things fails; avoiding failure requires experience with failure. This echoes another book I’ve read recently, The Black Swan.
• Brian Moon’s chapter on unexpected traffic spikes. If you get a chance to hear Brian speak, take it. He’s an engaging guy with interesting and relevant stories to tell. Stories are always a better experience than bullet points.
• Jake Loomis’s chapter on postmortems. My own research into prevention of emergencies agrees almost perfectly with his list of things to do on page 225. Read this chapter carefully! Now, knowing how to put this into action is hard — very hard — but at least you’ll have a place to start. The worst compliment I ever got after fixing a system that’d run out of hard drive space (due to utter lack of basic monitoring) was that I’d “saved the day.” Baloney. Postmortems can be a great way to learn your infrastructure’s weaknesses and prevent emergencies in the future. I’m fully confident that this particular client will again deploy new servers without adding them into Nagios, and the results will be predictable.
• Naturally, my chapter about choosing a relational database architecture for web applications (skewed towards MySQL). There is a chapter on NoSQL databases by Eric Florenzano as well, but it is more introductionary-level.

What wasn’t so good? I didn’t get a lot of value out of John’s interview with Heather Champ, on community management and web operations. I did not think the interview format worked well in a book full of essays. But that might just be me. Also, a couple of places in two or three chapters felt a bit rant-ish without a lot of clear actionable advice; I think readers won’t get so much out of this.

Overall, though, this is a great book, badly needed, on a topic that is simply not yet recognized for its true importance. As Theo writes, we’re seeing the emergence of web operations as a very large profession; it’s one whose definition is not yet formalized or agreed-upon, but that’ll change. It’s too important not to. Jesse’s introduction repeats this sentiment: the world now relies on the web, and so the world relies also on the engineers who make it run. Web operations is work that matters.

This is the follow-up to my post on how I keep track of tasks. It’s important for me to have a good system for keeping notes and other files organized. The problem usually turns out to be that I want them organized several different ways simultaneously: by date, by project, by person, by subject. Alas, if I keep them in files on a hard drive, I can only choose one such organizing strategy, because filesystems are a single hierarchy.

I choose to organize by date, simply because most of the time I need access to notes and files about things I’m working on now or recently. If I need to find files by project or subject, there’s a search feature in my file browser, and it works really well! So date-organization is good enough for me.

Inside my home directory, I have a directory per year, and inside that, a directory per month. If I write a note today, it goes into the $HOME/etc/2010/07/03/ directory. The filename starts with today’s date. That’s the simple organizing principle behind my note system. It also lets me eventually move things off my computer into permanent storage, so I don’t have to keep backing things up forever and carrying around infinite amounts of data. I keep the last couple of years; if I need access to notes or projects from 2006, I can go pull a hard drive off the shelf and pop it into my hard drive dock (buy one of those, and you’ll never get ripped off again by external drives with their own enclosures and power supplies).

I still need a quick way to create files and place them there, or move them there after I create them. For creating files, I use Vim. There is nothing better than a plain-text editor for me. My Vim settings are such that if I begin a line with a hyphen, Vim keeps nice indentation for me, making it easy to take notes in bulleted lists with proper indentation. If you’re on a call with me and you hear typing, I’m probably taking notes into Vim.

But it’s a pain to type out the full path to the file including the year, month, and date. So I created some helper scripts and put them into my $PATH. The most important are ‘t’ and ‘c’. ‘t’ simply uses Vim to edit a file. (It also creates any required directories, based on today’s date.) So if I am on a call with Joe, I just type ‘t joe’ into a terminal, and I’m editing /home/baron/etc/2010/07/03-joe.txt.

The ‘c’ tool cats the file’s contents. If I type ‘c joe’, it executes ‘cat /home/baron/etc/2010/07/03-joe.txt’. This makes it easy to grep, copy and paste, and so on.

There are a few more tools: the ‘m’ tool moves any file into the date-based hierarchy, so if I save a PDF of an order-confirmation page, for example, I can then ‘m’ it and it goes into its proper place. And I have a few tools to list files I created today, yesterday, this week, and this month.

I have a very important convention: when I’m taking notes and something becomes my responsibility to follow up on, I type TODO in the notes. After the call ends, I can grep for TODO in the file and quickly transfer the item into the task system I described in the post linked from above. This is how I can be confident that I’m not forgetting anything I’m supposed to do: I take notes and write it out as it happens, and then review the notes afterwards.

All told, this system kind of feels too simple to be a system. Everyone else seems to use complicated online gizmos named after groceries, or whizbang apps created by 37Signals, but I’ve found none of them to meet my needs, and just went back to basics. Basic is good. Basic works. Basic lets me concentrate on what I’m doing.

As I said in my previous post, part of this is based on the GTD book, which I read through a couple of times (with a year in between) and picked the parts that made sense to me. I think it’s a useful book to read, if you’re having trouble organizing yourself. I would just caution against spending all your energy getting organized — leave a little energy for actually doing your work!

I use a super-simple system for keeping track of tasks that are mine personally to manage. I use issue-tracking systems for software projects and consulting work, but there is still a bunch of work-related and personal work that I need to make sure I don’t forget.

The main point is not to ensure that I don’t forget, actually. It is to be able to put it out of my mind with confidence that I won’t lose it. I have a crowded mind, and the cleaner I can keep it, the better.

My system has three parts: my pockets, my notepad, and a directory on my computer.

In my front pocket I have a ballpoint pen. Currently it says Holiday Inn on the side. In my back pocket, I have a small piece of paper — usually about half the size of a standard letter paper, folded small enough to fit. It might be a used envelope, or a napkin, or a piece of actual notebook paper. I write down everything that matters to me. If I hear a song on the radio and I think my wife will like it, I write down some key lyrics I can search later, such as “your arms are my castle, your heart is my sky.” I write down anything I feel guilty about not doing, or neat ideas about stuff I could do, or whatever occurs to me. The goal is to write it down and trust that it’s now permanently in the system, then clear my head.

I do much the same thing with my notebook. I tend to pick these things up at conferences. I use two or three pages a week. A small size, like the size of a paperback book, is best. Legal pads are too big. One of the best pads I ever got was an InnoDB pad. I keep one page for random whatever-comes-to-me. At the beginning of each week, I collate these items; some of them I move off to the directory on my computer, others go into a single page, grouped by importance or topic as I see fit, in the notebook. The page needs to fit everything I’ll do that week. There’s no way I can do more than a page’s worth of things in a week. Typically about half the page is carried over to next week. (I just cross things off as I complete them or move them to the clean page.) This ensures that all of the important and/or urgent things are easy for me to reference, without a bunch of other stuff intruding. I also write down things I do that aren’t in my list — if I jump in and help out on a project, for example, I’ll write that down and then cross it off. This is a good record for my weekly report.

I just came back from a conference, so there are pages and pages of thoughts stimulated from conversations, people to follow up with, thank-you notes to send, and so on. A lot of this is going to be easy to take care of: I’ll just do it if it takes only a second, or move it to my computer for later followup. After I collate and organize, I tear out the old pages, feed them into my weekly report, and throw them away. They are redundant.

In my computer’s home directory, I have a directory to hold text files. Here I hold medium-term and long-term items, things that I want to do “someday” or reference material, project notes, and so on. I name each text file by topic, and there are dozens. I keep these as simple and few as possible. There’s one for music, for example. I looked up those lyrics, and then put the artist and album into todo-music.txt in my directory. Next time I decide to order a batch of CDs, I’ll refer to this list. The names of the files aren’t scientific — I just started out with what seemed right, and changed as I saw the need to; the current files have served well for a long time, so I think it’s stable and useful. I organize the lists in two sections: the top N priority items, and everything else. They are separated by a blank line. There is no need to be fancier, I find. Most things go into the everything-else category.

So in the end, my pockets and one page in the notebook are for capturing ideas as they come to me, another page is for what I decided to prioritize for the week, and the computer is the long-term spillover for things that need to get out of the notebook. This is a lot like levels of cache in a computer. I’m keeping the most important stuff in a compact way, easy to work with. And paper is definitely easier to work with than anything with an ON switch. I have no categories, sorting, tagging, hierarchies, or anything else like that. If it’s a single page the size of my hand, there’s no need.

This system was inspired by multiple attempts to use task lists on computers, personal organizers, the Getting Things Done system, the Seven Habits of Highly Effective People method, and many more. Name your favorite app or method — I’ve probably tried it or something like it. These days it’s pen, paper, and Vim. The system has been working for me for a couple of years with excellent results; I rarely to never forget anything. Although I may deprioritize it, which is effectively the same as saying I’ll never do it, I have the peace of mind that comes from knowing I have ten years or so of ideas I’ll never forget, should I ever find myself with ten years to spare and nothing to do with them.

The Getting Things Done system is very valuable in one specific way for me: capture everything and get it out of the head, to keep the head clear. I can’t overstate how important that is to me.

Next I’ll write about how I get things into the system in a way that lets me also have confidence I’m not losing track of something I’m taking on (or being asked to do).