Xaprb

Have you ever run a command on a Unix-like system and then realized it’ll take a really long time, and you want to log out of the computer and leave it running? Or maybe you’ve connected over SSH to a remote server and executed a command that will take a while, and you’re really worried that your network connection will fail and kill the command.

There are several ways to log out without stopping the command. Let’s take a look.

Method 1: Job Control

When I learned about job control in the bash shell, I was in the middle of an AI project on my computer science department’s servers. I had written a LISP program that simulated a little robot going from place to place delivering parts to an assembly line. The challenge was that he had to figure out the optimal route to do so, all the while never running out of fuel. (As it turns out, he was cursed by my professor with a small fuel tank.) The experiment was going to take at least 3 days — he had a lot of learning to do and a lot of parts to deliver. But I didn’t realize this until I checked the next morning and saw his progress. What to do?

After reading on the web, I learned that I could press Control-Z to suspend the job, and then ‘bg’ to make it run in the background! Success! Then I logged out. And my job died. FAIL. Moral: test first.

Why did it die? Because when I logged out, the shell sent the SIGHUP signal to all of its jobs.

There’s a way to do this, though: press Control-Z, type ‘bg’, and then type ‘disown %1′. Now the job won’t get the SIGHUP signal. Here’s a demo:

baron@kanga:~$ run_forrest_run 
CTRL-Z
[1]+  Stopped                 run_forrest_run
baron@kanga:~$ bg
[1]+ run_forrest_run &
baron@kanga:~$ disown %1
baron@kanga:~$

That’s method one: disown the job. It has a variety of shortcomings, though. Let’s see what Method 2 has to offer.

Method 2: nohup

Method 2 is to use the nohup command. This starts the program in the background and directs its output to a file called nohup.out. Now you can log out and come back later.

What’s the difference? Whereas disown is a job control feature that is part of the bash shell, nohup is a built-in standard program. It also takes care of saving the program’s output for you. You can come back to it later and tail the file to see what your program has been telling you.

Demo time:

baron@kanga:~$ nohup run_forrest_run 
nohup: ignoring input and appending output to `nohup.out'
CTRL-Z
[1]+  Stopped                 nohup run_forrest_run
baron@kanga:~$ bg
[1]+ nohup run_forrest_run &
baron@kanga:~$ 

I consider both of these really crude, though. For example, after you log back in, how do you attach your terminal to the program’s standard input to type answers if it wants to ask you questions? This is just one thing that’s not ideal. You know what’s elegant? Method 3.

Method 3: screen

GNU Screen is the bomb. There is only one thing it doesn’t do for me, and I’ll talk about that in another post. Before I tell you about other things, let me paste some text from the man page:

Screen is a full-screen window manager that multiplexes a physical terminal between several processes (typically interactive shells). Each virtual terminal provides the functions of a DEC VT100 terminal and, in addition, several control functions from the ISO 6429 (ECMA 48, ANSI X3.64) and ISO 2022 standards (e.g. insert/delete line and support for multiple character sets). There is a scrollback history buffer for each virtual terminal and a copy-and-paste mechanism that allows moving text regions between windows.

If that sounds intimidating, it ought to — screen has a really absurd amount of functionality and can take a long time to learn. I confess that I am not familiar with about 90% of what it can do. Every so often someone shows me something new and I feel humble. It’s a lot like vim: ten years on, and I still know only a little about it. I do know I can’t live without it.

Enough about that: how can it run commands after you log out?

Simple. Just type “screen -R -D” and then start working as you normally would. When you’re ready to quit, you can detach with “Control-A D”. Screen keeps running. When you log in again later, you can type “screen -R -D” and re-attach to the screen session. I would show you a screenshot of this, but it’s hard to do — you’ll see why if you type the commands yourself. A screencast would be the only way to do a decent demo, and I’m too lazy.

That’s not all screen can do: if your network connection goes down, screen doesn’t die. You only detached, you didn’t kill it. It keeps running in the background and you can re-attach to it after your network connection resumes. When I’m working on a client’s systems, I start screen by default! That way if something happens, I can get back to where I was. I can also log everything in my session to a file with “Control-A H” and other people can log in and share the connection to watch me or help me with “screen -xx”.

Summary

So there you have it: good, better, best. I still use the first two methods sometimes when I run a command and then realize I should have started a screen session (or when screen isn’t installed, heaven forbid) but in general, screen is the arrow I’m always pulling out of the quiver.

By the way I’m intentionally leaving out some more shell features, such as starting a program in the background from the get-go with &. If you want to fill that in, leave a comment — I just wanted to keep this article on topic.

Yesterday I went to beCamp 2008 along with four roomfuls of other people interested in technology (perhaps close to 100 people total). The conference was a lot of fun. Not everything went as planned, but that was as planned. This was an Open Spaces conference and I thought it worked very well. From an email Eric Pugh sent:

Basically it all boils down to:

Open Space is the Law of Two Feet: if anyone finds themselves in a place where they are neither learning nor contributing they should move to somewhere more productive. And from the law flow four principles:

• Whoever comes are the right people
• Whatever happens is the only thing that could have
• Whenever it starts is the right time
• When it’s over, it’s over

From Hadoop to Bang-Splat

I used the law of two feet a time or two. In fact, the first session I wanted to go to, which was about Hadoop and MapReduce, had no knowledgeable attendees. Someone overslept. OK, that’s the way it goes: move along.

From there I went to a session about Unix command-line productivity. Most of the sessions I saw were traditional in that they had one person standing up talking and many people sitting and listening, but not all. This one had several clever command-line gurus mentioning their favorite power tips.

I learned about bang-splat and bang-dollar. The bangs have always gotten me in Bash: I avoid them because I’ve never felt like reading the Bash man page section on them. (Am I too lazy, or not lazy enough?) So it was great to hear some people say “bang-splat and bang-dollar are great” and then explain them. That was easy for me, and now I know how they can be useful to me.

This problem-first type of tip is great for me: tell me the problem, then how to solve it, rather than telling me what the solution is and leaving me guessing what kinds of problems I can solve with it. (The Bash man page is solution-first).

In case you’re wondering, bang-splat substitutes the arguments to the last command, and bang-dollar substitutes the last argument of the last command. So, instead of this:

$ touch file1 file2 file3
$ rm file1 file2 file3

I can do this:

$ touch file1 file2 file3
$ rm !*

There were lots of other nice tips too.

MySQL Performance

I ended up doing a talk on MySQL performance basics. I had no idea what the audience was looking for, so I winged it. I did make some slides, but most of the talk isn’t on the slides. You can get the slides from Percona’s slide page. It seemed to be useful to the folks attending, who had a wide variety of experience and knowledge about MySQL.

Cloud Computing

This session began with a demo of how to create an entire application stack in a few minutes with Cohesive Flexible Technologies. Someone else then demoed a similar thing using RightScale. rPath’s Jeff Uphoff was also in the room, but we didn’t get to see a demo of that. During this session the talk turned to various topics including a little bit of the topics I wanted to hear about in the Hadoop session.

Lunch

Lunch was catered Indian food provided by the Rimm-Kaufman Group. Yum.

Large Scale Storage

This session was sort of a round-table. The two people who talked the most were Josh Malone from the National Radio Astronomy Observatory and the Library of Congress, both of whom have a lot of storage needs they are unsure how to meet. Some people from UVA’s library were there as well, but I didn’t ask what they were working on.

This reminded me a lot of a recent keynote Jacek Becla gave at another conference. He’s with the Stanford Linear Accelerator Center, who are going to be generating a lotta data pretty soon.

High Availability Linux

This one started off with more from Josh Malone, who demoed Nagios briefly and then talked about his storage and backup systems. He uses BackupPC, which sounds pretty neat and very smart. We then talked about some of the things he’s looking into doing, with audience suggestions to look into shared storage or DRBD. We also looked at UltraMonkey briefly — it looks like it’s stagnating, though. And the Linux HA project.

Google App Engine

Finally, someone showed us a calculator application they’d built on Google App Engine, including the code and talking about the data model somewhat. It looks like a neat idea, but the lock-in worries me, a sentiment that was voiced by many others in the room.