One of the things I preach about a lot is good monitoring of your database servers; having tools in place to tell you both what good looks like and when things go bad is critical for large scale success. But sometimes you just need to monitor a momenta…

(echo foss.org.my; echo http://www.debian.org; echo http://www.freenetproject.org) | parallel traceroute

In my personal opinion this is easier to read:

cat test.data | parallel ./test.sh

than this:

cat test.data | xargs -L1 -P5 sh -c ‘./test.sh $* | sed “s/^/$$:/”‘ -

Parallel also deals nicely with filenames containing obscure characters (space quotes tabs parenthesis greater-than less-than and the likes) – even without -print0.

Parallel can run no_of_cpus jobs in parallel (use -j+0).

Parallel can keep the order of the output, so output of the second job can be postponed till the first job is done (use -k).

Parallel has support for context replace, so you create the arguments from a template like pict{}.jpg

stdbuf -oL ./test.sh

However that will not work for commands that don’t use stdio, where was your sed tip will

Here’s another example & workaround:
cat <test.sh
#!/bin/bash
count=`echo $* | wc -w`
sleep `expr $count % 10` #simulate processing
echo -n $count ” “
for w in $*; do
count=`echo -n $w | wc -c`
sleep `expr $count % 10` # simulate more processing
echo -n $count ” “
done
echo $*
EOF

cat < test.data
testing 123
the quick brown fox
jumps over the lazy dog
hello world
foo bar baz
EOF

Run 1: no parallelism; process data line by line

$ time cat test.data | xargs -L1 ./test.sh
2 7 3 testing 123
4 3 5 5 3 the quick brown fox
5 5 4 3 4 3 jumps over the lazy dog
2 5 5 hello world
3 3 3 3 foo bar baz

real 1m20.253s
user 0m0.060s
sys 0m0.180s

Run 2: First try at parallelism; runs faster, but output isn’t usable

$ time cat test.data | xargs -L1 -P5 ./test.sh
2 2 3 4 5 3 3 5 7 3 5 5 3 testing 123
5 hello world
3 foo bar baz
4 5 3 3 the quick brown fox
4 3 jumps over the lazy dog

real 0m24.096s
user 0m0.059s
sys 0m0.178s

Run 3: Use sed to buffer line output, prepend each line w/ pid that processed the line

$ time cat test.data | xargs -L1 -P5 sh -c ‘./test.sh $* | sed “s/^/$$:/”‘ –
88984:2 7 3 testing 123
88987:2 5 5 hello world
88988:3 3 3 3 foo bar baz
88985:4 3 5 5 3 the quick brown fox
88986:5 5 4 3 4 3 jumps over the lazy dog

real 0m24.112s
user 0m0.064s
sys 0m0.192s

Unlike the perl “parallel” or “annotate” workarounds, using sed doesn’t handle the stderr problem, but you could easily write a similar wrapper script which writes stderr to a tmp file and buffer stdout via sed.