SSH Multiplexer: parallel-ssh //at 03:10 //by abe

There are many SSH multiplexers in Debian and most of them have one or two features which make them unique and especially useful for that one use case. I use some of them regularily (I even maintain the Debian package of one of them, namely pconsole :-) and I’ll present then and when one of them here.

For non-interactive purposes I really like parallel-ssh aka pssh. It takes a file of hostnames and a bunch of common ssh parameters as parameters, executes the given command in parallel in up to 32 threads (by default, adjustable with -p) and waits by default for 60 seconds (adjustable with -t). For example to restart hobbit-client on all hosts in kiva.txt, the following command is suitable:

$ parallel-ssh -h kiva.txt -l root /etc/init.d/hobbit-client restart
[1] 19:56:03 [FAILURE] kiva6 Exited with error code 127
[2] 19:56:04 [SUCCESS] kiva
[3] 19:56:04 [SUCCESS] kiva4
[4] 19:56:04 [SUCCESS] kiva2
[5] 19:56:04 [SUCCESS] kiva5
[6] 19:56:04 [SUCCESS] kiva3
[7] 19:57:03 [FAILURE] kiva1 Timed out, Killed by signal 9

(Coloured “Screenshots” done with ANSI HTML Adapter from the package aha.)

You easily see on which hosts the command failed and partially also why: On kiva6 hobbit-client is not installed and therefore the init.d script is not present. kiva1 is currently offline so the ssh connection timed out.

If you want to see the output of the commands, you have a two choices. Which one to choose depends on the expected amount of output:

If you don’t expect a lot of output, the -i (or --inline) option for inline aggregated output is probably the right choice:

$ parallel-ssh -h kiva.txt -l root -t 10 -i uptime
[1] 20:30:20 [SUCCESS] kiva
 20:30:20 up 7 days,  5:51,  0 users,  load average: 0.12, 0.08, 0.06
[2] 20:30:20 [SUCCESS] kiva2
 20:30:20 up 7 days,  5:50,  0 users,  load average: 0.19, 0.08, 0.02
[3] 20:30:20 [SUCCESS] kiva3
 20:30:20 up 7 days,  5:49,  0 users,  load average: 0.10, 0.06, 0.06
[4] 20:30:20 [SUCCESS] kiva4
 20:30:20 up 7 days,  5:49,  0 users,  load average: 0.25, 0.17, 0.14
[5] 20:30:20 [SUCCESS] kiva6
 20:30:20 up 7 days,  5:49, 10 users,  load average: 0.16, 0.08, 0.02
[6] 20:30:21 [SUCCESS] kiva5
 20:30:21 up 7 days,  5:49,  0 users,  load average: 3.11, 3.36, 3.06
[7] 20:30:29 [FAILURE] kiva1 Timed out, Killed by signal 9

If you expect a lot of output you can give directories with the -o (or --outdir) and -e (or --errdir) option:

$ parallel-ssh -h kiva.txt -l root -t 20 -o kiva-output lsb_release -a
[1] 20:36:51 [SUCCESS] kiva
[2] 20:36:51 [SUCCESS] kiva2
[3] 20:36:51 [SUCCESS] kiva3
[4] 20:36:51 [SUCCESS] kiva4
[5] 20:36:53 [SUCCESS] kiva6
[6] 20:36:54 [SUCCESS] kiva5
[7] 20:37:10 [FAILURE] kiva1 Timed out, Killed by signal 9
$ ls -l kiva-output
total 24
-rw-r--r-- 1 abe abe  98 Aug 28 20:36 kiva
-rw-r--r-- 1 abe abe   0 Aug 28 20:36 kiva1
-rw-r--r-- 1 abe abe  98 Aug 28 20:36 kiva2
-rw-r--r-- 1 abe abe  98 Aug 28 20:36 kiva3
-rw-r--r-- 1 abe abe  98 Aug 28 20:36 kiva4
-rw-r--r-- 1 abe abe 102 Aug 28 20:36 kiva5
-rw-r--r-- 1 abe abe 100 Aug 28 20:36 kiva6
$ cat kiva-output/kiva5
Distributor ID:	Debian
Description:	Debian GNU/Linux 6.0.2 (squeeze)
Release:	6.0.2
Codename:	squeeze

The only annoying thing IMHO is that the host list needs to be in a file. With zsh, bash and the original ksh (but neither tcsh, pdksh nor mksh), you can circumvent this restriction with one of the following command lines:

$ parallel-ssh -h <(printf "host1\nhost2\nhost3\n…") -l root uptime
[…]
$ parallel-ssh -h <(echo host1 host2 host3 … | xargs -n1) -l root uptime
[…]

And in zsh there’s an even easier way to type this:

$ parallel-ssh -h <(print -l host1 host2 host3 …) -l root uptime
[…]

In addition to parallel-ssh the pssh package also contains some more ssh based tools:

• parallel-scp and parallel-rsync for parallel copying files onto a set of hosts.
• parallel-slurp for fetching files in parallel from a list of hosts.
• parallel-nuke to kill a bunch of processes in parallel on a set of machines.

I though think that parallel-ssh is by far the most useful tool from the pssh package. (Probably no wonder as it’s the most generic one. :-)