Nothing Undone

A weblog, by Peter Jaros.

Process Substitution: The Coolest Shell Trick You Didn't Know

The syntax in the blog post is compatible with bash and zsh. With other shells, YMMV.

Well, maybe you know about it. I didn’t. It amazes me that after all this time the shell can still surprise me.

You probably know about the man command. I sure hope you do. Give it a topic, and it’ll tell you all about it. The topics are usually commands. For instance, ssh is a fairly complex and full-featured command. I wonder how it works. Let’s find out:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$ man ssh
SSH(1)                    BSD General Commands Manual                   SSH(1)

NAME
     ssh -- OpenSSH SSH client (remote login program)

SYNOPSIS
     ssh [-1246AaCfgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]
         [-D [bind_address:]port] [-e escape_char] [-F configfile] [-I pkcs11]
         [-i identity_file] [-L [bind_address:]port:host:hostport] [-l login_name]
         [-m mac_spec] [-O ctl_cmd] [-o option] [-p port]
         [-R [bind_address:]port:host:hostport] [-S ctl_path] [-W host:port]
         [-w local_tun[:remote_tun]] [user@]hostname [command]

DESCRIPTION
     ssh (SSH client) is a program for logging into a remote machine and for executing com-
     mands on a remote machine.  It is intended to replace rlogin and rsh, and provide secure
     encrypted communications between two untrusted hosts over an insecure network.  X11 con-
     nections and arbitrary TCP ports can also be forwarded over the secure channel.

Awesome, everything we need to know about ssh. But sometimes you don’t know what command to look up. Maybe we just know that we want to log in remotely. How would we find the right command for that?

Enter whatis:

1
2
3
4
$ whatis "remote login"
rlogin(1)                - remote login
rlogind(8)               - remote login server
ssh(1)                   - OpenSSH SSH client (remote login program)

Cool, we even see that there’s an older remote login system we can look into.

That’s cool, but what’s this process substitution thing?

Okay, okay. I knew about whatis. I also knew about a command called apropos. In fact, my understanding the two was the same; I wasn’t sure what the difference was. They both searched for man pages based on keywords. So I looked at man whatis and man apropos. Very similar entries. (Go ahead, take a look for yourself.)

I wondered, is there a good way to see the differences between these pages? Well, diff should do that for us. But diff wants to take two files. We can pipe text into it, but that can only serve as one of the files; you can’t pipe two things into the same program.

Or can you?

In the past, I would have redirected the man pages to files and diffed them.

1
2
3
4
$ man whatis > whatis.man
$ man apropos > apropos.man
$ diff whatis.man apropos.man
$ rm whatis.man apropos.man

But that’s messy. There’s a cleaner way, and process substitution is it. Check it out.

1
$ diff <(man whatis) <(man apropos)

The shell will now do essentially what we did above, only using file descriptors (which avoids writing to the disk) instead of files, if possible. The <(...) tokens will be replaced with the name of a file descriptor from which diff can read the output of the command inside the parentheses.

Making the diff even better.

That’s all I’ve got on process substitution, but here’s some more awesome. It’s still not dead obvious from that output what’s different between the two. Partly, that’s because man typesets its output to be fully-justified, which involves fiddling with the whitespace between words. Lines with the same words can end up with different whitespace. We can ignore whitespace with diff -w, but we can do even better than that.

Let’s install wdiff. That’s a word-based diff, rather than a line-based diff. While we’re at it, let’s install colordiff, which can colorize diff output, to really make things clear. On a Mac with Homebrew, that’s:

1
2
$ brew install wdiff
$ brew install colordiff

Now we’re going to run wdiff on the two man pages, and pass the result to colordiff to make it pretty. (The -n makes it easier for colordiff to understand the output; see man wdiff and man colordiff for more info.)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
$ wdiff -n <(man whatis) <(man apropos) | colordiff
[-whatis(1)                                                            whatis(1)-]{+apropos(1)                                                          apropos(1)+}



NAME
       [-whatis-]
       {+apropos+} - search the whatis database for [-complete words.-] {+strings+}

SYNOPSIS
       [-whatis-]
       {+apropos+} keyword ...

DESCRIPTION
       [-whatis-]
       {+apropos+}  searches a set of database files containing short descriptions
       of system commands for keywords and displays the result on the standard
       output.  [-Only complete word matches are displayed.-]
[--]
[-       The  whatis  database  is  created using the command /usr/libexec/make--]
[-       whatis.-]

AUTHOR
       John  W.  Eaton  was  the  original  author  of man.  Zeyd M. Ben-Halim
       released man 1.2, and Andries Brouwer followed  up  with  versions  1.3
       thru  1.5p.   Federico  Lucifredi  <flucifredi@acm.org>  is the current
       maintainer.

SEE ALSO
       [-apropos(1),-]
       {+whatis(1),+} man(1).



                              September 19, 2005                     [-whatis(1)-]                    {+apropos(1)+}

I can’t get my blog engine to display the color, so just pretend it’s there, or try it for yourself.

Ah, got it: whatis searches for complete words, apropos searches for any substring matches.

Now you know.

Comments