Pipes and filters

Pipes and filters is a software design pattern invented by Douglas McIlroy. Its inclusion on the Unix operating system was the first implementation of his vision of software componentry. The main difference between pipes and filters and newer software component technologies is that it only allows information to flow in one direction, like water flows in a pipe.

In the context of Unix operating systems, a pipe signifies that the output of one program feeds directly as input to another program. The Unix shell uses the pipe character (|) to join programs together. A sequence of commands joined together by pipes is known as a pipeline. For creating this mechanism, all Unix tools have access to three distinct files:

• stdin the standard input file
• stdout the standard output file
• stderr the standard error file

By joining one tool's stdout to another tool's stdin, a pipeline is formed. Errors are sent to a side track and accumulated.

Often filter programs form the constituent programs in a pipeline.

An example of a pipeline:

cat * | grep "alice" | grep -v "wonderland" | wc -l

will print out the number of lines in all files in the current directory which contain the text "alice", but not the text "wonderland".

The pipeline has four parts:

• cat * concatenates the text of all files to its stdout;
• grep "alice" reads its stdin as lines, and prints on its stdout only those lines which contain the word "alice";
• grep -v "wonderland" reads its stdin and prints on its stdout only those remaining lines which do not contain the word "wonderland";
• wc -l counts the lines on its stdin, and prints a line count on its stdout.

Pipes and filters can be viewed as a form of functional programming, using byte streams as data objects.