This chapter is a whistle stop tour of the accumulated wisdom of the free software community, with respect to best practices for portable shell scripting, as encoded in the sources for Autoconf and Libtool, as interpreted and filtered by me.  It is by no means comprehensive -- entire books have been devoted to the subject -- though it is, I hope, authoritative.

Why Use the Bourne Shell?
=========================

Unix has been around for more than thirty years and has splintered into hundreds of small and not so small variants, see The Diversity of Unix Systems.  Much of the subject matter of this book is concerned with how best to approach writing programs which will work on as many of these variants as possible.  One of the few programming tools that is absolutely guaranteed to be present on every flavour of Unix in use today is Steve Bourne's original shell, sh -- the Bourne Shell.  That is why Libtool is written as a Bourne Shell script, and why the configure files generated by Autoconf are Bourne Shell scripts: they can be executed on all known Unix flavours, and as a bonus on most POSIX based non-Unix operating systems too.

However, there are complications.  Over the years, OS vendors have improved Steve Bourne's original shell or have reimplemented it in an almost, but not quite, compatible way.  There also a great number of Bourne compatible shells which are often used as a system's default /bin/sh: ash, bash, bsh, ksh, sh5 and zsh are some that you may come across.  For the rest of this chapter, when I say ``shell'', I mean a Bourne compatible shell.

This leads us to the black art known as portable shell programming, the art of writing a single script which will run correctly through all of these varying implementations of /bin/sh.  Of course, Unix systems are constantly evolving and new variations are being introduced all the time (and very old systems which have fallen into disuse can perhaps be ignored by the pragmatic).  The amount of system knowledge required to write a truly portable shell script is vast, and a great deal of the information that sets a precedent for a given idiom is necessarily second or third (or tenth) hand.  Practically, this means that some of the knowledge accumulated in popular portable shell scripts is very probably folklore -- but that doesn't really matter too much, the important thing is that if you adhere to these idioms, you shouldn't have any problems from people who can't run your program on their system.

Implementation
==============

By their very nature, a sizeable part of the functionality of shell scripts, is provided by the many utility programs that they routinely call to perform important subsidiary tasks.  Addressing the portability of the script involves issues of portability in the host operating system environment, and portability of the utility programs as well as the portability of the shell implementation itself.

This section discusses differences between shell implementations to which you must cater when writing a portable script.  It is broken into several subsections, each covering a single aspect of shell programming that needs to be approached carefully to avoid pitfalls with unexpected behaviour in some shell implementations.  The following section discusses how to cope with the host environment in a portable fashion. The last section in this chapter addresses the portability of common shell utilities.

Size Limitations
----------------

Quite a lot of the Unix vendor implementations of the Bourne shell have a fixed buffer for storing command lines, as small as 512 characters in the worst cases.  You may have an error akin to this:

     $ ls -d /usr/bin/* | wc -l
     sh: error: line too long

Notice that the limit applies to the expanded command line, not just the characters typed in for the line.  A portable way to write this would be:

     $ ( cd /usr/bin && ls | wc -l )
        1556

#!
--

When the kernel executes a program from the file system, it checks the first few bytes of the file, and compares them with its internal list of known magic numbers, which encode how the file can be executed. This is a similar, but distinct, system to the /etc/magic magic number list used by user space programs.

Having determined that the file is a script by examining its magic number, the kernel finds the path of the interpreter by removing the #! and any intervening space from the first line of the script. One optional argument is allowed (additional arguments are not ignored, they constitute a syntax error), and the resulting command line is executed.  There is a 32 character limit to the significant part of the #! line, so you must ensure that the full path to the interpreter plus any switches you need to pass to it do not exceed this limit. Also, the interpreter must be a real binary program, it cannot be a #! file itself.

It used to be thought, that the semantics between different kernels' idea of the magic number for the start of an interpreted script varied slightly between implementations.  In actual fact, all look for #! in the first two bytes -- in spite of commonly held beliefs, there is no evidence that there are others which require #! /.

A portable script must give an absolute path to the interpreter, which causes problems when, say, some machines have a better version of Bourne shell in an unusual directory -- say /usr/sysv/bin/sh.  See () for a way to re-execute the script with a better interpreter.

For example, imagine a script file called /tmp/foo.pl with the following first line:

     #! /usr/local/bin/perl

Now, the script can be executed from the tmp directory, with the following sequence of commands:

     $ cd /tmp
     $ ./foo.pl

When executing these commands, the kernel will actually execute the following from the /tmp directory directory:

     /usr/local/bin/perl ./foo.pl

This can pose problems of its own though.  A script such as the one described above will not work on a machine where the perl interpreter is installed as /usr/bin/perl.  There is a way to circumvent this problem, by using the env program to find the interpreter by looking in the user's PATH environment variable.  Change the first line of the foo.pl to read as follows:

     #! /usr/bin/env perl

This idiom does rely on the env command being installed as /usr/bin/env, and that, in this example, perl can be found in the user's PATH.  But that is indeed the case on the great majority of machines.  In contrast, perl is installed in usr/local/bin as often as /usr/bin, so using env like this is a net win overall.  You can also use this method to get around the 32 character limit if the path to the interpreter is too long.

Unfortunately, you lose the ability to pass an option flag to the interpreter if you choose to use env.  For example, you can't do the following, since it requires two arguments:

     #! /usr/bin/env guile -s

:
-

In the beginning, the magic number for Bourne shell scripts used to be a colon followed by a newline.  Most Unices still support this, and will correctly pass a file with a single colon as its first line to /bin/sh for interpretation.  Nobody uses this any more and I suspect some very new Unices may have forgotten about it entirely, so you should stick to the more usual #! /bin/sh syntax for your own scripts.  You may occasionally come across a very old script that starts with a : though, and it is nice to know why!

In addition, all known Bourne compatible shells have a builtin command, : which always returns success.  It is equivalent to the system command /bin/true, but can be used from a script without the overhead of starting another process.  When setting a shell variable as a flag, it is good practice to use the commands, : and false as values, and choose the sense of the variable to be : in the common case:  When you come to test the value of the variable, you will avoid the overhead of additional processes most of the time.

     var=:
     if $var; then
       foo
     fi

The : command described above can take any number of arguments, which it will fastidiously ignore.  This allows the : character to double up as a comment leader of sorts.  Be aware that the characters that follow are not discarded, they are still interpreted by the shell, so metacharacters can have unexpected effects:

     $ cat foo
     :
     : echo foo
     m4_changequote(,)m4_dnl
     : `echo bar`
     : `echo baz >&2'
     m4_changequote(`,')m4_dnl
     $ ./foo
     baz

You may find very old shell scripts that are commented using :, or new scripts that exploit this behavior in some esoteric fashion.  My advice is, don't:  It will bite you later.

()
--

There are still a great number of shells that, like Steve Bourne's original implementation, do not have functions! So, strictly speaking, you can't use shell functions in your scripts.  Luckily, in this day and age, even though /bin/sh itself may not support shell functions, it is not too far from the truth to say that almost every machine will have some shell that does.

Taking this assumption to its logical conclusion, it is a simple matter of writing your script to find a suitable shell, and then feed itself to that shell so that the rest of the script can use functions with impunity:

     m4_include(examples/reexec.texi)

Note that this script finds a shell that supports functions of the following syntax, since the use of the function keyword is much less widely supported:

     foo () { ... }

A notable exception to the assertion that all machines have a shell that can handle functions is 4.3BSD, which has only a single shell: a shell function deprived Bourne shell.  There are two ways you can deal with this:

 1. Ask 4.3BSD users of your script to install a more featureful shell such as bash, so that the technique above will work.

 2. Have your script run itself through sed, chopping itself into pieces, with each function written to it's own script file, and then feed what's left into the original shell.  Whenever a function call is encountered, one of the fragments from the original script will be executed in a subshell.

If you decide to split the script with sed, you will need to be careful not to rely on shell variables to communicate between functions, since each ``function'' will be executed in its own subshell.

.
-

The semantics of . are rather peculiar to say the least.  Here is a simple script -- it just displays its positional parameters:

     #! /bin/sh
     echo "$0" ${1+"$@"}

Put this in a file, foo.  Here is another simple script -- it calls the first script.  Put this in another file, wrapper:

     #! /bin/sh
     . ./foo
     . ./foo bar baz

Observe what happens when you run this from the command line:

     $ ./wrapper
     ./wrapper
     ./wrapper bar baz

So $0 is inherited from the calling script, and the positional parameters are as passed to the command.  Observe what happens when you call the wrapper script with arguments:

     $ ./wrapper 1 2 3
     ./wrapper 1 2 3
     ./wrapper bar baz

So the sourced script has access to the calling scripts positional parameters, unless you override them in the . command.

This can cause no end of trouble if you are not expecting it, so you must either be careful to omit all parameters to any . command, or else don't reference the parameters inside the sourced script.  If you are reexecuting your script with a shell that understands functions, the best use for the . command is to load libraries of functions which can subsequently be used in the calling script.

Most importantly, don't forget that, if you call the exit command in a script that you load with ., it will cause the calling script to exit too!

[
-

Although technically equivalent, test is preferable to [ in shell code written in conjunction with Autoconf, since [ is also used for M4 quoting in Autoconf.  Your code will be much easier to read (and write) if you abstain from the use of [.

Except in the most degenerate shells, test is a shell builtin to save the overhead of starting another process, and is no slower than [.  It does mean, however, that there is a huge range of features which are not implemented often enough that you can use them freely within a truly portable script.   The less obvious ones to avoid are -a and -o -- the logical ``and'' and ``or'' operations.  A good litmus test for the portability of any shell feature is to see whether that feature is used in the source of Autoconf, and it turns out that -a and -o are used here and there, but never more than once in a single command.  All the same, to avoid any confusion, I always avoid them entirely.  I would not use the following, for example:

     test foo -a bar

Instead I would run test twice, like this:

     test foo && test bar

The negation operator of test is quite portable and can be used in portable shell scripts.  For example:

     if test ! foo; then bar; fi

The negation operator of if is not at all portable and should be avoided.  The following would generate a syntax error on some shell implementations:

     if ! test foo; then bar; fi

An implication of this axiom is that when you need to branch if a command fails, and that command is not test, you cannot use the negation operator.  The easiest way to work around this is to use the else clause of the un-negated if, like this:

     if foo; then :; else bar; fi

Notice the use of the : builtin as a null operation when foo doesn't fail.

The test command does not cope with missing or additional arguments, so you must take care to ensure that the shell does not remove arguments or introduce new ones during variable and quote expansions.  The best way to do that is to enclose any variables in double quotes.  You should also add a single character prefix to both sides in case the value of the expansion is a valid option to test:

     $ for foo in "" "!" "bar" "baz quux"; do
     >   test x"$foo" = x"bar" && echo 1 || echo 0
     > done
     0
     0
     1
     0

Here, you can see that using the x prefix for the first operand saves test from interpreting the ! argument as a real option, or from choking on an empty string -- something you must always be aware of, or else the following behaviour will ensue:

     $ foo=!
     $ test "$foo" = "bar" && echo 1 || echo 0
     test: argument expected
     0
     $ foo=""
     $ test "$foo" = "bar" && echo 1 || echo 0
     test: argument expected
     0

Also, the double quote marks help test cope with strings that contain whitespace.  Without the double quotes, you will see this errors:

     $ foo="baz quux"
     $ test x$foo = "bar" && echo 1 || echo 0
     test: too many arguments
     0

You shouldn't rely on the default behaviour of test (to return ``true'' if its single argument has non-zero length), use the -n option to force that behaviour if it is what you want.  Beyond that, the other thing you need to know about test, is that if you use operators other than those below, you are reducing the portability of your code:

-n STRING
     STRING is non-empty.

-z STRING
     STRING is empty.

STRING1 = STRING2
     Both strings are identical.

STRING1 != STRING2
     The strings are not the same.

-d FILE
     FILE exists and is a directory.

-f FILE
     FILE exists and is a regular file.

You can also use the following, provided that you don't mix them within a single invocation of test:

EXPRESSION -a EXPRESSION
     Both expressions evaluate to ``true''.

EXPRESSION -o EXPRESSION
     Neither expression evaluates to ``false''.

$
-

When using shell variables in your portable scripts, you need to write them in a somewhat stylised fashion to maximise the number of shell implementations that will interpret your code as expected:

  * Convenient though it is, the POSIX $(command parameters) syntax for command substitution is not remotely portable.  Despite it being more difficult to nest, you must use m4_changequote(,)m4_dnl `command parameters` instead. m4_changequote(`,')m4_dnl

  * The most portable way to set a default value for a shell variable is:

          $ echo ${no_such_var-"default value"}
          default value

     If there is any whitespace in the default value, as there is here, you must be careful to quote the entire value, since some shells will raise an error:

          $ echo ${no_such_var-default value}
          sh: bad substitution

  * The unset command is not available in many of the degenerate Bourne shell implementations.  Generally, it is not too difficult to get by without it, but following the logic that led to the shell script in (), it would be trivial to extend the test case for confirming a shell's suitability to include a check for unset. Although it has not been put to the test, the theory is that all the interesting machines in use today have some shell that supports unset.

  * Be religious about double quoting variable expansions.  Using "$foo" will avoid trouble with unexpected spaces in filenames, and compression of all whitespace to a single space in unquoted variable expansions.

  * To avoid accidental interpretation of variable expansions as command options you can use the following technique:

          $ foo=-n
          $ echo $foo
          $ echo x"$foo" | sed -e 's/^x//'
          -n

  * If it is set, IFS splits words on whitespace by default.  If you change it, be sure to put it back when you're done, or the shell may behave very strangely from that point.  For example, when you need to examine each element of $PATH in turn:

          # The whitespace at the end of the following line is a space
          # followed by literal tab and newline characters.
          save_IFS="${IFS=
          }"; IFS=":"
          set dummy $PATH
          IFS="$save_IFS"
          shift

     Alternatively, you can take advantage of the fact that command substitutions occur in a separate subshell, and do not corrupt the environment of the calling shell:

          set dummy `IFS=:; echo $PATH`
          shift

     Strictly speaking, the dummy argument is required to stop the set command from interpreting the first word of the expanded backquote expression as a command option. Realistically, no one is going to have -x, for example, as the first element of their PATH variable, so the dummy could be omitted -- as I did earlier in the script in ().

  * Some shells expand $@ to the empty string, even when there are no actual parameters ($# is 0).  If you need to replicate the parameters that were passed to the executing script, when feeding the script to a more suitable interpreter for example, you must use the following:

          ${1+"$@"}

     Similarly, although all known shells do correctly use $@ as the default argument to a for command, you must write it like this:

          for arg
          do
            stuff
          done

     When you rely on implicit $@ like this, it is important to write the do keyword on a separate line.  Some degenerate shells can not parse the following:

          for arg; do
            stuff
          done

* versus .*
-----------

This section compares file globbing with regular expression matching.   There are many Unix commands which are regularly used from shell scripts, and which provide some sort of pattern matching mechanism: expr, egrep and sed, to name a few.  Unfortunately they each have different quoting rules regarding whether particular meta-characters must be backslash escaped to revert to their literal meaning and vice-versa.  There is no real logic to the particular dialect of regular expressions accepted by these commands.  To confirm the correctness of each regular expression, you should always check them from the shell prompt with the relevant tool before committing to a script, so I won't belabour the specifics.

Shell globbing however is much more regular (no pun intended), and provides a reasonable and sometimes more cpu efficient solution to many shell matching problems.  The key is to make good use of the case command, which is easier to use (because it uses globbing rules) and doesn't require additional processes to be spawned. Unfortunately, GNU Bash doesn't handle backslashes correctly in glob character classes -- the backslash must be the first character in the class, or else it will never match.  For example, if you want to detect absolute directory paths on Unix and Windows using case, you should write the code like this:

     case $dir in
       [\\/]* | ?:[\\/]* ) echo absolute ;;
       * )                 echo relative ;;
     esac

Even though expr uses regular expressions rather than shell globbing, it is often (Notable exceptions are GNU Bash, and both Ksh and the Bourne shell on Solaris.) a shell builtin, so using it to extract sections of strings can be faster than spawning a sed process to do the same.  As with echo and set, for example, you must be careful that variable or command expansions for the first argument to expr are not accidentally interpreted as reserved keywords.   As with echo, you can work around this problem by prefixing any expansions with a literal x, as follows:

     $ foo=substr
     $ expr $foo : '.*\(str\)'
     expr: syntax error
     $ expr x$foo : '.*\(str\)'
     str

Environment
===========

In addition to the problems with portability in shell implementations discussed in the previous section, the behaviour of the shell can also be drastically affected by the contents of certain environment variables, and the operating environment provided by the host machine.

It is important to be aware of the behavior of some of the operating systems within which your shell script might run.  Although not directly related to the implementation of the shell interpreter, the characteristics of some of target architectures do influence what is considered to be portable.  To ensure your script will work on as many shell implementations as possible, you must observe the following points.

SCO Unix doesn't like LANG=C and friends, but without LC_MESSAGES=C, Solaris will translate variable values in set!  Similarly, without LC_CTYPE=C, compiled C code can behave unexpectedly.  The trick is to set the values to C, except for if they are not already set at all:

     for var in LANG LC_ALL LC_MESSAGES LC_CTYPES LANGUAGES
     do
       if eval test x"\${$var+set}" = xset; then
         eval $var=C; eval export $var
       fi
     done

HP-UX ksh and all POSIX shells print the target directory to standard output if CDPATH is set.

     if test x"${CDPATH+set}" = xset; then CDPATH=:; export CDPATH; fi

The target architecture file system may impose limits on your scripts. IF you want your scripts to run on the architectures which impose these limits, then your script must adhere to these limits:

  * The ISO9660 filesystem, as used on most CD-ROMs, limits nesting of directories to a maximum depth of twelve levels.

  * Many old Unix filesystems place a 14 character limit on the length of any filename.  If you care about portability to DOS, that has an 8 character limit with an optional extension of 3 or fewer characters (known as 8.3 notation).

A useful idiom when you need to determine whether a particular pathname is relative or absolute, which works for DOS targets to follows:

     case "$file" in
       [\\/]* | ?:[\\/]*) echo absolute ;;
       *)                 echo default ;;
     esac

Utilities
=========

The utility programs commonly executed by shell scripts can have a huge impact on the portability of shell scripts, and it is important to know which utilities are universally available, and any differences certain implementations of these utilities may exhibit.  According to the GNU standards document, you can rely on having access to these utilities from your scripts:

     cat cmp cp diff echo egrep expr false grep install-info
     ln ls mkdir mv pwd rm rmdir sed sleep sort tar test touch true

Here are some things that you must be aware of when using some of the tools listed above:

cat
     Host architectures supply cat implementations with conflicting interpretations of, or entirely missing, the various command line options.  You should avoid using any command line options to this command.

cp and mv
     Unconditionally duplicated or otherwise open file descriptors can not be deleted on many operating systems, and worse on Windows the destination files cannot even be moved.  Constructs like this must be avoided, for example.

          exec > foo
          mv foo bar

echo
     The echo command has at least two flavors:  the one takes a -n option to suppress the automatic newline at the end of the echoed string; the other uses an embedded \c notation as the last character in the echoed string for the same purpose.

     If you need to emit a string without a trailing newline character, you can use the following script fragment to discover which flavor of echo you are using:

          case `echo "testing\c"`,`echo -n testing` in
            *c*,-n*) echo_n=   echo_c=' (This is a literal newline.)
          ' ;;
            *c*,*)   echo_n=-n echo_c= ;;
            *)       echo_n=   echo_c='\c' ;;
          esac

     Any echo command after the shell fragment above, which shouldn't move the cursor to a new line, can now be written like so:

          echo $echo_n "prompt:$echo_c"

     In addition, you should try to avoid backslashes in echo arguments unless they are expanded by the shell.  Some implementations interpret them and effectively perform another backslash expansion pass, where equally many implementations do not.  This can become a really hairy problem if you need to have an echo command which doesn't perform backslash expansion, and in fact the first 150 lines of the ltconfig script distributed with Libtool are devoted to finding such a command.

ln
     Not all systems support soft links.  You should use the Autoconf macro AC_PROG_LN_S to discover what the target architecture supports, and assign the result of that test to a variable.  Whenever you subsequently need to create a link you can use the command stored in the variable to do so.

          LN_S=@LN_S@
          ...
          $LN_S $top_srcdir/foo $dist_dir/foo

     Also, you cannot rely on support for the -f option from all implementations of ln.  Use rm before calling ln instead.

mkdir
     Unfortunately, mkdir -p is not as portable as we might like.  You must either create each directory in the path in turn, or use the mkinstalldirs script supplied by Automake.

sed
     When you resort to using sed (rather, use case or expr if you can), there is no need to introduce command line scripts using the -e option.  Even when you want to supply more than one script, you can use ; as a command separator.  The following two lines are equivalent, though the latter is cleaner:

          $ sed -e 's/foo/bar/g' -e '12q' < infile > outfile
          $ sed 's/foo/bar/g;12q' < infile > outfile

     Some portability zealots still go to great lengths to avoid here documents of more than twelve lines.  The twelve line limit is actually a limitation in some implementations of sed, which has gradually seeped into the portable shell folklore as a general limit in all here documents.  Autoconf, however, includes many here documents with far more than twelve lines, and has not generated any complaints from users.  This is testament to the fact that at worst the limit is only encountered in very obscure cases -- and most likely that it is not a real limit after all.

     Also, be aware that branch labels of more than eight characters are not portable to some imPlementations of sed.

     Here documents are a way of redirecting literal strings into the standard input of a command.  You have certainly seen them before if you have looked at other peoples shell scripts, though you may not have realised what they were called:

          cat >> /tmp/file$$ << _EOF_
          This is the text of a "here document"
          _EOF_

Something else to be aware of is that the temporary files created by your scripts can become a security problem if they are left in /tmp or if the names are predictable.  A simple way around this is to create a directory in /tmp that is unique to the process and owned by the process user.  Some machines have a utility program for just this purpose -- mktemp -d -- or else you can always fall back to umask 077 && mkdir /tmp/$$.  Having created this directory, all of the temporary files for this process should be written to that directory, and its contents removed as soon as possible.

Armed with the knowledge of how to write shell code in a portable fashion as discussed in this chapter, in combination with the M4 details from the last chapter, the specifics of combining the two to write your own Autoconf macros are covered in the next chapter.