CMPSC 311, Shell Scripts, part 1

CMPSC 311, Introduction to Systems Programming

Shell Scripts, Part 1

References

Large parts of this presentation are taken from the Solaris man pages for sh and test, but with added examples.

A shell script is a plain text file that is interpreted line-by-line by a command interpreter (a shell).

The same commands would be acceptable if used interactively.

Put the shell script in a file named xyz, or xyz.sh for emphasis.

Assuming the file name xyz, run the script with the command

sh xyz

Or, change the file permissions (set the executable bit)

chmod +x xyz

and run the script directly

xyz

Common shells

sh	Bourne shell, the Posix standard
csh	C shell
tcsh	Tenex C shell
ksh	Korn shell
bash	Bourne-Again SHell (GNU)

Bash is sh-compatible, with features from ksh and csh.

The examples here were run on Solaris unless stated otherwise. The command prompt % is from tcsh, while the command prompt $ is from sh. We usually add blank lines and highlight commands for clarity.

Example

% sh

$ ls

dir1   file1  file2

        

        $ exit

        

        %

The Bourne Shell (sh)

Execution of commands

Input to the shell will be treated as one of

a function call (executed within the shell)
a special command (executed within the shell)
some program (executed as a child process via fork(), exec() and wait())

Commands return a value, which is an exit status.

Definitions

A blank is a tab or a space.
A word is a non-empty sequence of non-blank characters.
A name is a sequence of ASCII letters, digits, or underscores, beginning with a letter or an underscore.
A parameter is a name, a digit, or any of the characters *, @, #, ?, -, $, and !.
A sub-shell is another instance of the shell, run as a child process of the current shell.

Comments

A word beginning with # causes that word and all the following characters up to a newline to be ignored.

Simple-command

A simple-command is a sequence of non-blank words separated by blanks.
The first word specifies the name of the command to be executed.
The remaining words are passed as arguments to the invoked command.
The value of a simple-command is its exit status if it terminates normally, or 128+status if it terminates abnormally.

See <signal.h> for a list of status values.

Command

A command is either a simple-command or one of the following,

Conditionals

if-then-else
case selection with pattern matching

Loops

for
while
until

Unless otherwise stated, the value returned by a command is that of the last simple-command executed in the command.

Pipeline

A pipeline is a sequence of one or more commands separated by | (vertical bar).
The standard output of each command but the last is connected by a pipe to the standard input of the next command.

See the Interprocess Communication notes for some details of how this is implemented.

Each command is run as a separate process.
Each command terminates after its standard input reaches end-of-file (that is, the previous command closed its standard output).
The shell waits for the last command to terminate.
The exit status of a pipeline is the exit status of the last command in the pipeline.

List

A list is a sequence of one or more pipelines separated by ;, &, &&, or ||, and optionally terminated by ; or &.

; and & have equal precedence, which is lower than that of && and ||.

&& and || have equal precedence.

A semicolon (;) causes sequential execution of the preceding pipeline, that is, the shell waits for the pipeline to finish before executing any commands following the semicolon.

An arbitrary number of newlines may appear in a list, instead of semicolons, to delimit commands.

An ampersand (&) causes asynchronous execution of the preceding pipeline, that is, the shell does not wait for that pipeline to finish.

This indicates a background process (more later).

The symbol && causes the list following it to be executed only if the preceding pipeline returns a zero exit status.
The symbol || causes the list following it to be executed only if the preceding pipeline returns a non-zero exit status.

Grouping

( list )

Execute list in a sub-shell.

{ list ;}

Execute list in the current shell.

The { must be followed by a space.

Functions

name () { list ;}

Define a function which is referenced by name.
The body of the function is the list of commands between { and }.
The { must be followed by a space.
The { and } can be omitted if the body of the function is a single command as defined above, but you should avoid doing this.

Functions are executed in the context of the shell, not in a sub-shell.
Use the special command return or return n to exit the function with return status n. If n is omitted, the return status is that of the last command executed in the function.

Quoting

backslash-quoted single characters

As you would expect, mostly (but the set of characters requiring \ is not the same set as in C).
You may need to quote ; & ( ) | ^ < > newline space tab
For line continuation, end the line with a backslash \

single-quoted strings

take the string as-is
do not use ' inside the string (backslash has no special meaning in this context)

double-quoted strings

do parameter substitution and command substitution inside the quotes, otherwise as-is

Sometimes you need the empty strings '' and "".

Command Substitution (single-back-quoted strings)

` command `

The standard output of the command (minus any trailing newlines) may be used as all or part of a word, or as multiple words.
Run the command, capture its standard output, and replace all the newlines by spaces. The result then replaces `command` in the context in which it appeared.

Example

% sh

        

        $ ls

        dir1   file1  file2

        

        $ echo ls

        ls

        

        $ echo `ls`

        dir1 file1 file2

        

        $ ls | cat

        dir1

        file1

        file2

        

        $ echo `ls | cat`

        dir1 file1 file2

Note that echo repeats the words in its command line, but with only one space between them.

Parameter Substitution

evaluate with $parameter or with ${parameter}

a parameter that has no value evaluates to nothing, which is different from an empty string.
Exercise. Why not $(parameter) ?

keyword parameters, shell variables (identified by a name)

name=value

positional parameters (identified by a digit)

set ...
$0 is set when the shell is invoked
$1, $2, ... are set to the arguments of the function, if required
Exercise. Try echo $0 as an interactive command, then inside a shell script, then inside a function in a shell script

certain parameter symbols expand to the positional parameters starting with $1, separated by spaces

* or @
An example follows.

additional forms, useful for dealing with default values

${parameter:-word}
${parameter:=word}
${parameter:?word}
${parameter:+word}

Parameters set by the shell

#	The number of positional parameters in decimal.
-	Flags supplied to the shell on invocation or by the set command.
?	The decimal value returned by the last synchronously executed command.
$	The process number of this shell.
!	The process number of the last background command invoked.

Parameters used by the shell (not the complete list)

HOME	default argument (home directory) for the cd command, set at login from the password file
PATH	search path for commands
CDPATH	search path for the `cd` command
PS1	primary prompt string, by default "$ "
PS2	secondary prompt string, by default "> "

Example - prompt strings

% sh
$ PS1="Feed me! "
Feed me! PS2="FEED ME!! "
Feed me! echo one two
one two
Feed me! echo "one
FEED ME!! two"
one
two
Feed me! exit
%

Example - the difference between $* and $@ and "$*" and "$@"

Unquoted, $* and $@ are the same, except perhaps for spaces.
If $* is within a pair of double quotes, the positional parameters are substituted and quoted, separated by quoted spaces.

"$1 $2 ..."
one word

If $@ is within a pair of double quotes, the positional parameters are substituted and quoted, separated by unquoted spaces.

"$1" "$2" ...
multiple words

% cat z.sh
echo ' ' .$*. ; printf "$# ."
for w in $*
do
printf "%s." $w
done
printf "\n\n"

echo ' ' .$@. ; printf "$# ."
for w in $@
do
printf "%s." $w
done
printf "\n\n"

echo ' ' ."$*". ; printf "$# ."
for w in "$*"
do
printf "%s." $w
done
printf "\n\n"

echo ' ' ."$@". ; printf "$# ."
for w in "$@"
do
printf "%s." $w
done
printf "\n\n"

% sh z.sh 1 2 3 ' ' 4 5 6
.1 2 3 4 5 6.
7 .1.2.3.4.5.6.

.1 2 3 4 5 6.
7 .1.2.3.4.5.6.

.1 2 3 4 5 6.
7 .1.2.3.4.5.6.

.1 2 3 4 5 6.
7 .1.2.3..4.5.6.

Filename Generation, pattern matching

*	Matches any string, including the null string (empty string).
?	Matches any single character.
[...]	Matches any one of the enclosed characters. A pair of characters separated by - matches any character lexically between the pair, inclusive. If the first character following the opening [ is a !, any character not enclosed is matched.

The list of candidates for matching comes from the file names in the current directory (including subdirectory names but not including hidden names). If there is no match, the word is left unchanged.

Example

$ ls
dir1 file1 file2

$ ls -a
. .. .file3 dir1 file1 file2

$ echo *
dir1 file1 file2

$ echo file*
file1 file2

$ echo file[123]
file1 file2

$ echo file[13]
file1

$ echo *[345]
*[345]

$ echo *?
dir1 file1 file2

$ echo ?
?

$ echo .*
. .. .file3

Expressions

The special command test is used to evaluate expressions for an exit status (0 acts as true, nonzero acts as false). For convenience, these commands are equivalent:

test ...
[ ... ]

The program version of test is at /usr/bin/test .

Operators for the test command (not the complete list)

file properties

-d file

True if file exists and is a directory.

-f file

True if file exists and is a regular file.

-x file

True if file exists and is executable. True will indicate only that the execute flag is on. If file is a directory, true indicates that file can be searched.

arithmetic comparisons

n1 -eq n2

True if the integers n1 and n2 are algebraically equal.

n1 -ne n2
n1 -gt n2
n1 -ge n2
n1 -lt n2
n1 -le n2

string comparisons

-n string

True if the length of string is non-zero.

-z string

True if the length of string is zero.

string1 = string2

True if the strings string1 and string2 are identical.

string1 != string2

True if the strings string1 and string2 are not identical.

logical connectors

condition1 -a condition2

True if both condition1 and condition2 are true.

condition1 -o condition2

True if either condition1 or condition2 is true.

! condition

True if condition is false.

( condition )

True if condition is true.

The parentheses ( ) can be used to alter the normal precedence and associativity.

Notice also that parentheses are meaningful to the shell and, therefore, must be backslash-quoted in this context.

Conditionals

If-then-else

if   list
then
   list
elif list
then
   list
else
     list
fi

The list following if is executed and, if it returns a zero exit status, the list following the first then is executed.
Otherwise, the list following elif is executed and, if its exit status is zero, the list following the next then is executed.
Failing that, the else list is executed.
If no else list or then list is executed, then the if command returns a zero exit status.

Example - which versions of the program should we try?

if [ `uname -s` = "SunOS" ]
then
Versions="pr1.sun pr1.gcc"
else
Versions=pr1.gcc
fi

for Program in $Versions
do
make $Program
# ... more later
done

Case selection with pattern matching

case word in
   pattern )
   list ;;
   pattern | pattern )
   list ;;
esac

Find the first pattern that matches the word, then execute the associated list.
The vertical bar represents alternatives - match one pattern or the other.
The patterns follow the filename-matching forms (with some minor differences).

Example - which versions of the program should we try? (continued)

for Program in $Versions
do
case $Program in
    *.sun)
      echo "Testing Sun's C compiler"
      ;;
    *.gcc)
      echo "Testing the GNU C compiler"
      ;;
    *)    # the default case
      echo "No match for testing."
      exit
      ;;
esac
done

Exercise. Why was it necessary to quote the strings for the echo command?

Loops

In these examples, list is some arbitrary command sequence.

Iterate over words in a sequence of words

for name in word ...
do
list
done

Iterate over the positional parameters 1, 2, 3 ...

for name
do
list
done

Examples

% cat x.sh

for foo
do
echo $foo
done

% sh x.sh a b c d
a
b
c
d

% cat x.sh
for foo in x y z
do
echo $foo
done

% sh x.sh a b c d
x
y
z

Example - which versions of the program should we try? (continued)

for Program in $Versions
do
make $Program
for TestCase in testdata/*
do
    echo "---- $Program < $TestCase ----"
    ls -l $TestCase
    $Program < $TestCase
    echo
done
done

While loop

while list
do
list
done

Execute the while list and, if the exit status of the last command in the list is zero, execute the do list, otherwise the loop terminates.
If no commands in the do list are executed, then the while command returns a zero exit status.

Example - read standard input until end of file

while read input
do
echo "input: " $input
done

Until loop

until list
do
list
done

The same, but reverses the sense of the test for loop termination.

Example - read standard input until you find magic, with a function definition

% cat y.sh
test_for_magic () {
for word in $*
do
    echo "trying: " $word
    case $word in
      *magic*)
         return 0       # success
       ;;
    esac
done
return 1              # not success
}

until
read input
test_for_magic $input
do
echo "rejected: " $input
done

echo "accepted: " $input

% sh y.sh
one two three
trying: one
trying: two
trying: three
rejected: one two three
is this the magic word?
trying: is
trying: this
trying: the
trying: magic
accepted: is this the magic word?

Special commands (not the complete list; more info in Part 2)

break
continue
return
shift

I/O Redirection

<word	take standard input from file word
>word	send standard output to file word, but first create or truncate the file
>>word	same, but create or append to the file
<<word	take standard input up to word or end-of-file; see example
`digit>`	affect file descriptor `digit`
`digit<`	affect file descriptor `digit`
<&digit	see example
>&digit	see example

The default standard input for a background command (one ending with &) is /dev/null, the empty file.

Example

% cat w.sh
while read input
do
echo "found: " $input
done

% sh w.sh <<END
? q w e r t
? y
? u
? i
? o
? p
? END
found: q w e r t
found: y
found: u
found: i
found: o
found: p

Example, using sh interactively

$ cat w.c
#include <stdio.h>
int main(int argc, char *argv[])
{
fprintf(stdout, "abc %d\n", argc);
fprintf(stderr, "xyz %d\n", argc);
return 0;
}
$ cc w.c
$ a.out 1
abc 2
xyz 2
$ a.out 1> foo 2> bar
$ cat foo
abc 1
$ cat bar
xyz 1

Example - 2>&1 - merge standard output (file descriptor 1) and standard error (file descriptor 2)

% cat foo.c
void main(void) { printf("xyz\n"); return; }
// missing #include <stdio.h>

% cat bar.c
void main(void) { print("xyz\n"); return; }
// missing f? undefined print?

% cat cr.sh
{ cc -v $1.c
if [ -f a.out -a -x a.out ]
then
    a.out
fi
} 1>compile_and_run.out 2>&1

% sh cr.sh foo

% cat comp*
"foo.c", line 1: warning: implicit function declaration: printf
xyz

% sh cr.sh bar

% cat comp*
"bar.c", line 1: warning: implicit function declaration: print
Undefined                       first referenced
symbol                             in file
print                               bar.o
ld: fatal: Symbol referencing errors. No output written to a.out

Last revised, 25 Feb. 2013