Bash Scripting – the for command
Iterating through a series of commands is a common programming practice. Often, you need to repeat a set of commands until a specific condition has been met, such as processing all the files in a directory, all the users on a system, or all the lines in a text file.
The bash shell provides the for command to allow you to create a loop that iterates through a series of values. Each iteration performs a defined set of commands using one of the values in the series. Here’s the basic format of the bash shell for command:
for var in list do commands done
In each iteration, the variable var contains the current value in the list. The first iteration uses the first item in the list, the second iteration the second item, and so on until all the items in the list have been used.
$ cat test1 #!/bin/bash # basic for command for test in Alabama Alaska Arizona Arkansas California Colorado do echo The next state is $test done $ ./test1 The next state is Alabama The next state is Alaska The next state is Arizona The next state is Arkansas The next state is California The next state is Colorado $
Each time the for command iterates through the list of values provided, it assigns the $test variable the next value in the list. The $testvariable can be used just like any other script variable within the for command statements. After the last iteration, the $test variable remains valid throughout the remainder of the shell script. It retains the last iteration value (unless you change its value):
$ cat test1b #!/bin/bash # testing the for variable after the looping for test in Alabama Alaska Arizona Arkansas California Colorado do echo “The next state is $test” done echo “The last state we visited was $test” test=Connecticut echo “Wait, now we're visiting $test” $ ./test1b The next state is Alabama The next state is Alaska The next state is Arizona The next state is Arkansas The next state is California The next state is Colorado The last state we visited was Colorado Wait, now we're visiting Connecticut $
Things aren’t always as easy as they seem with the for loop. There are times when you run into data that causes problems. Here’s a classic example of what can cause problems for shell script programmers:
$ cat badtest1 #!/bin/bash # another example of how not to use the for command for test in I don't know if this'll work do echo “word:$test” done $ ./badtest1 word:I word:dont know if thisll word:work $
- Use the escape character (the backslash) to escape the single quotation mark.
- Use double quotation marks to define the values that use single quotation marks.
$ cat test2 #!/bin/bash # another example of how not to use the for command for test in I don\'t know if “this'll” work do echo “word:$test” done $ ./test2 word:I word:don't word:know word:if word:this'll word:work $
In the first problem value, you added the backslash character to escape the single quotation mark in the don’t value. In the second problem value, you enclosed the this’ll value in double quotation marks. Both methods worked fine to distinguish the value.
Another problem you may run into is multi-word values. Remember that the for loop assumes that each value is separated with a space. If you have data values that contain spaces, you run into yet another problem:
$ cat badtest2 #!/bin/bash # another example of how not to use the for command for test in Nevada New Hampshire New Mexico New York North Carolina do echo “Now going to $test” done $ ./badtest1 Now going to Nevada Now going to New Now going to Hampshire Now going to New Now going to Mexico Now going to New Now going to York Now going to North Now going to Carolina $
Oops, that’s not exactly what we wanted. The for command separates each value in the list with a space. If there are spaces in the individual data values, you must accommodate them using double quotation marks:
$ cat test3 #!/bin/bash # an example of how to properly define values for test in Nevada “New Hampshire” “New Mexico” “New York” do echo “Now going to $test” done $ ./test3 Now going to Nevada Now going to New Hampshire Now going to New Mexico Now going to New York $
Now the for command can properly distinguish between the different values. Also, notice that when you use double quotation marks around a value, the shell doesn’t include the quotation marks as part of the value.
$ cat test4 #!/bin/bash # using a variable to hold the list list=“Alabama Alaska Arizona Arkansas Colorado” list=$list“ Connecticut” for state in $list do echo “Have you ever visited $state?” done $ ./test4 Have you ever visited Alabama? Have you ever visited Alaska? Have you ever visited Arizona? Have you ever visited Arkansas? Have you ever visited Colorado? Have you ever visited Connecticut? $
The $list variable contains the standard text list of values to use for the iterations. Notice that the code also uses another assignment statement to add (or concatenate) an item to the existing list contained in the $list variable. This is a common method for adding text to the end of an existing text string stored in a variable.
Another way to generate values for use in the list is to use the output of a command. You use command substitution to execute any command that produces output and then use the output of the command in the for command:
$ cat test5 #!/bin/bash # reading values from a file file=“states” for state in $(cat $file) do echo “Visit beautiful $state” done $ cat states Alabama Alaska Arizona Arkansas Colorado Connecticut Delaware Florida Georgia $ ./test5 Visit beautiful Alabama Visit beautiful Alaska Visit beautiful Arizona Visit beautiful Arkansas Visit beautiful Colorado Visit beautiful Connecticut Visit beautiful Delaware Visit beautiful Florida Visit beautiful Georgia $
This example uses the cat command in the command substitution to display the contents of the file states. Notice that the states file includes each state on a separate line, not separated by spaces. The for command still iterates through the output of the cat command one line at a time, assuming that each state is on a separate line. However, this doesn’t solve the problem of having spaces in data. If you list a state with a space in it, the for command still takes each word as a separate value. There’s a reason for this, which we look at in the next section.
The test5 code example assigned the filename to the variable using just the filename without a path. This requires that the file be in the same directory as the script. If this isn’t the case, you need to use a full pathname (either absolute or relative) to reference the file location.
The cause of this problem is the special environment variable IFS, called the internal field separator. The IFS environment variable defines a list of characters the bash shell uses as field separators. By default, the bash shell considers the following characters as field separators:
- A space
- A tab
- A newline
If the bash shell sees any of these characters in the data, it assumes that you’re starting a new data field in the list. When working with data that can contain spaces (such as filenames), this can be annoying, as you saw in the previous script example.
To solve this problem, you can temporarily change the IFS environment variable values in your shell script to restrict the characters the bash shell recognizes as field separators. For example, if you want to change the IFS value to recognize only the newline character, you need to do this:
$ cat test5b #!/bin/bash # reading values from a file file=“states” IFS=$‘\n’ for state in $(cat $file) do echo “Visit beautiful $state” done $ ./test5b Visit beautiful Alabama Visit beautiful Alaska Visit beautiful Arizona Visit beautiful Arkansas Visit beautiful Colorado Visit beautiful Connecticut Visit beautiful Delaware Visit beautiful Florida Visit beautiful Georgia Visit beautiful New York Visit beautiful New Hampshire Visit beautiful North Carolina $
When working on long scripts, it’s possible to change the IFS value in one place, and then forget about it and assume the default value elsewhere in the script. A safe practice to get into is to save the original IFS value before changing it and then restore it when you’re finished.
IFS.OLD=$IFS IFS=$‘\n’ <use the new IFS value in code> IFS=$IFS.OLD
Other excellent applications of the IFS environment variable are possible. Suppose you want to iterate through values in a file that are separated by a colon (such as in the /etc/ passwd file). You just need to set the IFS value to a colon:
Finally, you can use the for command to automatically iterate through a directory of files. To do this, you must use a wildcard character in the file or pathname. This forces the shell to use file globbing. File globbing is the process of producing filenames or pathnames that match a specified wildcard character.
$ cat test6 #!/bin/bash # iterate through all the files in a directory for file in /home/rich/test/* do if [ -d “$file” ] then echo “$file is a directory” elif [ -f “$file” ] then echo “$file is a file” fi done $ ./test6 /home/rich/test/dir1 is a directory /home/rich/test/myprog.c is a file /home/rich/test/myprog is a file /home/rich/test/myscript is a file /home/rich/test/newdir is a directory /home/rich/test/newfile is a file /home/rich/test/newfile2 is a file /home/rich/test/testdir is a directory /home/rich/test/testing is a file /home/rich/test/testprog is a file /home/rich/test/testprog.c is a file $
The for command iterates through the results of the /home/rich/test/* listing. The code tests each entry using the test command (using the square bracket method) to see if it’s a directory, using the -d parameter, or a file, using the -f parameter (See Chapter 12).
if [ -d “$file” ]
In Linux, it’s perfectly legal to have directory and filenames that contain spaces. To accommodate that, you should enclose the $filevariable in double quotation marks. If you don’t, you’ll get an error if you run into a directory or filename that contains spaces:
./test6: line 6: [: too many arguments ./test6: line 9: [: too many arguments
$ cat test7 #!/bin/bash # iterating through multiple directories for file in /home/rich/.b* /home/rich/badtest do if [ -d “$file” ] then echo “$file is a directory” elif [ -f “$file” ] then echo “$file is a file” else echo “$file doesn't exist” fi done $ ./test7 /home/rich/.backup.timestamp is a file /home/rich/.bash_history is a file /home/rich/.bash_logout is a file /home/rich/.bash_profile is a file /home/rich/.bashrc is a file /home/rich/badtest doesn't exist $
The for statement first uses file globbing to iterate through the list of files that result from the wildcard character; then it iterates through the next file in the list. You can combine any number of wildcard entries in the list to iterate through.
Notice that you can enter anything in the list data. Even if the file or directory doesn’t exist, the for statement attempts to process whatever you place in the list. This can be a problem when working with files and directories. You have no way of knowing if you’re trying to iterate through a nonexistent directory: It’s always a good idea to test each file or directory before trying to process it.