Part 2 – Mastering Grep – Regular Expressions

Using Regular Expressions with Grep

A “regular expression” is a pattern that describes a set of strings. Regular expressions are constructed analogously to arithmetic expressions, by using various operators to combine smaller expressions.

`grep’ understands three different versions of regular expression syntax: “basic,” (BRE) “extended” (ERE) and “perl”. In GNU `grep’, there is no difference in available functionality between the basic and extended syntaxes. In other implementations, basic regular expressions are less powerful.

The following description applies to extended regular expressions; differences for basic regular expressions are summarized afterwards. Perl regular expressions give additional functionality, and are documented in the pcresyntax(3) and pcrepattern(3) manual pages, but may not be available on every system.

1 Fundamental Structure

The fundamental building blocks are the regular expressions that match a single character. Most characters, including all letters and digits, are regular expressions that match themselves. Any meta-character with special meaning may be quoted by preceding it with a backslash.

A regular expression may be followed by one of several repetition operators:

• `.’ The period `.’ matches any single character.
• `?’ The preceding item is optional and will be matched at most once.
• `*’ The preceding item will be matched zero or more times.
• `+’ The preceding item will be matched one or more times.
• `{N}’ The preceding item is matched exactly N times.
• `{N,}’ The preceding item is matched N or more times.
• `{,M}’ The preceding item is matched at most M times. This is a GNU extension.
• `{N,M}’ The preceding item is matched at least N times, but not more than M times.

The empty regular expression matches the empty string. Two regular expressions may be concatenated; the resulting regular expression matches any string formed by concatenating two substrings that respectively match the concatenated expressions.

Two regular expressions may be joined by the infix operator `|’; the resulting regular expression matches any string matching either alternate expression.

Repetition takes precedence over concatenation, which in turn takes precedence over alternation. A whole expression may be enclosed in parentheses to override these precedence rules and form a subexpression.

2 Character Classes and Bracket Expressions

A “bracket expression” is a list of characters enclosed by `[‘ and `]’. It matches any single character in that list; if the first character of the list is the caret `^’, then it matches any character *not* in the list.

For example, the regular expression `[0123456789]’ matches any single digit.

Within a bracket expression, a “range expression” consists of two characters separated by a hyphen. It matches any single character that sorts between the two characters, inclusive, using the locale’s collating sequence and character set.

For example, in the default C locale, `[a-d]’ is equivalent to `[abcd]’. Many locales sort characters in dictionary order, and in these locales `[a-d]’ is typically not equivalent to `[abcd]’; it might be equivalent to `[aBbCcDd]’, for example.

To obtain the traditional interpretation of bracket expressions, you can use the `C’ locale by setting the `LC_ALL’ environment variable to the value `C’.

Finally, certain named classes of characters are predefined within bracket expressions, as follows. Their interpretation depends on the `LC_CTYPE’ locale; for example, `[[:alnum:]]’ means the character class of numbers and letters in the current locale.

`[:alnum:]’
Alphanumeric characters: `[:alpha:]’ and `[:digit:]’; in the `C’ locale and ASCII character encoding, this is the same as `[0-9A-Za-z]a’.

`[:alpha:]’
Alphabetic characters: `[:lower:]’ and `[:upper:]’; in the `C’ locale and ASCII character encoding, this is the same as `[A-Za-z]’.

`[:blank:]’
Blank characters: space and tab.

`[:cntrl:]’
Control characters. In ASCII, these characters have octal codes 000 through 037, and 177 (`DEL’). In other character sets, these are the equivalent characters, if any.

`[:digit:]’
Digits: `0 1 2 3 4 5 6 7 8 9′.

`[:graph:]’
Graphical characters: `[:alnum:]’ and `[:punct:]’.

`[:lower:]’
Lower-case letters; in the `C’ locale and ASCII character encoding, this is `a b c d e f g h i j k l m n o p q r s t u v w x y z’.

`[:print:]’
Printable characters: `[:alnum:]’, `[:punct:]’, and space.

`[:punct:]’
Punctuation characters; in the `C’ locale and ASCII character encoding, this is `! ” # $ % & ‘ ( ) * + , – . / : ; < = > ? @ [ \ ] ^ _ ` { | } ~’.

`[:space:]’
Space characters: in the `C’ locale, this is tab, newline, vertical tab, form feed, carriage return, and space. *Note Usage::, for more discussion of matching newlines.

`[:upper:]’
Upper-case letters: in the `C’ locale and ASCII character encoding, this is `A B C D E F G H I J K L M N O P Q R S T U V W X Y Z’.

`[:xdigit:]’
Hexadecimal digits: `0 1 2 3 4 5 6 7 8 9 A B C D E F a b c d e f’.

Note that the brackets in these class names are part of the symbolic names, and must be included in addition to the brackets delimiting the bracket expression.

If you mistakenly omit the outer brackets, and search for say, `[:upper:]’, GNU `grep’ prints a diagnostic and exits with status 2, on the assumption that you did not intend to search for the nominally equivalent regular expression: `[:epru]’. Set the `POSIXLY_CORRECT’ environment variable to disable this feature.

Most meta-characters lose their special meaning inside bracket expressions.

`]’          ends the bracket expression if it’s not the first list item. So, if you want to make the `]’ character a list item, you must put it first.

`[.’          represents the open collating symbol.

`.]’          represents the close collating symbol.

`[=’         represents the open equivalence class.

`=]’          represents the close equivalence class.

`[:’        represents the open character class symbol, and should be followed by a valid character class name.

`:]’          represents the close character class symbol.

`-‘           represents the range if it’s not first or last in a list or the ending point of a range.

`^’          represents the characters not in the list. If you want to make the `^’ character a list item, place it anywhere but first.

3 The Backslash Character and Special Expressions

The `\’ character, when followed by certain ordinary characters, takes a special meaning:

`\b’ Match the empty string at the edge of a word.

`\B’ Match the empty string provided it’s not at the edge of a word.

`\<‘ Match the empty string at the beginning of word. `\>’ Match the empty string at the end of word.

`\w’ Match word constituent, it is a synonym for `[_[:alnum:]]’.

`\W’ Match non-word constituent, it is a synonym for `[^_[:alnum:]]’.

`\s’ Match whitespace, it is a synonym for `[[:space:]]’.

`\S’ Match non-whitespace, it is a synonym for `[^[:space:]]’.

For example, `\brat\b’ matches the separate word `rat’, `\Brat\B matches `crate’ but not `furry rat’.

4 Anchoring

The caret `^’ and the dollar sign `$’ are meta-characters that respectively match the empty string at the beginning and end of a line. They are termed “anchors”, since they force the match to be “anchored” to beginning or end of a line, respectively.

5 Back-references and Subexpressions

The back-reference `\N’, where N is a single digit, matches the substring previously matched by the Nth parenthesized subexpression of the regular expression. For example, `(a)\1′ matches `aa’. When used with alternation, if the group does not participate in the match then the back-reference makes the whole match fail. For example, `a(.)|b\1′ will not match `ba’. When multiple regular expressions are given with `-e’ or from a file (`-f FILE’), back-references are local to each expression.

6 Basic vs Extended Regular Expressions

In basic regular expressions the meta-characters

`?’, `+’, `{‘, `|’, `(‘, and `)’ lose their special meaning;

instead use the backslashed versions `\?’, `\+’, `\{‘, `\|’, `\(‘, and `\)’.

Traditional `egrep’ did not support the `{‘ meta-character, and some `egrep’ implementations support `\{‘ instead, so portable scripts should avoid `{‘ in `grep -E’ patterns and should use `[{]’ to match a literal `{‘.

GNU `grep -E’ attempts to support traditional usage by assuming that `{‘ is not special if it would be the start of an invalid interval specification.

For example, the command `grep -E ‘{1” searches for the two-character string `{1’ instead of reporting a syntax error in the regular expression.