Regexes, Tokens, and Rules

• Regular expressions are now known by the name "regex".

• Regexes are a convenient way to create pattern-matching subroutines using a specialized syntax (see S05).

• A "token" is simply a regex that does not backtrack by default.

• Examples:

token ident { [ <alpha> | _ ] \w* }

regex ident { [ <alpha>: | _ ]: \w*: }

$ident = qr{ (?> (?: [[:alpha:]] | _ ) \w* ) }x    ## Perl 5