This file documents awk, a program that you can use to select particular records in a file and perform operations upon them.
Copyright © 1989, 1991, 1992, 1993, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003 Free Software Foundation, Inc.
This is Edition 3 of GAWK: Effective AWK Programming: A User's Guide for GNU Awk, for the 3.1.4 (or later) version of the GNU implementation of AWK.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with the Invariant Sections being “GNU General Public License”, the Front-Cover texts being (a) (see below), and with the Back-Cover Texts being (b) (see below). A copy of the license is included in the section entitled “GNU Free Documentation License”.
getline
getline with No Arguments
getline into a Variable
getline from a File
getline into a Variable from a File
getline from a Pipe
getline into a Variable from a Pipe
getline from a Coprocess
getline into a Variable from a Coprocess
getline
getline Variants
delete Statement
Arnold Robbins and I are good friends. We were introduced 11 years ago
by circumstances—and our favorite programming language, AWK.
The circumstances started a couple of years
earlier. I was working at a new job and noticed an unplugged
Unix computer sitting in the corner. No one knew how to use it,
and neither did I. However,
a couple of days later it was running, and
I was root and the one-and-only user.
That day, I began the transition from statistician to Unix programmer.
On one of many trips to the library or bookstore in search of books on Unix, I found the gray AWK book, a.k.a. Aho, Kernighan and Weinberger, The AWK Programming Language, Addison-Wesley, 1988. AWK's simple programming paradigm—find a pattern in the input and then perform an action—often reduced complex or tedious data manipulations to few lines of code. I was excited to try my hand at programming in AWK.
Alas, the awk on my computer was a limited version of the
language described in the AWK book. I discovered that my computer
had “old awk” and the AWK book described “new awk.”
I learned that this was typical; the old version refused to step
aside or relinquish its name. If a system had a new awk, it was
invariably called nawk, and few systems had it.
The best way to get a new awk was to ftp the source code for
gawk from prep.ai.mit.edu. gawk was a version of
new awk written by David Trueman and Arnold, and available under
the GNU General Public License.
(Incidentally, it's no longer difficult to find a new awk. gawk ships with Linux, and you can download binaries or source code for almost any system; my wife uses gawk on her VMS box.)
My Unix system started out unplugged from the wall; it certainly was not
plugged into a network. So, oblivious to the existence of gawk
and the Unix community in general, and desiring a new awk, I wrote
my own, called mawk.
Before I was finished I knew about gawk,
but it was too late to stop, so I eventually posted
to a comp.sources newsgroup.
A few days after my posting, I got a friendly email from Arnold introducing himself. He suggested we share design and algorithms and attached a draft of the POSIX standard so that I could update mawk to support language extensions added after publication of the AWK book.
Frankly, if our roles had been reversed, I would not have been so open and we probably would have never met. I'm glad we did meet. He is an AWK expert's AWK expert and a genuinely nice person. Arnold contributes significant amounts of his expertise and time to the Free Software Foundation.
This book is the gawk reference manual, but at its core it is a book about AWK programming that will appeal to a wide audience. It is a definitive reference to the AWK language as defined by the 1987 Bell Labs release and codified in the 1992 POSIX Utilities standard.
On the other hand, the novice AWK programmer can study a wealth of practical programs that emphasize the power of AWK's basic idioms: data driven control-flow, pattern matching with regular expressions, and associative arrays. Those looking for something new can try out gawk's interface to network protocols via special /inet files.
The programs in this book make clear that an AWK program is typically much smaller and faster to develop than a counterpart written in C. Consequently, there is often a payoff to prototype an algorithm or design in AWK to get it running quickly and expose problems early. Often, the interpreted performance is adequate and the AWK prototype becomes the product.
The new pgawk (profiling gawk), produces program execution counts. I recently experimented with an algorithm that for n lines of input, exhibited ~ C n^2 performance, while theory predicted ~ C n log n behavior. A few minutes poring over the awkprof.out profile pinpointed the problem to a single line of code. pgawk is a welcome addition to my programmer's toolbox.
Arnold has distilled over a decade of experience writing and using AWK programs, and developing gawk, into this book. If you use AWK or want to learn how, then read this book.
Michael Brennan
Author of mawk
Several kinds of tasks occur repeatedly when working with text files. You might want to extract certain lines and discard the rest. Or you may need to make changes wherever certain patterns appear, but leave the rest of the file alone. Writing single-use programs for these tasks in languages such as C, C++, or Pascal is time-consuming and inconvenient. Such jobs are often easier with awk. The awk utility interprets a special-purpose programming language that makes it easy to handle simple data-reformatting jobs.
The GNU implementation of awk is called gawk; it is fully compatible with the System V Release 4 version of awk. gawk is also compatible with the POSIX specification of the awk language. This means that all properly written awk programs should work with gawk. Thus, we usually don't distinguish between gawk and other awk implementations.
In addition, gawk provides facilities that make it easy to:
This Web page teaches you about the awk language and how you can use it effectively. You should already be familiar with basic system commands, such as cat and ls,1 as well as basic shell facilities, such as input/output (I/O) redirection and pipes.
Implementations of the awk language are available for many different computing environments. This Web page, while describing the awk language in general, also describes the particular implementation of awk called gawk (which stands for “GNU awk”). gawk runs on a broad range of Unix systems, ranging from 80386 PC-based computers up through large-scale systems, such as Crays. gawk has also been ported to Mac OS X, MS-DOS, Microsoft Windows (all versions) and OS/2 PCs, Atari and Amiga microcomputers, BeOS, Tandem D20, and VMS.
1 part egrep | 1 part snobol
| |
2 parts ed | 3 parts C
|
Blend all parts well usinglexandyacc. Document minimally and release.After eight years, add another part
egrepand two more parts C. Document very well and release.
The name awk comes from the initials of its designers: Alfred V. Aho, Peter J. Weinberger and Brian W. Kernighan. The original version of awk was written in 1977 at AT&T Bell Laboratories. In 1985, a new version made the programming language more powerful, introducing user-defined functions, multiple input streams, and computed regular expressions. This new version became widely available with Unix System V Release 3.1 (SVR3.1). The version in SVR4 added some new features and cleaned up the behavior in some of the “dark corners” of the language. The specification for awk in the POSIX Command Language and Utilities standard further clarified the language. Both the gawk designers and the original Bell Laboratories awk designers provided feedback for the POSIX specification.
Paul Rubin wrote the GNU implementation, gawk, in 1986. Jay Fenlason completed it, with advice from Richard Stallman. John Woods contributed parts of the code as well. In 1988 and 1989, David Trueman, with help from me, thoroughly reworked gawk for compatibility with the newer awk. Circa 1995, I became the primary maintainer. Current development focuses on bug fixes, performance improvements, standards compliance, and occasionally, new features.
In May of 1997, Jürgen Kahrs felt the need for network access from awk, and with a little help from me, set about adding features to do this for gawk. At that time, he also wrote the bulk of TCP/IP Internetworking with gawk (a separate document, available as part of the gawk distribution). His code finally became part of the main gawk distribution with gawk version 3.1.
See Contributors, for a complete list of those who made important contributions to gawk.
The awk language has evolved over the years. Full details are provided in Language History. The language described in this Web page is often referred to as “new awk” (nawk).
Because of this, many systems have multiple versions of awk. Some systems have an awk utility that implements the original version of the awk language and a nawk utility for the new version. Others have an oawk version for the “old awk” language and plain awk for the new one. Still others only have one version, which is usually the new one.2
All in all, this makes it difficult for you to know which version of awk you should run when writing your programs. The best advice I can give here is to check your local documentation. Look for awk, oawk, and nawk, as well as for gawk. It is likely that you already have some version of new awk on your system, which is what you should use when running your programs. (Of course, if you're reading this Web page, chances are good that you have gawk!)
Throughout this Web page, whenever we refer to a language feature that should be available in any complete implementation of POSIX awk, we simply use the term awk. When referring to a feature that is specific to the GNU implementation, we use the term gawk.
The term awk refers to a particular program as well as to the language you use to tell this program what to do. When we need to be careful, we call the language “the awk language,” and the program “the awk utility.” This Web page explains both the awk language and how to run the awk utility. The term awk program refers to a program written by you in the awk programming language.
Primarily, this Web page explains the features of awk, as defined in the POSIX standard. It does so in the context of the gawk implementation. While doing so, it also attempts to describe important differences between gawk and other awk implementations.3 Finally, any gawk features that are not in the POSIX standard for awk are noted.
This Web page has the difficult task of being both a tutorial and a reference. If you are a novice, feel free to skip over details that seem too complex. You should also ignore the many cross-references; they are for the expert user and for the online Info version of the document.
There are subsections labelled as Advanced Notes scattered throughout the Web page. They add a more complete explanation of points that are relevant, but not likely to be of interest on first reading. All appear in the index, under the heading “advanced features.”
Most of the time, the examples use complete awk programs. In some of the more advanced sections, only the part of the awk program that illustrates the concept currently being described is shown.
While this Web page is aimed principally at people who have not been exposed to awk, there is a lot of information here that even the awk expert should find useful. In particular, the description of POSIX awk and the example programs in Library Functions, and in Sample Programs, should be of interest.
Getting Started, provides the essentials you need to know to begin using awk.
Regexp, introduces regular expressions in general, and in particular the flavors supported by POSIX awk and gawk.
Reading Files,
describes how awk reads your data.
It introduces the concepts of records and fields, as well
as the getline command.
I/O redirection is first described here.
Printing,
describes how awk programs can produce output with
print and printf.
Expressions, describes expressions, which are the basic building blocks for getting most things done in a program.
Patterns and Actions, describes how to write patterns for matching records, actions for doing something when a record is matched, and the built-in variables awk and gawk use.
Arrays, covers awk's one-and-only data structure: associative arrays. Deleting array elements and whole arrays is also described, as well as sorting arrays in gawk.
Functions, describes the built-in functions awk and gawk provide, as well as how to define your own functions.
Internationalization, describes special features in gawk for translating program messages into different languages at runtime.
Advanced Features, describes a number of gawk-specific advanced features. Of particular note are the abilities to have two-way communications with another process, perform TCP/IP networking, and profile your awk programs.
Invoking Gawk, describes how to run gawk, the meaning of its command-line options, and how it finds awk program source files.
Library Functions, and Sample Programs, provide many sample awk programs. Reading them allows you to see awk solving real problems.
Language History, describes how the awk language has evolved since first release to present. It also describes how gawk has acquired features over time.
Installation, describes how to get gawk, how to compile it under Unix, and how to compile and use it on different non-Unix systems. It also describes how to report bugs in gawk and where to get three other freely available implementations of awk.
Notes, describes how to disable gawk's extensions, as well as how to contribute new code to gawk, how to write extension libraries, and some possible future directions for gawk development.
Basic Concepts, provides some very cursory background material for those who are completely unfamiliar with computer programming. Also centralized there is a discussion of some of the issues surrounding floating-point numbers.
The Glossary, defines most, if not all, the significant terms used throughout the book. If you find terms that you aren't familiar with, try looking them up here.
Copying, and GNU Free Documentation License, present the licenses that cover the gawk source code and this Web page, respectively.
This Web page is written using Texinfo, the GNU documentation formatting language. A single Texinfo source file is used to produce both the printed and online versions of the documentation. Because of this, the typographical conventions are slightly different than in other books you may have read.
Examples you would type at the command-line are preceded by the common shell primary and secondary prompts, `$' and `>'. Output from the command is preceded by the glyph “-|”. This typically represents the command's standard output. Error messages, and other output on the command's standard error, are preceded by the glyph “error-->”. For example:
$ echo hi on stdout
-| hi on stdout
$ echo hello on stderr 1>&2
error--> hello on stderr
In the text, command names appear in this font, while code segments
appear in the same font and quoted, `like this'. Some things are
emphasized like this, and if a point needs to be made
strongly, it is done like this. The first occurrence of
a new term is usually its definition and appears in the same
font as the previous occurrence of “definition” in this sentence.
file names are indicated like this: /path/to/ourfile.
Characters that you type at the keyboard look like this. In particular, there are special characters called “control characters.” These are characters that you type by holding down both the CONTROL key and another key, at the same time. For example, a Ctrl-d is typed by first pressing and holding the CONTROL key, next pressing the d key and finally releasing both keys.
Dark corners are basically fractal — no matter how much you illuminate, there's always a smaller but darker one.
Brian Kernighan
Until the POSIX standard (and The Gawk Manual), many features of awk were either poorly documented or not documented at all. Descriptions of such features (often called “dark corners”) are noted in this Web page with “(d.c.)”. They also appear in the index under the heading “dark corner.”
As noted by the opening quote, though, any coverage of dark corners is, by definition, something that is incomplete.
The Free Software Foundation (FSF) is a nonprofit organization dedicated to the production and distribution of freely distributable software. It was founded by Richard M. Stallman, the author of the original Emacs editor. GNU Emacs is the most widely used version of Emacs today.
The GNU4 Project is an ongoing effort on the part of the Free Software Foundation to create a complete, freely distributable, POSIX-compliant computing environment. The FSF uses the “GNU General Public License” (GPL) to ensure that their software's source code is always available to the end user. A copy of the GPL is included in this Web page for your reference (see Copying). The GPL applies to the C language source code for gawk. To find out more about the FSF and the GNU Project online, see the GNU Project's home page. This Web page may also be read from their web site.
A shell, an editor (Emacs), highly portable optimizing C, C++, and Objective-C compilers, a symbolic debugger and dozens of large and small utilities (such as gawk), have all been completed and are freely available. The GNU operating system kernel (the HURD), has been released but is still in an early stage of development.
Until the GNU operating system is more fully developed, you should consider using GNU/Linux, a freely distributable, Unix-like operating system for Intel 80386, DEC Alpha, Sun SPARC, IBM S/390, and other systems.5 There are many books on GNU/Linux. One that is freely available is Linux Installation and Getting Started, by Matt Welsh. Many GNU/Linux distributions are often available in computer stores or bundled on CD-ROMs with books about Linux. (There are three other freely available, Unix-like operating systems for 80386 and other systems: NetBSD, FreeBSD, and OpenBSD. All are based on the 4.4-Lite Berkeley Software Distribution, and they use recent versions of gawk for their versions of awk.)
The Web page you are reading is actually free—at least, the information in it is free to anyone. The machine-readable source code for the Web page comes with gawk; anyone may take this Web page to a copying machine and make as many copies as they like. (Take a moment to check the Free Documentation License in GNU Free Documentation License.)
Although you could just print it out yourself, bound books are much easier to read and use. Furthermore, the proceeds from sales of this book go back to the FSF to help fund development of more free software.
The Web page itself has gone through a number of previous editions. Paul Rubin wrote the very first draft of The GAWK Manual; it was around 40 pages in size. Diane Close and Richard Stallman improved it, yielding a version that was around 90 pages long and barely described the original, “old” version of awk.
I started working with that version in the fall of 1988. As work on it progressed, the FSF published several preliminary versions (numbered 0.x). In 1996, Edition 1.0 was released with gawk 3.0.0. The FSF published the first two editions under the title The GNU Awk User's Guide.
This edition maintains the basic structure of Edition 1.0, but with significant additional material, reflecting the host of new features in gawk version 3.1. Of particular note is Array Sorting, as well as Bitwise Functions, Internationalization, and also Advanced Features, and Dynamic Extensions.
GAWK: Effective AWK Programming will undoubtedly continue to evolve. An electronic version comes with the gawk distribution from the FSF. If you find an error in this Web page, please report it! See Bugs, for information on submitting problem reports electronically, or write to me in care of the publisher.
As the maintainer of GNU awk, I am starting a collection of publicly available awk programs. For more information, see ftp://ftp.freefriends.org/arnold/Awkstuff. If you have written an interesting awk program, or have written a gawk extension that you would like to share with the rest of the world, please contact me (arnold@skeeve.com). Making things available on the Internet helps keep the gawk distribution down to manageable size.
The initial draft of The GAWK Manual had the following acknowledgments:
Many people need to be thanked for their assistance in producing this manual. Jay Fenlason contributed many ideas and sample programs. Richard Mlynarik and Robert Chassell gave helpful comments on drafts of this manual. The paper A Supplemental Document for awk by John W. Pierce of the Chemistry Department at UC San Diego, pinpointed several issues relevant both to awk implementation and to this manual, that would otherwise have escaped us.
I would like to acknowledge Richard M. Stallman, for his vision of a better world and for his courage in founding the FSF and starting the GNU Project.
The following people (in alphabetical order) provided helpful comments on various versions of this book, up to and including this edition. Rick Adams, Nelson H.F. Beebe, Karl Berry, Dr. Michael Brennan, Rich Burridge, Claire Cloutier, Diane Close, Scott Deifik, Christopher (“Topher”) Eliot, Jeffrey Friedl, Dr. Darrel Hankerson, Michal Jaegermann, Dr. Richard J. LeBlanc, Michael Lijewski, Pat Rankin, Miriam Robbins, Mary Sheehan, and Chuck Toporek.
Robert J. Chassell provided much valuable advice on the use of Texinfo. He also deserves special thanks for convincing me not to title this Web page How To Gawk Politely. Karl Berry helped significantly with the TeX part of Texinfo.
I would like to thank Marshall and Elaine Hartholz of Seattle and Dr. Bert and Rita Schreiber of Detroit for large amounts of quiet vacation time in their homes, which allowed me to make significant progress on this Web page and on gawk itself.
Phil Hughes of SSC contributed in a very important way by loaning me his laptop GNU/Linux system, not once, but twice, which allowed me to do a lot of work while away from home.
David Trueman deserves special credit; he has done a yeoman job of evolving gawk so that it performs well and without bugs. Although he is no longer involved with gawk, working with him on this project was a significant pleasure.
The intrepid members of the GNITS mailing list, and most notably Ulrich Drepper, provided invaluable help and feedback for the design of the internationalization features.
Nelson Beebe, Martin Brown, Andreas Buening, Scott Deifik, Darrel Hankerson, Isamu Hasegawa, Michal Jaegermann, Jürgen Kahrs, Pat Rankin, Kai Uwe Rommel, and Eli Zaretskii (in alphabetical order) make up the gawk “crack portability team.” Without their hard work and help, gawk would not be nearly the fine program it is today. It has been and continues to be a pleasure working with this team of fine people.
David and I would like to thank Brian Kernighan of Bell Laboratories for invaluable assistance during the testing and debugging of gawk, and for help in clarifying numerous points about the language. We could not have done nearly as good a job on either gawk or its documentation without his help.
Chuck Toporek, Mary Sheehan, and Claire Coutier of O'Reilly & Associates contributed significant editorial help for this Web page for the 3.1 release of gawk.
I must thank my wonderful wife, Miriam, for her patience through the many versions of this project, for her proofreading, and for sharing me with the computer. I would like to thank my parents for their love, and for the grace with which they raised and educated me. Finally, I also must acknowledge my gratitude to G-d, for the many opportunities He has sent my way, as well as for the gifts He has given me with which to take advantage of those opportunities.
Arnold Robbins
The basic function of awk is to search files for lines (or other units of text) that contain certain patterns. When a line matches one of the patterns, awk performs specified actions on that line. awk keeps processing input lines in this way until it reaches the end of the input files.
Programs in awk are different from programs in most other languages, because awk programs are data-driven; that is, you describe the data you want to work with and then what to do when you find it. Most other languages are procedural; you have to describe, in great detail, every step the program is to take. When working with procedural languages, it is usually much harder to clearly describe the data your program will process. For this reason, awk programs are often refreshingly easy to read and write.
When you run awk, you specify an awk program that tells awk what to do. The program consists of a series of rules. (It may also contain function definitions, an advanced feature that we will ignore for now. See User-defined.) Each rule specifies one pattern to search for and one action to perform upon finding the pattern.
Syntactically, a rule consists of a pattern followed by an action. The action is enclosed in curly braces to separate it from the pattern. Newlines usually separate rules. Therefore, an awk program looks like this:
pattern { action }
pattern { action }
...
There are several ways to run an awk program. If the program is short, it is easiest to include it in the command that runs awk, like this:
awk 'program' input-file1 input-file2 ...
When the program is long, it is usually more convenient to put it in a file and run it with a command like this:
awk -f program-file input-file1 input-file2 ...
This section discusses both mechanisms, along with several variations of each.
Once you are familiar with awk, you will often type in simple programs the moment you want to use them. Then you can write the program as the first argument of the awk command, like this:
awk 'program' input-file1 input-file2 ...
where program consists of a series of patterns and actions, as described earlier.
This command format instructs the shell, or command interpreter, to start awk and use the program to process records in the input file(s). There are single quotes around program so the shell won't interpret any awk characters as special shell characters. The quotes also cause the shell to treat all of program as a single argument for awk, and allow program to be more than one line long.
This format is also useful for running short or medium-sized awk programs from shell scripts, because it avoids the need for a separate file for the awk program. A self-contained shell script is more reliable because there are no other files to misplace.
Very Simple, later in this chapter, presents several short, self-contained programs.
You can also run awk without any input files. If you type the following command line:
awk 'program'
awk applies the program to the standard input, which usually means whatever you type on the terminal. This continues until you indicate end-of-file by typing Ctrl-d. (On other operating systems, the end-of-file character may be different. For example, on OS/2 and MS-DOS, it is Ctrl-z.)
As an example, the following program prints a friendly piece of advice
(from Douglas Adams's The Hitchhiker's Guide to the Galaxy),
to keep you from worrying about the complexities of computer programming
(BEGIN is a feature we haven't discussed yet):
$ awk "BEGIN { print \"Don't Panic!\" }"
-| Don't Panic!
This program does not read any input. The `\' before each of the inner double quotes is necessary because of the shell's quoting rules—in particular because it mixes both single quotes and double quotes.6
This next simple awk program emulates the cat utility; it copies whatever you type on the keyboard to its standard output (why this works is explained shortly).
$ awk '{ print }'
Now is the time for all good men
-| Now is the time for all good men
to come to the aid of their country.
-| to come to the aid of their country.
Four score and seven years ago, ...
-| Four score and seven years ago, ...
What, me worry?
-| What, me worry?
Ctrl-d
Sometimes your awk programs can be very long. In this case, it is more convenient to put the program into a separate file. In order to tell awk to use that file for its program, you type:
awk -f source-file input-file1 input-file2 ...
The -f instructs the awk utility to get the awk program from the file source-file. Any file name can be used for source-file. For example, you could put the program:
BEGIN { print "Don't Panic!" }
into the file advice. Then this command:
awk -f advice
does the same thing as this one:
awk "BEGIN { print \"Don't Panic!\" }"
This was explained earlier (see Read Terminal). Note that you don't usually need single quotes around the file name that you specify with -f, because most file names don't contain any of the shell's special characters. Notice that in advice, the awk program did not have single quotes around it. The quotes are only needed for programs that are provided on the awk command line.
If you want to identify your awk program files clearly as such, you can add the extension .awk to the file name. This doesn't affect the execution of the awk program but it does make “housekeeping” easier.
Once you have learned awk, you may want to write self-contained awk scripts, using the `#!' script mechanism. You can do this on many Unix systems7 as well as on the GNU system. For example, you could update the file advice to look like this:
#! /bin/awk -f
BEGIN { print "Don't Panic!" }
After making this file executable (with the chmod utility), simply type `advice' at the shell and the system arranges to run awk8 as if you had typed `awk -f advice':
$ chmod +x advice
$ advice
-| Don't Panic!
(We assume you have the current directory in your shell's search
path variable (typically $PATH). If not, you may need
to type `./advice' at the shell.)
Self-contained awk scripts are useful when you want to write a program that users can invoke without their having to know that the program is written in awk.
Some systems limit the length of the interpreter name to 32 characters. Often, this can be dealt with by using a symbolic link.
You should not put more than one argument on the `#!' line after the path to awk. It does not work. The operating system treats the rest of the line as a single argument and passes it to awk. Doing this leads to confusing behavior—most likely a usage diagnostic of some sort from awk.
Finally,
the value of ARGV[0]
(see Built-in Variables)
varies depending upon your operating system.
Some systems put `awk' there, some put the full pathname
of awk (such as /bin/awk), and some put the name
of your script (`advice'). Don't rely on the value of ARGV[0]
to provide your script name.
A comment is some text that is included in a program for the sake of human readers; it is not really an executable part of the program. Comments can explain what the program does and how it works. Nearly all programming languages have provisions for comments, as programs are typically hard to understand without them.
In the awk language, a comment starts with the sharp sign character (`#') and continues to the end of the line. The `#' does not have to be the first character on the line. The awk language ignores the rest of a line following a sharp sign. For example, we could have put the following into advice:
# This program prints a nice friendly message. It helps
# keep novice users from being afraid of the computer.
BEGIN { print "Don't Panic!" }
You can put comment lines into keyboard-composed throwaway awk programs, but this usually isn't very useful; the purpose of a comment is to help you or another person understand the program when reading it at a later time.
Caution: As mentioned in One-shot, you can enclose small to medium programs in single quotes, in order to keep your shell scripts self-contained. When doing so, don't put an apostrophe (i.e., a single quote) into a comment (or anywhere else in your program). The shell interprets the quote as the closing quote for the entire program. As a result, usually the shell prints a message about mismatched quotes, and if awk actually runs, it will probably print strange messages about syntax errors. For example, look at the following:
$ awk '{ print "hello" } # let's be cute'
>
The shell sees that the first two quotes match, and that a new quoted object begins at the end of the command line. It therefore prompts with the secondary prompt, waiting for more input. With Unix awk, closing the quoted string produces this result:
$ awk '{ print "hello" } # let's be cute'
> '
error--> awk: can't open file be
error--> source line number 1
Putting a backslash before the single quote in `let's' wouldn't help, since backslashes are not special inside single quotes. The next subsection describes the shell's quoting rules.
For short to medium length awk programs, it is most convenient to enter the program on the awk command line. This is best done by enclosing the entire program in single quotes. This is true whether you are entering the program interactively at the shell prompt, or writing it as part of a larger shell script:
awk 'program text' input-file1 input-file2 ...
Once you are working with the shell, it is helpful to have a basic knowledge of shell quoting rules. The following rules apply only to POSIX-compliant, Bourne-style shells (such as bash, the GNU Bourne-Again Shell). If you use csh, you're on your own.
Since certain characters within double-quoted text are processed by the shell, they must be escaped within the text. Of note are the characters `$', ``', `\', and `"', all of which must be preceded by a backslash within double-quoted text if they are to be passed on literally to the program. (The leading backslash is stripped first.) Thus, the example seen previously in Read Terminal, is applicable:
$ awk "BEGIN { print \"Don't Panic!\" }"
-| Don't Panic!
Note that the single quote is not special within double quotes.
FS should
be set to the null string, use:
awk -F "" 'program' files # correct
awk -F"" 'program' files # wrong!
In the second case, awk will attempt to use the text of the program
as the value of FS, and the first file name as the text of the program!
This results in syntax errors at best, and confusing behavior at worst.
Mixing single and double quotes is difficult. You have to resort to shell quoting tricks, like this:
$ awk 'BEGIN { print "Here is a single quote <'"'"'>" }'
-| Here is a single quote <'>
This program consists of three concatenated quoted strings. The first and the third are single-quoted, the second is double-quoted.
This can be “simplified” to:
$ awk 'BEGIN { print "Here is a single quote <'\''>" }'
-| Here is a single quote <'>
Judge for yourself which of these two is the more readable.
Another option is to use double quotes, escaping the embedded, awk-level double quotes:
$ awk "BEGIN { print \"Here is a single quote <'>\" }"
-| Here is a single quote <'>
This option is also painful, because double quotes, backslashes, and dollar signs are very common in awk programs.
A third option is to use the octal escape sequence equivalents for the single- and double-quote characters, like so:
$ awk 'BEGIN { print "Here is a single quote <\47>" }'
-| Here is a single quote <'>
$ awk 'BEGIN { print "Here is a double quote <\42>" }'
-| Here is a double quote <">
This works nicely, except that you should comment clearly what the escapes mean.
A fourth option is to use command-line variable assignment, like this:
$ awk -v sq="'" 'BEGIN { print "Here is a single quote <" sq ">" }'
-| Here is a single quote <'>
If you really need both single and double quotes in your awk program, it is probably best to move it into a separate file, where the shell won't be part of the picture, and you can say what you mean.
Many of the examples in this Web page take their input from two sample data files. The first, BBS-list, represents a list of computer bulletin board systems together with information about those systems. The second data file, called inventory-shipped, contains information about monthly shipments. In both files, each line is considered to be one record.
In the data file BBS-list, each record contains the name of a computer bulletin board, its phone number, the board's baud rate(s), and a code for the number of hours it is operational. An `A' in the last column means the board operates 24 hours a day. A `B' in the last column means the board only operates on evening and weekend hours. A `C' means the board operates only on weekends:
aardvark 555-5553 1200/300 B
alpo-net 555-3412 2400/1200/300 A
barfly 555-7685 1200/300 A
bites 555-1675 2400/1200/300 A
camelot 555-0542 300 C
core 555-2912 1200/300 C
fooey 555-1234 2400/1200/300 B
foot 555-6699 1200/300 B
macfoo 555-6480 1200/300 A
sdace 555-3430 2400/1200/300 A
sabafoo 555-2127 1200/300 C
The data file inventory-shipped represents information about shipments during the year. Each record contains the month, the number of green crates shipped, the number of red boxes shipped, the number of orange bags shipped, and the number of blue packages shipped, respectively. There are 16 entries, covering the 12 months of last year and the first four months of the current year.
Jan 13 25 15 115
Feb 15 32 24 226
Mar 15 24 34 228
Apr 31 52 63 420
May 16 34 29 208
Jun 31 42 75 492
Jul 24 34 67 436
Aug 15 34 47 316
Sep 13 55 37 277
Oct 29 54 68 525
Nov 20 87 82 577
Dec 17 35 61 401
Jan 21 36 64 620
Feb 26 58 80 652
Mar 24 75 70 495
Apr 21 70 74 514
The following command runs a simple awk program that searches the input file BBS-list for the character string `foo' (a grouping of characters is usually called a string; the term string is based on similar usage in English, such as “a string of pearls,” or “a string of cars in a train”):
awk '/foo/ { print $0 }' BBS-list
When lines containing `foo' are found, they are printed because `print $0' means print the current line. (Just `print' by itself means the same thing, so we could have written that instead.)
You will notice that slashes (`/') surround the string `foo' in the awk program. The slashes indicate that `foo' is the pattern to search for. This type of pattern is called a regular expression, which is covered in more detail later (see Regexp). The pattern is allowed to match parts of words. There are single quotes around the awk program so that the shell won't interpret any of it as special shell characters.
Here is what this program prints:
$ awk '/foo/ { print $0 }' BBS-list
-| fooey 555-1234 2400/1200/300 B
-| foot 555-6699 1200/300 B
-| macfoo 555-6480 1200/300 A
-| sabafoo 555-2127 1200/300 C
In an awk rule, either the pattern or the action can be omitted, but not both. If the pattern is omitted, then the action is performed for every input line. If the action is omitted, the default action is to print all lines that match the pattern.
Thus, we could leave out the action (the print statement and the curly
braces) in the previous example and the result would be the same: all
lines matching the pattern `foo' are printed. By comparison,
omitting the print statement but retaining the curly braces makes an
empty action that does nothing (i.e., no lines are printed).
Many practical awk programs are just a line or two. Following is a collection of useful, short programs to get you started. Some of these programs contain constructs that haven't been covered yet. (The description of the program will give you a good idea of what is going on, but please read the rest of the Web page to become an awk expert!) Most of the examples use a data file named data. This is just a placeholder; if you use these programs yourself, substitute your own file names for data. For future reference, note that there is often more than one way to do things in awk. At some point, you may want to look back at these examples and see if you can come up with different ways to do the same things shown here:
awk '{ if (length($0) > max) max = length($0) }
END { print max }' data
awk 'length($0) > 80' data
The sole rule has a relational expression as its pattern and it has no action—so the default action, printing the record, is used.
expand data | awk '{ if (x < length()) x = length() }
END { print "maximum line length is " x }'
The input is processed by the expand utility to change tabs into spaces, so the widths compared are actually the right-margin columns.
awk 'NF > 0' data
This is an easy way to delete blank lines from a file (or rather, to create a new file similar to the old file but from which the blank lines have been removed).
awk 'BEGIN { for (i = 1; i <= 7; i++)
print int(101 * rand()) }'
ls -l files | awk '{ x += $5 }
END { print "total bytes: " x }'
ls -l files | awk '{ x += $5 }
END { print "total K-bytes: " (x + 1023)/1024 }'
awk -F: '{ print $1 }' /etc/passwd | sort
awk 'END { print NR }' data
awk 'NR % 2 == 0' data
If you use the expression `NR % 2 == 1' instead, the program would print the odd-numbered lines.
The awk utility reads the input files one line at a time. For each line, awk tries the patterns of each of the rules. If several patterns match, then several actions are run in the order in which they appear in the awk program. If no patterns match, then no actions are run.
After processing all the rules that match the line (and perhaps there are none), awk reads the next line. (However, see Next Statement, and also see Nextfile Statement). This continues until the program reaches the end of the file. For example, the following awk program contains two rules:
/12/ { print $0 }
/21/ { print $0 }
The first rule has the string `12' as the pattern and `print $0' as the action. The second rule has the string `21' as the pattern and also has `print $0' as the action. Each rule's action is enclosed in its own pair of braces.
This program prints every line that contains the string `12' or the string `21'. If a line contains both strings, it is printed twice, once by each rule.
This is what happens if we run this program on our two sample data files, BBS-list and inventory-shipped:
$ awk '/12/ { print $0 }
> /21/ { print $0 }' BBS-list inventory-shipped
-| aardvark 555-5553 1200/300 B
-| alpo-net 555-3412 2400/1200/300 A
-| barfly 555-7685 1200/300 A
-| bites 555-1675 2400/1200/300 A
-| core 555-2912 1200/300 C
-| fooey 555-1234 2400/1200/300 B
-| foot 555-6699 1200/300 B
-| macfoo 555-6480 1200/300 A
-| sdace 555-3430 2400/1200/300 A
-| sabafoo 555-2127 1200/300 C
-| sabafoo 555-2127 1200/300 C
-| Jan 21 36 64 620
-| Apr 21 70 74 514
Note how the line beginning with `sabafoo' in BBS-list was printed twice, once for each rule.
Now that we've mastered some simple tasks, let's look at what typical awk programs do. This example shows how awk can be used to summarize, select, and rearrange the output of another utility. It uses features that haven't been covered yet, so don't worry if you don't understand all the details:
ls -l | awk '$6 == "Nov" { sum += $5 }
END { print sum }'
This command prints the total number of bytes in all the files in the current directory that were last modified in November (of any year). 9 The `ls -l' part of this example is a system command that gives you a listing of the files in a directory, including each file's size and the date the file was last modified. Its output looks like this:
-rw-r--r-- 1 arnold user 1933 Nov 7 13:05 Makefile
-rw-r--r-- 1 arnold user 10809 Nov 7 13:03 awk.h
-rw-r--r-- 1 arnold user 983 Apr 13 12:14 awk.tab.h
-rw-r--r-- 1 arnold user 31869 Jun 15 12:20 awk.y
-rw-r--r-- 1 arnold user 22414 Nov 7 13:03 awk1.c
-rw-r--r-- 1 arnold user 37455 Nov 7 13:03 awk2.c
-rw-r--r-- 1 arnold user 27511 Dec 9 13:07 awk3.c
-rw-r--r-- 1 arnold user 7989 Nov 7 13:03 awk4.c
The first field contains read-write permissions, the second field contains the number of links to the file, and the third field identifies the owner of the file. The fourth field identifies the group of the file. The fifth field contains the size of the file in bytes. The sixth, seventh, and eighth fields contain the month, day, and time, respectively, that the file was last modified. Finally, the ninth field contains the name of the file.10
The `$6 == "Nov"' in our awk program is an expression that
tests whether the sixth field of the output from `ls -l'
matches the string `Nov'. Each time a line has the string
`Nov' for its sixth field, the action `sum += $5' is
performed. This adds the fifth field (the file's size) to the variable
sum. As a result, when awk has finished reading all the
input lines, sum is the total of the sizes of the files whose
lines matched the pattern. (This works because awk variables
are automatically initialized to zero.)
After the last line of output from ls has been processed, the
END rule executes and prints the value of sum.
In this example, the value of sum is 80600.
These more advanced awk techniques are covered in later sections
(see Action Overview). Before you can move on to more
advanced awk programming, you have to know how awk interprets
your input and displays your output. By manipulating fields and using
print statements, you can produce some very useful and
impressive-looking reports.
Most often, each line in an awk program is a separate statement or separate rule, like this:
awk '/12/ { print $0 }
/21/ { print $0 }' BBS-list inventory-shipped
However, gawk ignores newlines after any of the following symbols and keywords:
, { ? : || && do else
A newline at any other point is considered the end of the statement.11
If you would like to split a single statement into two lines at a point where a newline would terminate it, you can continue it by ending the first line with a backslash character (`\'). The backslash must be the final character on the line in order to be recognized as a continuation character. A backslash is allowed anywhere in the statement, even in the middle of a string or regular expression. For example:
awk '/This regular expression is too long, so continue it\
on the next line/ { print $1 }'
We have generally not used backslash continuation in the sample programs in this Web page. In gawk, there is no limit on the length of a line, so backslash continuation is never strictly necessary; it just makes programs more readable. For this same reason, as well as for clarity, we have kept most statements short in the sample programs presented throughout the Web page. Backslash continuation is most useful when your awk program is in a separate source file instead of entered from the command line. You should also note that many awk implementations are more particular about where you may use backslash continuation. For example, they may not allow you to split a string constant using backslash continuation. Thus, for maximum portability of your awk programs, it is best not to split your lines in the middle of a regular expression or a string.
Caution: Backslash continuation does not work as described with the C shell. It works for awk programs in files and for one-shot programs, provided you are using a POSIX-compliant shell, such as the Unix Bourne shell or bash. But the C shell behaves differently! There, you must use two backslashes in a row, followed by a newline. Note also that when using the C shell, every newline in your awk program must be escaped with a backslash. To illustrate:
% awk 'BEGIN { \
? print \\
? "hello, world" \
? }'
-| hello, world
Here, the `%' and `?' are the C shell's primary and secondary prompts, analogous to the standard shell's `$' and `>'.
Compare the previous example to how it is done with a POSIX-compliant shell:
$ awk 'BEGIN {
> print \
> "hello, world"
> }'
-| hello, world
awk is a line-oriented language. Each rule's action has to begin on the same line as the pattern. To have the pattern and action on separate lines, you must use backslash continuation; there is no other option.
Another thing to keep in mind is that backslash continuation and comments do not mix. As soon as awk sees the `#' that starts a comment, it ignores everything on the rest of the line. For example:
$ gawk 'BEGIN { print "dont panic" # a friendly \
> BEGIN rule
> }'
error--> gawk: cmd. line:2: BEGIN rule
error--> gawk: cmd. line:2: ^ parse error
In this case, it looks like the backslash would continue the comment onto the
next line. However, the backslash-newline combination is never even
noticed because it is “hidden” inside the comment. Thus, the
BEGIN is noted as a syntax error.
When awk statements within one rule are short, you might want to put more than one of them on a line. This is accomplished by separating the statements with a semicolon (`;'). This also applies to the rules themselves. Thus, the program shown at the start of this section could also be written this way:
/12/ { print $0 } ; /21/ { print $0 }
NOTE: The requirement that states that rules on the same line must be separated with a semicolon was not in the original awk language; it was added for consistency with the treatment of statements within an action.
The awk language provides a number of predefined, or built-in, variables that your programs can use to get information from awk. There are other variables your program can set as well to control how awk processes your data.
In addition, awk provides a number of built-in functions for doing common computational and string-related operations. gawk provides built-in functions for working with timestamps, performing bit manipulation, and for runtime string translation.
As we develop our presentation of the awk language, we introduce most of the variables and many of the functions. They are defined systematically in Built-in Variables, and Built-in.
Now that you've seen some of what awk can do, you might wonder how awk could be useful for you. By using utility programs, advanced patterns, field separators, arithmetic statements, and other selection criteria, you can produce much more complex output. The awk language is very useful for producing reports from large amounts of raw data, such as summarizing information from the output of other utility programs like ls. (See More Complex.)
Programs written with awk are usually much smaller than they would be in other languages. This makes awk programs easy to compose and use. Often, awk programs can be quickly composed at your terminal, used once, and thrown away. Because awk programs are interpreted, you can avoid the (usually lengthy) compilation part of the typical edit-compile-test-debug cycle of software development.
Complex programs have been written in awk, including a complete retargetable assembler for eight-bit microprocessors (see Glossary, for more information), and a microcode assembler for a special-purpose Prolog computer. More recently, gawk was used for writing a Wiki clone.12 While the original awk's capabilities were strained by tasks of such complexity, modern versions are more capable. Even the Bell Labs version of awk has fewer predefined limits, and those that it has are much larger than they used to be.
If you find yourself writing awk scripts of more than, say, a few hundred lines, you might consider using a different programming language. Emacs Lisp is a good choice if you need sophisticated string or pattern matching capabilities. The shell is also good at string and pattern matching; in addition, it allows powerful use of the system utilities. More conventional languages, such as C, C++, and Java, offer better facilities for system programming and for managing the complexity of large programs. Programs in these languages may require more lines of source code than the equivalent awk programs, but they are easier to maintain and usually run more efficiently.
A regular expression, or regexp, is a way of describing a set of strings. Because regular expressions are such a fundamental part of awk programming, their format and use deserve a separate chapter.
A regular expression enclosed in slashes (`/')
is an awk pattern that matches every input record whose text
belongs to that set.
The simplest regular expression is a sequence of letters, numbers, or
both. Such a regexp matches any string that contains that sequence.
Thus, the regexp `foo' matches any string containing `foo'.
Therefore, the pattern /foo/ matches any input record containing
the three characters `foo' anywhere in the record. Other
kinds of regexps let you specify more complicated classes of strings.
Initially, the examples in this chapter are simple. As we explain more about how regular expressions work, we will present more complicated instances.
A regular expression can be used as a pattern by enclosing it in slashes. Then the regular expression is tested against the entire text of each record. (Normally, it only needs to match some part of the text in order to succeed.) For example, the following prints the second field of each record that contains the string `foo' anywhere in it:
$ awk '/foo/ { print $2 }' BBS-list
-| 555-1234
-| 555-6699
-| 555-6480
-| 555-2127
~ (tilde), ~ operator
Regular expressions can also be used in matching expressions. These
expressions allow you to specify the string to match against; it need
not be the entire current input record. The two operators `~'
and `!~' perform regular expression comparisons. Expressions
using these operators can be used as patterns, or in if,
while, for, and do statements.
(See Statements.)
For example:
exp ~ /regexp/
is true if the expression exp (taken as a string) matches regexp. The following example matches, or selects, all input records with the uppercase letter `J' somewhere in the first field:
$ awk '$1 ~ /J/' inventory-shipped
-| Jan 13 25 15 115
-| Jun 31 42 75 492
-| Jul 24 34 67 436
-| Jan 21 36 64 620
So does this:
awk '{ if ($1 ~ /J/) print }' inventory-shipped
This next example is true if the expression exp (taken as a character string) does not match regexp:
exp !~ /regexp/
The following example matches, or selects, all input records whose first field does not contain the uppercase letter `J':
$ awk '$1 !~ /J/' inventory-shipped
-| Feb 15 32 24 226
-| Mar 15 24 34 228
-| Apr 31 52 63 420
-| May 16 34 29 208
...
When a regexp is enclosed in slashes, such as /foo/, we call it
a regexp constant, much like 5.27 is a numeric constant and
"foo" is a string constant.
Some characters cannot be included literally in string constants
("foo") or regexp constants (/foo/).
Instead, they should be represented with escape sequences,
which are character sequences beginning with a backslash (`\').
One use of an escape sequence is to include a double-quote character in
a string constant. Because a plain double quote ends the string, you
must use `\"' to represent an actual double-quote character as a
part of the string. For example:
$ awk 'BEGIN { print "He said \"hi!\" to her." }'
-| He said "hi!" to her.
The backslash character itself is another character that cannot be
included normally; you must write `\\' to put one backslash in the
string or regexp. Thus, the string whose contents are the two characters
`"' and `\' must be written "\"\\".
Backslash also represents unprintable characters such as TAB or newline. While there is nothing to stop you from entering most unprintable characters directly in a string constant or regexp constant, they may look ugly.
The following table lists all the escape sequences used in awk and what they represent. Unless noted otherwise, all these escape sequences apply to both string constants and regexp constants:
\\\a\b\f\n\r\t\v\nnn\xhh...\/\"In gawk, a number of additional two-character sequences that begin with a backslash have special meaning in regexps. See GNU Regexp Operators.
In a regexp, a backslash before any character that is not in the previous list
and not listed in
GNU Regexp Operators,
means that the next character should be taken literally, even if it would
normally be a regexp operator. For example, /a\+b/ matches the three
characters `a+b'.
For complete portability, do not use a backslash before any character not shown in the previous list.
To summarize:
If you place a backslash in a string constant before something that is not one of the characters previously listed, POSIX awk purposely leaves what happens as undefined. There are two choices:
"a\qc" is the same as "aqc".
(Because this is such an easy bug both to introduce and to miss,
gawk warns you about it.)
Consider `FS = "[ \t]+\|[ \t]+"' to use vertical bars
surrounded by whitespace as the field separator. There should be
two backslashes in the string `FS = "[ \t]+\\|[ \t]+"'.)
"a\qc" is the same as typing
"a\\qc".
Suppose you use an octal or hexadecimal escape to represent a regexp metacharacter. (See Regexp Operators.) Does awk treat the character as a literal character or as a regexp operator?
Historically, such characters were taken literally.
(d.c.)
However, the POSIX standard indicates that they should be treated
as real metacharacters, which is what gawk does.
In compatibility mode (see Options),
gawk treats the characters represented by octal and hexadecimal
escape sequences literally when used in regexp constants. Thus,
/a\52b/ is equivalent to /a\*b/.
You can combine regular expressions with special characters, called regular expression operators or metacharacters, to increase the power and versatility of regular expressions.
The escape sequences described earlier in Escape Sequences, are valid inside a regexp. They are introduced by a `\' and are recognized and converted into corresponding real characters as the very first step in processing regexps.
Here is a list of metacharacters. All characters that are not escape sequences and that are not listed in the table stand for themselves:
\^It is important to realize that `^' does not match the beginning of a line embedded in a string. The condition is not true in the following example:
if ("line1\nLINE 2" ~ /^L/) ...
$ if ("line1\nLINE 2" ~ /1$/) ...
.In strict POSIX mode (see Options), `.' does not match the nul character, which is a character with all bits equal to zero. Otherwise, nul is just another character. Other versions of awk may not be able to match the nul character.
[...][^ ...]|The alternation applies to the largest possible regexps on either side.
(...)*The `*' repeats the smallest possible preceding expression. (Use parentheses if you want to repeat a larger expression.) It finds as many repetitions as possible. For example, `awk '/\(c[ad][ad]*r x\)/ { print }' sample' prints every record in sample containing a string of the form `(car x)', `(cdr x)', `(cadr x)', and so on. Notice the escaping of the parentheses by preceding them with backslashes.
+ awk '/\(c[ad]+r x\)/ { print }' sample
?{n}{n,}{n,m}wh{3}ywh{3,5}ywh{2,}yInterval expressions were not traditionally available in awk. They were added as part of the POSIX standard to make awk and egrep consistent with each other.
However, because old programs may use `{' and `}' in regexp constants, by default gawk does not match interval expressions in regexps. If either --posix or --re-interval are specified (see Options), then interval expressions are allowed in regexps.
For new programs that use `{' and `}' in regexp constants, it is good practice to always escape them with a backslash. Then the regexp constants are valid and work the way you want them to, using any version of awk.14
In regular expressions, the `*', `+', and `?' operators, as well as the braces `{' and `}', have the highest precedence, followed by concatenation, and finally by `|'. As in arithmetic, parentheses can change how operators are grouped.
In POSIX awk and gawk, the `*', `+', and `?' operators stand for themselves when there is nothing in the regexp that precedes them. For example, `/+/' matches a literal plus sign. However, many other versions of awk treat such a usage as a syntax error.
If gawk is in compatibility mode (see Options), POSIX character classes and interval expressions are not available in regular expressions.
Within a character list, a range expression consists of two characters separated by a hyphen. It matches any single character that sorts between the two characters, using the locale's collating sequence and character set. For example, in the default C locale, `[a-dx-z]' is equivalent to `[abcdxyz]'. Many locales sort characters in dictionary order, and in these locales, `[a-dx-z]' is typically not equivalent to `[abcdxyz]'; instead it might be equivalent to `[aBbCcDdxXyYz]', for example. To obtain the traditional interpretation of bracket expressions, you can use the C locale by setting the LC_ALL environment variable to the value `C'.
To include one of the characters `\', `]', `-', or `^' in a character list, put a `\' in front of it. For example:
[d\]]
matches either `d' or `]'.
This treatment of `\' in character lists is compatible with other awk implementations and is also mandated by POSIX. The regular expressions in awk are a superset of the POSIX specification for Extended Regular Expressions (EREs). POSIX EREs are based on the regular expressions accepted by the traditional egrep utility.
Character classes are a new feature introduced in the POSIX standard. A character class is a special notation for describing lists of characters that have a specific attribute, but the actual characters can vary from country to country and/or from character set to character set. For example, the notion of what is an alphabetic character differs between the United States and France.
A character class is only valid in a regexp inside the brackets of a character list. Character classes consist of `[:', a keyword denoting the class, and `:]'. table-char-classes lists the character classes defined by the POSIX standard.
| Class | Meaning
|
|---|---|
[:alnum:] | Alphanumeric characters.
|
[:alpha:] | Alphabetic characters.
|
[:blank:] | Space and TAB characters.
|
[:cntrl:] | Control characters.
|
[:digit:] | Numeric characters.
|
[:graph:] | Characters that are both printable and visible.
(A space is printable but not visible, whereas an `a' is both.)
|
[:lower:] | Lowercase alphabetic characters.
|
[:print:] | Printable characters (characters that are not control characters).
|
[:punct:] | Punctuation characters (characters that are not letters, digits,
control characters, or space characters).
|
[:space:] | Space characters (such as space, TAB, and formfeed, to name a few).
|
[:upper:] | Uppercase alphabetic characters.
|
[:xdigit:] | Characters that are hexadecimal digits.
|
Table 2.1: POSIX Character Classes
For example, before the POSIX standard, you had to write /[A-Za-z0-9]/
to match alphanumeric characters. If your
character set had other alphabetic characters in it, this would not
match them, and if your character set collated differently from
ASCII, this might not even match the ASCII alphanumeric characters.
With the POSIX character classes, you can write
/[[:alnum:]]/ to match the alphabetic
and numeric characters in your character set.
Two additional special sequences can appear in character lists. These apply to non-ASCII character sets, which can have single symbols (called collating elements) that are represented with more than one character. They can also have several characters that are equivalent for collating, or sorting, purposes. (For example, in French, a plain “e” and a grave-accented “è” are equivalent.) These sequences are:
[[.ch.]] is a regexp that matches this collating element, whereas
[ch] is a regexp that matches either `c' or `h'.
[[=e=]] is a regexp
that matches any of `e', `é', or `è'.
These features are very valuable in non-English-speaking locales.
Caution: The library functions that gawk uses for regular expression matching currently recognize only POSIX character classes; they do not recognize collating symbols or equivalence classes.
GNU software that deals with regular expressions provides a number of additional regexp operators. These operators are described in this section and are specific to gawk; they are not available in other awk implementations. Most of the additional operators deal with word matching. For our purposes, a word is a sequence of one or more letters, digits, or underscores (`_'):
\w[[:alnum:]_].
\W[^[:alnum:]_].
\</\<away/ matches `away' but not
`stowaway'.
\>/stow\>/ matches `stow' but not `stowaway'.
\y\B/\Brat\B/ matches `crate' but it does not match `dirty rat'.
`\B' is essentially the opposite of `\y'.
There are two other operators that work on buffers. In Emacs, a buffer is, naturally, an Emacs buffer. For other programs, gawk's regexp library routines consider the entire string to match as the buffer. The operators are:
\`\'Because `^' and `$' always work in terms of the beginning and end of strings, these operators don't add any new capabilities for awk. They are provided for compatibility with other GNU software.
In other GNU software, the word-boundary operator is `\b'. However, that conflicts with the awk language's definition of `\b' as backspace, so gawk uses a different letter. An alternative method would have been to require two backslashes in the GNU operators, but this was deemed too confusing. The current method of using `\y' for the GNU `\b' appears to be the lesser of two evils.
The various command-line options (see Options) control how gawk interprets characters in regexps:
--posix--traditional[[:alnum:]], etc.).
Characters described by octal and hexadecimal escape sequences are
treated literally, even if they represent regexp metacharacters.
--re-intervalCase is normally significant in regular expressions, both when matching ordinary characters (i.e., not metacharacters) and inside character sets. Thus, a `w' in a regular expression matches only a lowercase `w' and not an uppercase `W'.
The simplest way to do a case-independent match is to use a character list—for example, `[Ww]'. However, this can be cumbersome if you need to use it often, and it can make the regular expressions harder to read. There are two alternatives that you might prefer.
One way to perform a case-insensitive match at a particular point in the
program is to convert the data to a single case, using the
tolower or toupper built-in string functions (which we
haven't discussed yet;
see String Functions).
For example:
tolower($1) ~ /foo/ { ... }
converts the first field to lowercase before matching against it. This works in any POSIX-compliant awk.
Another method, specific to gawk, is to set the variable
IGNORECASE to a nonzero value (see Built-in Variables).
When IGNORECASE is not zero, all regexp and string
operations ignore case. Changing the value of
IGNORECASE dynamically controls the case-sensitivity of the
program as it runs. Case is significant by default because
IGNORECASE (like most variables) is initialized to zero:
x = "aB"
if (x ~ /ab/) ... # this test will fail
IGNORECASE = 1
if (x ~ /ab/) ... # now it will succeed
In general, you cannot use IGNORECASE to make certain rules
case-insensitive and other rules case-sensitive, because there is no
straightforward way
to set IGNORECASE just for the pattern of
a particular rule.15
To do this, use either character lists or tolower. However, one
thing you can do with IGNORECASE only is dynamically turn
case-sensitivity on or off for all the rules at once.
IGNORECASE can be set on the command line or in a BEGIN rule
(see Other Arguments; also
see Using BEGIN/END).
Setting IGNORECASE from the command line is a way to make
a program case-insensitive without having to edit it.
Prior to gawk 3.0, the value of IGNORECASE
affected regexp operations only. It did not affect string comparison
with `==', `!=', and so on.
Beginning with version 3.0, both regexp and string comparison
operations are also affected by IGNORECASE.
Beginning with gawk 3.0, the equivalences between upper- and lowercase characters are based on the ISO-8859-1 (ISO Latin-1) character set. This character set is a superset of the traditional 128 ASCII characters, which also provides a number of characters suitable for use with European languages.
As of gawk 3.1.4, the case equivalencies are fully
locale-aware. They are based on the C <ctype.h> facilities,
such as isalpha() and toupper().
The value of IGNORECASE has no effect if gawk is in
compatibility mode (see Options).
Case is always significant in compatibility mode.
echo aaaabcd | awk '{ sub(/a+/, "<A>"); print }'
This example uses the sub function (which we haven't discussed yet;
see String Functions)
to make a change to the input record. Here, the regexp /a+/
indicates “one or more `a' characters,” and the replacement
text is `<A>'.
The input contains four `a' characters. awk (and POSIX) regular expressions always match the leftmost, longest sequence of input characters that can match. Thus, all four `a' characters are replaced with `<A>' in this example:
$ echo aaaabcd | awk '{ sub(/a+/, "<A>"); print }'
-| <A>bcd
For simple match/no-match tests, this is not so important. But when doing
text matching and substitutions with the match, sub, gsub,
and gensub functions, it is very important.
Understanding this principle is also important for regexp-based record
and field splitting (see Records,
and also see Field Separators).
The righthand side of a `~' or `!~' operator need not be a regexp constant (i.e., a string of characters between slashes). It may be any expression. The expression is evaluated and converted to a string if necessary; the contents of the string are used as the regexp. A regexp that is computed in this way is called a dynamic regexp:
BEGIN { digits_regexp = "[[:digit:]]+" }
$0 ~ digits_regexp { print }
This sets digits_regexp to a regexp that describes one or more digits,
and tests whether the input record matches this regexp.
Caution: When using the `~' and `!~'
operators, there is a difference between a regexp constant
enclosed in slashes and a string constant enclosed in double quotes.
If you are going to use a string constant, you have to understand that
the string is, in essence, scanned twice: the first time when
awk reads your program, and the second time when it goes to
match the string on the lefthand side of the operator with the pattern
on the right. This is true of any string-valued expression (such as
digits_regexp, shown previously), not just string constants.
What difference does it make if the string is scanned twice? The answer has to do with escape sequences, and particularly with backslashes. To get a backslash into a regular expression inside a string, you have to type two backslashes.
For example, /\*/ is a regexp constant for a literal `*'.
Only one backslash is needed. To do the same thing with a string,
you have to type "\\*". The first backslash escapes the
second one so that the string actually contains the
two characters `\' and `*'.
Given that you can use both regexp and string constants to describe regular expressions, which should you use? The answer is “regexp constants,” for several reasons:
\n in Character Lists of Dynamic RegexpsSome commercial versions of awk do not allow the newline character to be used inside a character list for a dynamic regexp:
$ awk '$0 ~ "[ \t\n]"'
error--> awk: newline in character class [
error--> ]...
error--> source line number 1
error--> context is
error--> >>> <<<
But a newline in a regexp constant works with no problem:
$ awk '$0 ~ /[ \t\n]/'
here is a sample line
-| here is a sample line
Ctrl-d
gawk does not have this problem, and it isn't likely to occur often in practice, but it's worth noting for future reference.
Modern systems support the notion of locales: a way to tell the system about the local character set and language. The current locale setting can affect the way regexp matching works, often in surprising ways. In particular, many locales do case-insensitive matching, even when you may have specified characters of only one particular case.
The following example uses the sub function, which
does text replacement
(see String Functions).
Here, the intent is to remove trailing uppercase characters:
$ echo something1234abc | gawk '{ sub("[A-Z]*$", ""); print }'
-| something1234
This output is unexpected, since the `abc' at the end of `something1234abc' should not normally match `[A-Z]*'. This result is due to the locale setting (and thus you may not see it on your system). There are two fixes. The first is to use the POSIX character class `[[:upper:]]', instead of `[A-Z]'. The second is to change the locale setting in the environment, before running gawk, by using the shell statements:
LANG=C LC_ALL=C
export LANG LC_ALL
The setting `C' forces gawk to behave in the traditional Unix manner, where case distinctions do matter. You may wish to put these statements into your shell startup file, e.g., $HOME/.profile.
Similar considerations apply to other ranges. For example, `["-/]' is perfectly valid in ASCII, but is not valid in many Unicode locales, such as `en_US.UTF-8'. (In general, such ranges should be avoided; either list the characters individually, or use a POSIX character class such as `[[:punct:]]'.)
For the normal case of `RS = "\n"', the locale is largely irrelevant. For other single byte record separators, using `LC_ALL=C' will give you much better performance when reading records. Otherwise, gawk has to make several function calls, per input character to find the record terminator.
In the typical awk program, all input is read either from the
standard input (by default, this is the keyboard, but often it is a pipe from another
command) or from files whose names you specify on the awk
command line. If you specify input files, awk reads them
in order, processing all the data from one before going on to the next.
The name of the current input file can be found in the built-in variable
FILENAME
(see Built-in Variables).
The input is read in units called records, and is processed by the rules of your program one record at a time. By default, each record is one line. Each record is automatically split into chunks called fields. This makes it more convenient for programs to work on the parts of a record.
On rare occasions, you may need to use the getline command.
The getline command is valuable, both because it
can do explicit input from any number of files, and because the files
used with it do not have to be named on the awk command line
(see Getline).
The awk utility divides the input for your awk
program into records and fields.
awk keeps track of the number of records that have
been read
so far
from the current input file. This value is stored in a
built-in variable called FNR. It is reset to zero when a new
file is started. Another built-in variable, NR, is the total
number of input records read so far from all data files. It starts at zero,
but is never automatically reset to zero.
Records are separated by a character called the record separator.
By default, the record separator is the newline character.
This is why records are, by default, single lines.
A different character can be used for the record separator by
assigning the character to the built-in variable RS.
Like any other variable,
the value of RS can be changed in the awk program
with the assignment operator, `='
(see Assignment Ops).
The new record-separator character should be enclosed in quotation marks,
which indicate a string constant. Often the right time to do this is
at the beginning of execution, before any input is processed,
so that the very first record is read with the proper separator.
To do this, use the special BEGIN pattern
(see BEGIN/END).
For example:
awk 'BEGIN { RS = "/" }
{ print $0 }' BBS-list
changes the value of RS to "/", before reading any input.
This is a string whose first character is a slash; as a result, records
are separated by slashes. Then the input file is read, and the second
rule in the awk program (the action with no pattern) prints each
record. Because each print statement adds a newline at the end of
its output, this awk program copies the input
with each slash changed to a newline. Here are the results of running
the program on BBS-list:
$ awk 'BEGIN { RS = "/" }
> { print $0 }' BBS-list
-| aardvark 555-5553 1200
-| 300 B
-| alpo-net 555-3412 2400
-| 1200
-| 300 A
-| barfly 555-7685 1200
-| 300 A
-| bites 555-1675 2400
-| 1200
-| 300 A
-| camelot 555-0542 300 C
-| core 555-2912 1200
-| 300 C
-| fooey 555-1234 2400
-| 1200
-| 300 B
-| foot 555-6699 1200
-| 300 B
-| macfoo 555-6480 1200
-| 300 A
-| sdace 555-3430 2400
-| 1200
-| 300 A
-| sabafoo 555-2127 1200
-| 300 C
-|
Note that the entry for the `camelot' BBS is not split. In the original data file (see Sample Data Files), the line looks like this:
camelot 555-0542 300 C
It has one baud rate only, so there are no slashes in the record, unlike the others which have two or more baud rates. In fact, this record is treated as part of the record for the `core' BBS; the newline separating them in the output is the original newline in the data file, not the one added by awk when it printed the record!
Another way to change the record separator is on the command line, using the variable-assignment feature (see Other Arguments):
awk '{ print $0 }' RS="/" BBS-list
This sets RS to `/' before processing BBS-list.
Using an unusual character such as `/' for the record separator produces correct behavior in the vast majority of cases. However, the following (extreme) pipeline prints a surprising `1':
$ echo | awk 'BEGIN { RS = "a" } ; { print NF }'
-| 1
There is one field, consisting of a newline. The value of the built-in
variable NF is the number of fields in the current record.
Reaching the end of an input file terminates the current input record,
even if the last character in the file is not the character in RS.
(d.c.)
The empty string "" (a string without any characters)
has a special meaning
as the value of RS. It means that records are separated
by one or more blank lines and nothing else.
See Multiple Line, for more details.
If you change the value of RS in the middle of an awk run,
the new value is used to delimit subsequent records, but the record
currently being processed, as well as records already processed, are not
affected.
After the end of the record has been determined, gawk
sets the variable RT to the text in the input that matched
RS.
When using gawk,
the value of RS is not limited to a one-character
string. It can be any regular expression
(see Regexp).
In general, each record
ends at the next string that matches the regular expression; the next
record starts at the end of the matching string. This general rule is
actually at work in the usual case, where RS contains just a
newline: a record ends at the beginning of the next matching string (the
next newline in the input), and the following record starts just after
the end of this string (at the first character of the following line).
The newline, because it matches RS, is not part of either record.
When RS is a single character, RT
contains the same single character. However, when RS is a
regular expression, RT contains
the actual input text that matched the regular expression.
The following example illustrates both of these features.
It sets RS equal to a regular expression that
matches either a newline or a series of one or more uppercase letters
with optional leading and/or trailing whitespace:
$ echo record 1 AAAA record 2 BBBB record 3 |
> gawk 'BEGIN { RS = "\n|( *[[:upper:]]+ *)" }
> { print "Record =", $0, "and RT =", RT }'
-| Record = record 1 and RT = AAAA
-| Record = record 2 and RT = BBBB
-| Record = record 3 and RT =
-|
The final line of output has an extra blank line. This is because the
value of RT is a newline, and the print statement
supplies its own terminating newline.
See Simple Sed, for a more useful example
of RS as a regexp and RT.
If you set RS to a regular expression that allows optional
trailing text, such as `RS = "abc(XYZ)?"' it is possible, due
to implementation constraints, that gawk may match the leading
part of the regular expression, but not the trailing part, particularly
if the input text that could match the trailing part is fairly long.
gawk attempts to avoid this problem, but currently, there's
no guarantee that this will never happen.
The use of RS as a regular expression and the RT
variable are gawk extensions; they are not available in
compatibility mode
(see Options).
In compatibility mode, only the first character of the value of
RS is used to determine the end of the record.
RS = "\0" Is Not PortableThere are times when you might want to treat an entire data file as a
single record. The only way to make this happen is to give RS
a value that you know doesn't occur in the input file. This is hard
to do in a general way, such that a program always works for arbitrary
input files.
You might think that for text files, the nul character, which
consists of a character with all bits equal to zero, is a good
value to use for RS in this case:
BEGIN { RS = "\0" } # whole file becomes one record?
gawk in fact accepts this, and uses the nul character for the record separator. However, this usage is not portable to other awk implementations.
All other awk implementations16 store strings internally as C-style strings. C strings use the nul character as the string terminator. In effect, this means that `RS = "\0"' is the same as `RS = ""'. (d.c.)
The best way to treat a whole file as a single record is to simply read the file in, one record at a time, concatenating each record onto the end of the previous ones.
When awk reads an input record, the record is automatically parsed or separated by the interpreter into chunks called fields. By default, fields are separated by whitespace, like words in a line. Whitespace in awk means any string of one or more spaces, tabs, or newlines;17 other characters, such as formfeed, vertical tab, etc. that are considered whitespace by other languages, are not considered whitespace by awk.
The purpose of fields is to make it more convenient for you to refer to these pieces of the record. You don't have to use them—you can operate on the whole record if you want—but fields are what make simple awk programs so powerful.
A dollar-sign (`$') is used
to refer to a field in an awk program,
followed by the number of the field you want. Thus, $1
refers to the first field, $2 to the second, and so on.
(Unlike the Unix shells, the field numbers are not limited to single digits.
$127 is the one hundred twenty-seventh field in the record.)
For example, suppose the following is a line of input:
This seems like a pretty nice example.
Here the first field, or $1, is `This', the second field, or
$2, is `seems', and so on. Note that the last field,
$7, is `example.'. Because there is no space between the
`e' and the `.', the period is considered part of the seventh
field.
NF is a built-in variable whose value is the number of fields
in the current record. awk automatically updates the value
of NF each time it reads a record. No matter how many fields
there are, the last field in a record can be represented by $NF.
So, $NF is the same as $7, which is `example.'.
If you try to reference a field beyond the last
one (such as $8 when the record has only seven fields), you get
the empty string. (If used in a numeric operation, you get zero.)
The use of $0, which looks like a reference to the “zero-th” field, is
a special case: it represents the whole input record
when you are not interested in specific fields.
Here are some more examples:
$ awk '$1 ~ /foo/ { print $0 }' BBS-list
-| fooey 555-1234 2400/1200/300 B
-| foot 555-6699 1200/300 B
-| macfoo 555-6480 1200/300 A
-| sabafoo 555-2127 1200/300 C
This example prints each record in the file BBS-list whose first
field contains the string `foo'. The operator `~' is called a
matching operator
(see Regexp Usage);
it tests whether a string (here, the field $1) matches a given regular
expression.
By contrast, the following example looks for `foo' in the entire record and prints the first field and the last field for each matching input record:
$ awk '/foo/ { print $1, $NF }' BBS-list
-| fooey B
-| foot B
-| macfoo A
-| sabafoo C
The number of a field does not need to be a constant. Any expression in the awk language can be used after a `$' to refer to a field. The value of the expression specifies the field number. If the value is a string, rather than a number, it is converted to a number. Consider this example:
awk '{ print $NR }'
Recall that NR is the number of records read so far: one in the
first record, two in the second, etc. So this example prints the first
field of the first record, the second field of the second record, and so
on. For the twentieth record, field number 20 is printed; most likely,
the record has fewer than 20 fields, so this prints a blank line.
Here is another example of using expressions as field numbers:
awk '{ print $(2*2) }' BBS-list
awk evaluates the expression `(2*2)' and uses its value as the number of the field to print. The `*' sign represents multiplication, so the expression `2*2' evaluates to four. The parentheses are used so that the multiplication is done before the `$' operation; they are necessary whenever there is a binary operator in the field-number expression. This example, then, prints the hours of operation (the fourth field) for every line of the file BBS-list. (All of the awk operators are listed, in order of decreasing precedence, in Precedence.)
If the field number you compute is zero, you get the entire record.
Thus, `$(2-2)' has the same value as $0. Negative field
numbers are not allowed; trying to reference one usually terminates
the program. (The POSIX standard does not define
what happens when you reference a negative field number. gawk
notices this and terminates your program. Other awk
implementations may behave differently.)
As mentioned in Fields,
awk stores the current record's number of fields in the built-in
variable NF (also see Built-in Variables). The expression
$NF is not a special feature—it is the direct consequence of
evaluating NF and using its value as a field number.
The contents of a field, as seen by awk, can be changed within an awk program; this changes what awk perceives as the current input record. (The actual input is untouched; awk never modifies the input file.) Consider the following example and its output:
$ awk '{ nboxes = $3 ; $3 = $3 - 10
> print nboxes, $3 }' inventory-shipped
-| 25 15
-| 32 22
-| 24 14
...
The program first saves the original value of field three in the variable
nboxes.
The `-' sign represents subtraction, so this program reassigns
field three, $3, as the original value of field three minus ten:
`$3 - 10'. (See Arithmetic Ops.)
Then it prints the original and new values for field three.
(Someone in the warehouse made a consistent mistake while inventorying
the red boxes.)
For this to work, the text in field $3 must make sense
as a number; the string of characters must be converted to a number
for the computer to do arithmetic on it. The number resulting
from the subtraction is converted back to a string of characters that
then becomes field three.
See Conversion.
When the value of a field is changed (as perceived by awk), the
text of the input record is recalculated to contain the new field where
the old one was. In other words, $0 changes to reflect the altered
field. Thus, this program
prints a copy of the input file, with 10 subtracted from the second
field of each line:
$ awk '{ $2 = $2 - 10; print $0 }' inventory-shipped
-| Jan 3 25 15 115
-| Feb 5 32 24 226
-| Mar 5 24 34 228
...
It is also possible to also assign contents to fields that are out of range. For example:
$ awk '{ $6 = ($5 + $4 + $3 + $2)
> print $6 }' inventory-shipped
-| 168
-| 297
-| 301
...
We've just created $6, whose value is the sum of fields
$2, $3, $4, and $5. The `+' sign
represents addition. For the file inventory-shipped, $6
represents the total number of parcels shipped for a particular month.
Creating a new field changes awk's internal copy of the current
input record, which is the value of $0. Thus, if you do `print $0'
after adding a field, the record printed includes the new field, with
the appropriate number of field separators between it and the previously
existing fields.
This recomputation affects and is affected by
NF (the number of fields; see Fields).
For example, the value of NF is set to the number of the highest
field you create.
The exact format of $0 is also affected by a feature that has not been discussed yet:
the output field separator, OFS,
used to separate the fields (see Output Separators).
Note, however, that merely referencing an out-of-range field
does not change the value of either $0 or NF.
Referencing an out-of-range field only produces an empty string. For
example:
if ($(NF+1) != "")
print "can't happen"
else
print "everything is normal"
should print `everything is normal', because NF+1 is certain
to be out of range. (See If Statement,
for more information about awk's if-else statements.
See Typing and Comparison,
for more information about the `!=' operator.)
It is important to note that making an assignment to an existing field
changes the
value of $0 but does not change the value of NF,
even when you assign the empty string to a field. For example:
$ echo a b c d | awk '{ OFS = ":"; $2 = ""
> print $0; print NF }'
-| a::c:d
-| 4
The field is still there; it just has an empty value, denoted by the two colons between `a' and `c'. This example shows what happens if you create a new field:
$ echo a b c d | awk '{ OFS = ":"; $2 = ""; $6 = "new"
> print $0; print NF }'
-| a::c:d::new
-| 6
The intervening field, $5, is created with an empty value
(indicated by the second pair of adjacent colons),
and NF is updated with the value six.
Decrementing NF throws away the values of the fields
after the new value of NF and recomputes $0.
(d.c.)
Here is an example:
$ echo a b c d e f | awk '{ print "NF =", NF;
> NF = 3; print $0 }'
-| NF = 6
-| a b c
Caution: Some versions of awk don't
rebuild $0 when NF is decremented. Caveat emptor.
Finally, there are times when it is convenient to force
awk to rebuild the entire record, using the current
value of the fields and OFS. To do this, use the
seemingly innocuous assignment:
$1 = $1 # force record to be reconstituted
print $0 # or whatever else with $0
This forces awk rebuild the record. It does help to add a comment, as we've shown here.
There is a flip side to the relationship between $0 and
the fields. Any assignment to $0 causes the record to be
reparsed into fields using the current value of FS.
This also applies to any built-in function that updates $0,
such as sub and gsub
(see String Functions).
The field separator, which is either a single character or a regular expression, controls the way awk splits an input record into fields. awk scans the input record for character sequences that match the separator; the fields themselves are the text between the matches.
In the examples that follow, we use the bullet symbol (•) to represent spaces in the output. If the field separator is `oo', then the following line:
moo goo gai pan
is split into three fields: `m', `•g', and `•gai•pan'. Note the leading spaces in the values of the second and third fields.
The field separator is represented by the built-in variable FS.
Shell programmers take note: awk does not use the
name IFS that is used by the POSIX-compliant shells (such as
the Unix Bourne shell, sh, or bash).
The value of FS can be changed in the awk program with the
assignment operator, `=' (see Assignment Ops).
Often the right time to do this is at the beginning of execution
before any input has been processed, so that the very first record
is read with the proper separator. To do this, use the special
BEGIN pattern
(see BEGIN/END).
For example, here we set the value of FS to the string
",":
awk 'BEGIN { FS = "," } ; { print $2 }'
John Q. Smith, 29 Oak St., Walamazoo, MI 42139
this awk program extracts and prints the string `•29•Oak•St.'.
Sometimes the input data contains separator characters that don't separate fields the way you thought they would. For instance, the person's name in the example we just used might have a title or suffix attached, such as:
John Q. Smith, LXIX, 29 Oak St., Walamazoo, MI 42139
The same program would extract `•LXIX', instead of `•29•Oak•St.'. If you were expecting the program to print the address, you would be surprised. The moral is to choose your data layout and separator characters carefully to prevent such problems. (If the data is not in a form that is easy to process, perhaps you can massage it first with a separate awk program.)
Fields are normally separated by whitespace sequences
(spaces, tabs, and newlines), not by single spaces. Two spaces in a row do not
delimit an empty field. The default value of the field separator FS
is a string containing a single space, " ". If awk
interpreted this value in the usual way, each space character would separate
fields, so two spaces in a row would make an empty field between them.
The reason this does not happen is that a single space as the value of
FS is a special case—it is taken to specify the default manner
of delimiting fields.
If FS is any other single character, such as ",", then
each occurrence of that character separates two fields. Two consecutive
occurrences delimit an empty field. If the character occurs at the
beginning or the end of the line, that too delimits an empty field. The
space character is the only single character that does not follow these
rules.
The previous subsection
discussed the use of single characters or simple strings as the
value of FS.
More generally, the value of FS may be a string containing any
regular expression. In this case, each match in the record for the regular
expression separates fields. For example, the assignment:
FS = ", \t"
makes every area of an input line that consists of a comma followed by a space and a TAB into a field separator.
For a less trivial example of a regular expression, try using
single spaces to separate fields the way single commas are used.
FS can be set to "[ ]" (left bracket, space, right
bracket). This regular expression matches a single space and nothing else
(see Regexp).
There is an important difference between the two cases of `FS = " "'
(a single space) and `FS = "[ \t\n]+"'
(a regular expression matching one or more spaces, tabs, or newlines).
For both values of FS, fields are separated by runs
(multiple adjacent occurrences) of spaces, tabs,
and/or newlines. However, when the value of FS is " ",
awk first strips leading and trailing whitespace from
the record and then decides where the fields are.
For example, the following pipeline prints `b':
$ echo ' a b c d ' | awk '{ print $2 }'
-| b
However, this pipeline prints `a' (note the extra spaces around each letter):
$ echo ' a b c d ' | awk 'BEGIN { FS = "[ \t\n]+" }
> { print $2 }'
-| a
In this case, the first field is null or empty.
The stripping of leading and trailing whitespace also comes into
play whenever $0 is recomputed. For instance, study this pipeline:
$ echo ' a b c d' | awk '{ print; $2 = $2; print }'
-| a b c d
-| a b c d
The first print statement prints the record as it was read,
with leading whitespace intact. The assignment to $2 rebuilds
$0 by concatenating $1 through $NF together,
separated by the value of OFS. Because the leading whitespace
was ignored when finding $1, it is not part of the new $0.
Finally, the last print statement prints the new $0.
There are times when you may want to examine each character
of a record separately. This can be done in gawk by
simply assigning the null string ("") to FS. In this case,
each individual character in the record becomes a separate field.
For example:
$ echo a b | gawk 'BEGIN { FS = "" }
> {
> for (i = 1; i <= NF; i = i + 1)
> print "Field", i, "is", $i
> }'
-| Field 1 is a
-| Field 2 is
-| Field 3 is b
Traditionally, the behavior of FS equal to "" was not defined.
In this case, most versions of Unix awk simply treat the entire record
as only having one field.
(d.c.)
In compatibility mode
(see Options),
if FS is the null string, then gawk also
behaves this way.
FS from the Command Line
FS can be set on the command line. Use the -F option to
do so. For example:
awk -F, 'program' input-files
sets FS to the `,' character. Notice that the option uses
an uppercase `F' instead of a lowercase `f'. The latter
option (-f) specifies a file
containing an awk program. Case is significant in command-line
options:
the -F and -f options have nothing to do with each other.
You can use both options at the same time to set the FS variable
and get an awk program from a file.
The value used for the argument to -F is processed in exactly the
same way as assignments to the built-in variable FS.
Any special characters in the field separator must be escaped
appropriately. For example, to use a `\' as the field separator
on the command line, you would have to type:
# same as FS = "\\"
awk -F\\\\ '...' files ...
Because `\' is used for quoting in the shell, awk sees `-F\\'. Then awk processes the `\\' for escape characters (see Escape Sequences), finally yielding a single `\' to use for the field separator.
As a special case, in compatibility mode
(see Options),
if the argument to -F is `t', then FS is set to
the TAB character. If you type `-F\t' at the
shell, without any quotes, the `\' gets deleted, so awk
figures that you really want your fields to be separated with tabs and
not `t's. Use `-v FS="t"' or `-F"[t]"' on the command line
if you really do want to separate your fields with `t's.
For example, let's use an awk program file called baud.awk
that contains the pattern /300/ and the action `print $1':
/300/ { print $1 }
Let's also set FS to be the `-' character and run the
program on the file BBS-list. The following command prints a
list of the names of the bulletin boards that operate at 300 baud and
the first three digits of their phone numbers:
$ awk -F- -f baud.awk BBS-list
-| aardvark 555
-| alpo
-| barfly 555
-| bites 555
-| camelot 555
-| core 555
-| fooey 555
-| foot 555
-| macfoo 555
-| sdace 555
-| sabafoo 555
Note the second line of output. The second line in the original file looked like this:
alpo-net 555-3412 2400/1200/300 A
The `-' as part of the system's name was used as the field separator, instead of the `-' in the phone number that was originally intended. This demonstrates why you have to be careful in choosing your field and record separators.
Perhaps the most common use of a single character as the field separator occurs when processing the Unix system password file. On many Unix systems, each user has a separate entry in the system password file, one line per user. The information in these lines is separated by colons. The first field is the user's logon name and the second is the user's (encrypted or shadow) password. A password file entry might look like this:
arnold:xyzzy:2076:10:Arnold Robbins:/home/arnold:/bin/bash
The following program searches the system password file and prints the entries for users who have no password:
awk -F: '$2 == ""' /etc/passwd
It is important to remember that when you assign a string constant
as the value of FS, it undergoes normal awk string
processing. For example, with Unix awk and gawk,
the assignment `FS = "\.."' assigns the character string ".."
to FS (the backslash is stripped). This creates a regexp meaning
“fields are separated by occurrences of any two characters.”
If instead you want fields to be separated by a literal period followed
by any single character, use `FS = "\\.."'.
The following table summarizes how fields are split, based on the value
of FS (`==' means “is equal to”):
FS == " "FS == any other single characterFS == regexpFS == ""FS Does Not Affect the FieldsAccording to the POSIX standard, awk is supposed to behave
as if each record is split into fields at the time it is read.
In particular, this means that if you change the value of FS
after a record is read, the value of the fields (i.e., how they were split)
should reflect the old value of FS, not the new one.
However, many implementations of awk do not work this way. Instead,
they defer splitting the fields until a field is actually
referenced. The fields are split
using the current value of FS!
(d.c.)
This behavior can be difficult
to diagnose. The following example illustrates the difference
between the two methods.
(The sed18
command prints just the first line of /etc/passwd.)
sed 1q /etc/passwd | awk '{ FS = ":" ; print $1 }'
which usually prints:
root
on an incorrect implementation of awk, while gawk prints something like:
root:nSijPlPhZZwgE:0:0:Root:/:
FS and IGNORECASEThe IGNORECASE variable
(see User-modified)
affects field splitting only when the value of FS is a regexp.
It has no effect when FS is a single character, even if
that character is a letter. Thus, in the following code:
FS = "c"
IGNORECASE = 1
$0 = "aCa"
print $1
The output is `aCa'. If you really want to split fields on an
alphabetic character while ignoring case, use a regexp that will
do it for you. E.g., `FS = "[c]"'. In this case, IGNORECASE
will take effect.
NOTE: This section discusses an advanced feature of gawk. If you are a novice awk user, you might want to skip it on the first reading.
gawk version 2.13 introduced a facility for dealing with fixed-width fields with no distinctive field separator. For example, data of this nature arises in the input for old Fortran programs where numbers are run together, or in the output of programs that did not anticipate the use of their output as input for other programs.
An example of the latter is a table where all the columns are lined up by
the use of a variable number of spaces and empty fields are just
spaces. Clearly, awk's normal field splitting based on FS
does not work well in this case. Although a portable awk program
can use a series of substr calls on $0
(see String Functions),
this is awkward and inefficient for a large number of fields.
The splitting of an input record into fixed-width fields is specified by
assigning a string containing space-separated numbers to the built-in
variable FIELDWIDTHS. Each number specifies the width of the field,
including columns between fields. If you want to ignore the columns
between fields, you can specify the width as a separate field that is
subsequently ignored.
It is a fatal error to supply a field width that is not a positive number.
The following data is the output of the Unix w utility. It is useful
to illustrate the use of FIELDWIDTHS:
10:06pm up 21 days, 14:04, 23 users
User tty login idle JCPU PCPU what
hzuo ttyV0 8:58pm 9 5 vi p24.tex
hzang ttyV3 6:37pm 50 -csh
eklye ttyV5 9:53pm 7 1 em thes.tex
dportein ttyV6 8:17pm 1:47 -csh
gierd ttyD3 10:00pm 1 elm
dave ttyD4 9:47pm 4 4 w
brent ttyp0 26Jun91 4:46 26:46 4:41 bash
dave ttyq4 26Jun9115days 46 46 wnewmail
The following program takes the above input, converts the idle time to number of seconds, and prints out the first two fields and the calculated idle time:
NOTE: This program uses a number of awk features that haven't been introduced yet.
BEGIN { FIELDWIDTHS = "9 6 10 6 7 7 35" }
NR > 2 {
idle = $4
sub(/^ */, "", idle) # strip leading spaces
if (idle == "")
idle = 0
if (idle ~ /:/) {
split(idle, t, ":")
idle = t[1] * 60 + t[2]
}
if (idle ~ /days/)
idle *= 24 * 60 * 60
print $1, $2, idle
}
Running the program on the data produces the following results:
hzuo ttyV0 0
hzang ttyV3 50
eklye ttyV5 0
dportein ttyV6 107
gierd ttyD3 1
dave ttyD4 0
brent ttyp0 286
dave ttyq4 1296000
Another (possibly more practical) example of fixed-width input data
is the input from a deck of balloting cards. In some parts of
the United States, voters mark their choices by punching holes in computer
cards. These cards are then processed to count the votes for any particular
candidate or on any particular issue. Because a voter may choose not to
vote on some issue, any column on the card may be empty. An awk
program for processing such data could use the FIELDWIDTHS feature
to simplify reading the data. (Of course, getting gawk to run on
a system with card readers is another story!)
Assigning a value to FS causes gawk to use
FS for field splitting again. Use `FS = FS' to make this happen,
without having to know the current value of FS.
In order to tell which kind of field splitting is in effect,
use PROCINFO["FS"]
(see Auto-set).
The value is "FS" if regular field splitting is being used,
or it is "FIELDWIDTHS" if fixed-width field splitting is being used:
if (PROCINFO["FS"] == "FS")
regular field splitting ...
else
fixed-width field splitting ...
This information is useful when writing a function
that needs to temporarily change FS or FIELDWIDTHS,
read some records, and then restore the original settings
(see Passwd Functions,
for an example of such a function).
In some databases, a single line cannot conveniently hold all the information in one entry. In such cases, you can use multiline records. The first step in doing this is to choose your data format.
One technique is to use an unusual character or string to separate
records. For example, you could use the formfeed character (written
`\f' in awk, as in C) to separate them, making each record
a page of the file. To do this, just set the variable RS to
"\f" (a string containing the formfeed character). Any
other character could equally well be used, as long as it won't be part
of the data in a record.
Another technique is to have blank lines separate records. By a special
dispensation, an empty string as the value of RS indicates that
records are separated by one or more blank lines. When RS is set
to the empty string, each record always ends at the first blank line
encountered. The next record doesn't start until the first nonblank
line that follows. No matter how many blank lines appear in a row, they
all act as one record separator.
(Blank lines must be completely empty; lines that contain only
whitespace do not count.)
You can achieve the same effect as `RS = ""' by assigning the
string "\n\n+" to RS. This regexp matches the newline
at the end of the record and one or more blank lines after the record.
In addition, a regular expression always matches the longest possible
sequence when there is a choice
(see Leftmost Longest).
So the next record doesn't start until
the first nonblank line that follows—no matter how many blank lines
appear in a row, they are considered one record separator.
There is an important difference between `RS = ""' and `RS = "\n\n+"'. In the first case, leading newlines in the input data file are ignored, and if a file ends without extra blank lines after the last record, the final newline is removed from the record. In the second case, this special processing is not done. (d.c.)
Now that the input is separated into records, the second step is to
separate the fields in the record. One way to do this is to divide each
of the lines into fields in the normal manner. This happens by default
as the result of a special feature. When RS is set to the empty
string, and FS is a set to a single character,
the newline character always acts as a field separator.
This is in addition to whatever field separations result from
FS.19
The original motivation for this special exception was probably to provide
useful behavior in the default case (i.e., FS is equal
to " "). This feature can be a problem if you really don't
want the newline character to separate fields, because there is no way to
prevent it. However, you can work around this by using the split
function to break up the record manually
(see String Functions).
If you have a single character field separator, you can work around
the special feature in a different way, by making FS into a
regexp for that single character. For example, if the field
separator is a percent character, instead of
`FS = "%"', use `FS = "[%]"'.
Another way to separate fields is to
put each field on a separate line: to do this, just set the
variable FS to the string "\n". (This single
character seperator matches a single newline.)
A practical example of a data file organized this way might be a mailing
list, where each entry is separated by blank lines. Consider a mailing
list in a file named addresses, which looks like this:
Jane Doe
123 Main Street
Anywhere, SE 12345-6789
John Smith
456 Tree-lined Avenue
Smallville, MW 98765-4321
...
A simple program to process this file is as follows:
# addrs.awk --- simple mailing list program
# Records are separated by blank lines.
# Each line is one field.
BEGIN { RS = "" ; FS = "\n" }
{
print "Name is:", $1
print "Address is:", $2
print "City and State are:", $3
print ""
}
Running the program produces the following output:
$ awk -f addrs.awk addresses
-| Name is: Jane Doe
-| Address is: 123 Main Street
-| City and State are: Anywhere, SE 12345-6789
-|
-| Name is: John Smith
-| Address is: 456 Tree-lined Avenue
-| City and State are: Smallville, MW 98765-4321
-|
...
See Labels Program, for a more realistic
program that deals with address lists.
The following
table
summarizes how records are split, based on the
value of
RS:
RS == "\n"RS == any single characterRS == ""FS may have. Leading and trailing newlines in a file are ignored.
RS == regexpIn all cases, gawk sets RT to the input text that matched the
value specified by RS.
getlineSo far we have been getting our input data from awk's main
input stream—either the standard input (usually your terminal, sometimes
the output from another program) or from the
files specified on the command line. The awk language has a
special built-in command called getline that
can be used to read input under your explicit control.
The getline command is used in several different ways and should
not be used by beginners.
The examples that follow the explanation of the getline command
include material that has not been covered yet. Therefore, come back
and study the getline command after you have reviewed the
rest of this Web page and have a good knowledge of how awk works.
The getline command returns one if it finds a record and zero if
it encounters the end of the file. If there is some error in getting
a record, such as a file that cannot be opened, then getline
returns −1. In this case, gawk sets the variable
ERRNO to a string describing the error that occurred.
In the following examples, command stands for a string value that represents a shell command.
getline with No ArgumentsThe getline command can be used without arguments to read input
from the current input file. All it does in this case is read the next
input record and split it up into fields. This is useful if you've
finished processing the current record, but want to do some special
processing on the next record right now. For example:
{
if ((t = index($0, "/*")) != 0) {
# value of `tmp' will be "" if t is 1
tmp = substr($0, 1, t - 1)
u = index(substr($0, t + 2), "*/")
while (u == 0) {
if (getline <= 0) {
m = "unexpected EOF or error"
m = (m ": " ERRNO)
print m > "/dev/stderr"
exit
}
t = -1
u = index($0, "*/")
}
# substr expression will be "" if */
# occurred at end of line
$0 = tmp substr($0, u + 2)
}
print $0
}
This awk program deletes all C-style comments (`/* ... */') from the input. By replacing the `print $0' with other statements, you could perform more complicated processing on the decommented input, such as searching for matches of a regular expression. (This program has a subtle problem—it does not work if one comment ends and another begins on the same line.)
This form of the getline command sets NF,
NR, FNR, and the value of $0.
NOTE: The new value of$0is used to test the patterns of any subsequent rules. The original value of$0that triggered the rule that executedgetlineis lost. By contrast, thenextstatement reads a new record but immediately begins processing it normally, starting with the first rule in the program. See Next Statement.
getline into a Variable
You can use `getline var' to read the next record from
awk's input into the variable var. No other processing is
done.
For example, suppose the next line is a comment or a special string,
and you want to read it without triggering
any rules. This form of getline allows you to read that line
and store it in a variable so that the main
read-a-line-and-check-each-rule loop of awk never sees it.
The following example swaps every two lines of input:
{
if ((getline tmp) > 0) {
print tmp
print $0
} else
print $0
}
It takes the following list:
wan
tew
free
phore
and produces these results:
tew
wan
phore
free
The getline command used in this way sets only the variables
NR and FNR (and of course, var). The record is not
split into fields, so the values of the fields (including $0) and
the value of NF do not change.
getline from a FileUse `getline < file' to read the next record from file. Here file is a string-valued expression that specifies the file name. `< file' is called a redirection because it directs input to come from a different place. For example, the following program reads its input record from the file secondary.input when it encounters a first field with a value equal to 10 in the current input file:
{
if ($1 == 10) {
getline < "secondary.input"
print
} else
print
}
Because the main input stream is not used, the values of NR and
FNR are not changed. However, the record it reads is split into fields in
the normal manner, so the values of $0 and the other fields are
changed, resulting in a new value of NF.
According to POSIX, `getline < expression' is ambiguous if expression contains unparenthesized operators other than `$'; for example, `getline < dir "/" file' is ambiguous because the concatenation operator is not parenthesized. You should write it as `getline < (dir "/" file)' if you want your program to be portable to other awk implementations.
getline into a Variable from a FileUse `getline var < file' to read input from the file file, and put it in the variable var. As above, file is a string-valued expression that specifies the file from which to read.
In this version of getline, none of the built-in variables are
changed and the record is not split into fields. The only variable
changed is var.
For example, the following program copies all the input files to the
output, except for records that say `@include filename'.
Such a record is replaced by the contents of the file
filename:
{
if (NF == 2 && $1 == "@include") {
while ((getline line < $2) > 0)
print line
close($2)
} else
print
}
Note here how the name of the extra input file is not built into the program; it is taken directly from the data, specifically from the second field on the `@include' line.
The close function is called to ensure that if two identical
`@include' lines appear in the input, the entire specified file is
included twice.
See Close Files And Pipes.
One deficiency of this program is that it does not process nested `@include' statements (i.e., `@include' statements in included files) the way a true macro preprocessor would. See Igawk Program, for a program that does handle nested `@include' statements.
getline from a PipeThe output of a command can also be piped into getline, using
`command | getline'. In
this case, the string command is run as a shell command and its output
is piped into awk to be used as input. This form of getline
reads one record at a time from the pipe.
For example, the following program copies its input to its output, except for
lines that begin with `@execute', which are replaced by the output
produced by running the rest of the line as a shell command:
{
if ($1 == "@execute") {
tmp = substr($0, 10)
while ((tmp | getline) > 0)
print
close(tmp)
} else
print
}
The close function is called to ensure that if two identical
`@execute' lines appear in the input, the command is run for
each one.
See Close Files And Pipes.
Given the input:
foo
bar
baz
@execute who
bletch
the program might produce:
foo
bar
baz
arnold ttyv0 Jul 13 14:22
miriam ttyp0 Jul 13 14:23 (murphy:0)
bill ttyp1 Jul 13 14:23 (murphy:0)
bletch
Notice that this program ran the command who and printed the previous result. (If you try this program yourself, you will of course get different results, depending upon who is logged in on your system.)
This variation of getline splits the record into fields, sets the
value of NF, and recomputes the value of $0. The values of
NR and FNR are not changed.
According to POSIX, `expression | getline' is ambiguous if expression contains unparenthesized operators other than `$'—for example, `"echo " "date" | getline' is ambiguous because the concatenation operator is not parenthesized. You should write it as `("echo " "date") | getline' if you want your program to be portable to other awk implementations.
getline into a Variable from a Pipe
When you use `command | getline var', the
output of command is sent through a pipe to
getline and into the variable var. For example, the
following program reads the current date and time into the variable
current_time, using the date utility, and then
prints it:
BEGIN {
"date" | getline current_time
close("date")
print "Report printed on " current_time
}
In this version of getline, none of the built-in variables are
changed and the record is not split into fields.
getline from a Coprocess
Input into getline from a pipe is a one-way operation.
The command that is started with `command | getline' only
sends data to your awk program.
On occasion, you might want to send data to another program for processing and then read the results back. gawk allows you start a coprocess, with which two-way communications are possible. This is done with the `|&' operator. Typically, you write data to the coprocess first and then read results back, as shown in the following:
print "some query" |& "db_server"
"db_server" |& getline
which sends a query to db_server and then reads the results.
The values of NR and
FNR are not changed,
because the main input stream is not used.
However, the record is split into fields in
the normal manner, thus changing the values of $0, of the other fields,
and of NF.
Coprocesses are an advanced feature. They are discussed here only because
this is the section on getline.
See Two-way I/O,
where coprocesses are discussed in more detail.
getline into a Variable from a Coprocess
When you use `command |& getline var', the output from
the coprocess command is sent through a two-way pipe to getline
and into the variable var.
In this version of getline, none of the built-in variables are
changed and the record is not split into fields. The only variable
changed is var.
getlineHere are some miscellaneous points about getline that
you should bear in mind:
getline changes the value of $0 and NF,
awk does not automatically jump to the start of the
program and start testing the new record against every pattern.
However, the new record is tested against any subsequent rules.
getline without a
redirection inside a BEGIN rule. Because an unredirected getline
reads from the command-line data files, the first getline command
causes awk to set the value of FILENAME. Normally,
FILENAME does not have a value inside BEGIN rules, because you
have not yet started to process the command-line data files.
(d.c.)
(See BEGIN/END,
also see Auto-set.)
FILENAME with getline
(`getline < FILENAME')
is likely to be a source for
confusion. awk opens a separate input stream from the
current input file. However, by not using a variable, $0
and NR are still updated. If you're doing this, it's
probably by accident, and you should reconsider what it is you're
trying to accomplish.
getline Variants
table-getline-variants
summarizes the eight variants of getline,
listing which built-in variables are set by each one.
| Variant | Effect
|
|---|---|
getline | Sets $0, NF, FNR, and NR
|
getline var | Sets var, FNR, and NR
|
getline < file | Sets $0 and NF
|
getline var < file | Sets var
|
command | getline | Sets $0 and NF
|
command | getline var | Sets var
|
command |& getline | Sets $0 and NF. This is a gawk extension
|
command |& getline var | Sets var. This is a gawk extension
|
Table 3.1: getline Variants and What They Set
One of the most common programming actions is to print, or output,
some or all of the input. Use the print statement
for simple output, and the printf statement
for fancier formatting.
The print statement is not limited when
computing which values to print. However, with two exceptions,
you cannot specify how to print them—how many
columns, whether to use exponential notation or not, and so on.
(For the exceptions, see Output Separators, and
OFMT.)
For printing with specifications, you need the printf statement
(see Printf).
Besides basic and formatted printing, this chapter
also covers I/O redirections to files and pipes, introduces
the special file names that gawk processes internally,
and discusses the close built-in function.
print StatementThe print statement is used to produce output with simple, standardized
formatting. Specify only the strings or numbers to print, in a
list separated by commas. They are output, separated by single spaces,
followed by a newline. The statement looks like this:
print item1, item2, ...
The entire list of items may be optionally enclosed in parentheses. The parentheses are necessary if any of the item expressions uses the `>' relational operator; otherwise it could be confused with a redirection (see Redirection).
The items to print can be constant strings or numbers, fields of the
current record (such as $1), variables, or any awk
expression. Numeric values are converted to strings and then printed.
The simple statement `print' with no items is equivalent to
`print $0': it prints the entire current record. To print a blank
line, use `print ""', where "" is the empty string.
To print a fixed piece of text, use a string constant, such as
"Don't Panic", as one item. If you forget to use the
double-quote characters, your text is taken as an awk
expression, and you will probably get an error. Keep in mind that a
space is printed between any two items.
print StatementsEach print statement makes at least one line of output. However, it
isn't limited to only one line. If an item value is a string that contains a
newline, the newline is output along with the rest of the string. A
single print statement can make any number of lines this way.
The following is an example of printing a string that contains embedded newlines (the `\n' is an escape sequence, used to represent the newline character; see Escape Sequences):
$ awk 'BEGIN { print "line one\nline two\nline three" }'
-| line one
-| line two
-| line three
The next example, which is run on the inventory-shipped file, prints the first two fields of each input record, with a space between them:
$ awk '{ print $1, $2 }' inventory-shipped
-| Jan 13
-| Feb 15
-| Mar 15
...
A common mistake in using the print statement is to omit the comma
between two items. This often has the effect of making the items run
together in the output, with no space. The reason for this is that
juxtaposing two string expressions in awk means to concatenate
them. Here is the same program, without the comma:
$ awk '{ print $1 $2 }' inventory-shipped
-| Jan13
-| Feb15
-| Mar15
...
To someone unfamiliar with the inventory-shipped file, neither
example's output makes much sense. A heading line at the beginning
would make it clearer. Let's add some headings to our table of months
($1) and green crates shipped ($2). We do this using the
BEGIN pattern
(see BEGIN/END)
so that the headings are only printed once:
awk 'BEGIN { print "Month Crates"
print "----- ------" }
{ print $1, $2 }' inventory-shipped
When run, the program prints the following:
Month Crates
----- ------
Jan 13
Feb 15
Mar 15
...
The only problem, however, is that the headings and the table data don't line up! We can fix this by printing some spaces between the two fields:
awk 'BEGIN { print "Month Crates"
print "----- ------" }
{ print $1, " ", $2 }' inventory-shipped
Lining up columns this way can get pretty
complicated when there are many columns to fix. Counting spaces for two
or three columns is simple, but any more than this can take up
a lot of time. This is why the printf statement was
created (see Printf);
one of its specialties is lining up columns of data.
NOTE: You can continue either aprintfstatement simply by putting a newline after any comma (see Statements/Lines).
As mentioned previously, a print statement contains a list
of items separated by commas. In the output, the items are normally
separated by single spaces. However, this doesn't need to be the case;
a single space is only the default. Any string of
characters may be used as the output field separator by setting the
built-in variable OFS. The initial value of this variable
is the string " "—that is, a single space.
The output from an entire print statement is called an
output record. Each print statement outputs one output
record, and then outputs a string called the output record separator
(or ORS). The initial
value of ORS is the string "\n"; i.e., a newline
character. Thus, each print statement normally makes a separate line.
In order to change how output fields and records are separated, assign
new values to the variables OFS and ORS. The usual
place to do this is in the BEGIN rule
(see BEGIN/END), so
that it happens before any input is processed. It can also be done
with assignments on the command line, before the names of the input
files, or using the -v command-line option
(see Options).
The following example prints the first and second fields of each input
record, separated by a semicolon, with a blank line added after each
newline:
$ awk 'BEGIN { OFS = ";"; ORS = "\n\n" }
> { print $1, $2 }' BBS-list
-| aardvark;555-5553
-|
-| alpo-net;555-3412
-|
-| barfly;555-7685
...
If the value of ORS does not contain a newline, the program's output
is run together on a single line.
printWhen the print statement is used to print numeric values,
awk internally converts the number to a string of characters
and prints that string. awk uses the sprintf function
to do this conversion
(see String Functions).
For now, it suffices to say that the sprintf
function accepts a format specification that tells it how to format
numbers (or strings), and that there are a number of different ways in which
numbers can be formatted. The different format specifications are discussed
more fully in
Control Letters.
The built-in variable OFMT contains the default format specification
that print uses with sprintf when it wants to convert a
number to a string for printing.
The default value of OFMT is "%.6g".
The way print prints numbers can be changed
by supplying different format specifications
as the value of OFMT, as shown in the following example:
$ awk 'BEGIN {
> OFMT = "%.0f" # print numbers as integers (rounds)
> print 17.23, 17.54 }'
-| 17 18
According to the POSIX standard, awk's behavior is undefined
if OFMT contains anything but a floating-point conversion specification.
(d.c.)
printf Statements for Fancier PrintingFor more precise control over the output format than what is
normally provided by print, use printf.
printf can be used to
specify the width to use for each item, as well as various
formatting choices for numbers (such as what output base to use, whether to
print an exponent, whether to print a sign, and how many digits to print
after the decimal point). This is done by supplying a string, called
the format string, that controls how and where to print the other
arguments.
printf StatementA simple printf statement looks like this:
printf format, item1, item2, ...
The entire list of arguments may optionally be enclosed in parentheses. The parentheses are necessary if any of the item expressions use the `>' relational operator; otherwise, it can be confused with a redirection (see Redirection).
The difference between printf and print is the format
argument. This is an expression whose value is taken as a string; it
specifies how to output each of the other arguments. It is called the
format string.
The format string is very similar to that in the ISO C library function
printf. Most of format is text to output verbatim.
Scattered among this text are format specifiers—one per item.
Each format specifier says to output the next item in the argument list
at that place in the format.
The printf statement does not automatically append a newline
to its output. It outputs only what the format string specifies.
So if a newline is needed, you must include one in the format string.
The output separator variables OFS and ORS have no effect
on printf statements. For example:
$ awk 'BEGIN {
> ORS = "\nOUCH!\n"; OFS = "+"
> msg = "Dont Panic!"
> printf "%s\n", msg
> }'
-| Dont Panic!
Here, neither the `+' nor the `OUCH' appear when the message is printed.
A format specifier starts with the character `%' and ends with
a format-control letter—it tells the printf statement
how to output one item. The format-control letter specifies what kind
of value to print. The rest of the format specifier is made up of
optional modifiers that control how to print the value, such as
the field width. Here is a list of the format-control letters:
%c%d, %i%e, %E printf "%4.3e\n", 1950
prints `1.950e+03', with a total of four significant figures, three of
which follow the decimal point.
(The `4.3' represents two modifiers,
discussed in the next subsection.)
`%E' uses `E' instead of `e' in the output.
%f printf "%4.3f", 1950
prints `1950.000', with a total of four significant figures, three of which follow the decimal point. (The `4.3' represents two modifiers, discussed in the next subsection.)
On systems supporting IEEE 754 floating point format, values
representing negative
infinity are formatted as
`-inf' or `-infinity',
and positive infinity as
`inf' and `-infinity'.
The special “not a number” value formats as `-nan' or `nan'.
%F%f but the infinity and “not a number” values are spelled
using uppercase letters.
The %F format is a POSIX extension to ISO C; not all systems
support. On those that don't, gawk uses %f instead.
%g, %G%o%s%u%x, %X%%NOTE: When using the integer format-control letters for values that are outside the range of the widest C integer type, gawk switches to the the `%g' format specifier. If --lint is provided on the command line (see Options), gawk warns about this. Other versions of awk may print invalid values or do something else entirely. (d.c.)
printf FormatsA format specification can also include modifiers that can control how much of the item's value is printed, as well as how much space it gets. The modifiers come between the `%' and the format-control letter. We will use the bullet symbol “•” in the following examples to represent spaces in the output. Here are the possible modifiers, in the order in which they may appear:
$ printf "%s %s\n", "don't", "panic"
printf "%2$s %1$s\n", "panic", "don't"
prints the famous friendly message twice.
At first glance, this feature doesn't seem to be of much use.
It is in fact a gawk extension, intended for use in translating
messages at runtime.
See Printf Ordering,
which describes how and why to use positional specifiers.
For now, we will not use them.
- printf "%-4s", "foo"
prints `foo•'.
+#0' $ cat thousands.awk Show source program
-| BEGIN { printf "%'d\n", 1234567 }
$ LC_ALL=C gawk -f thousands.awk Run it in "C" locale
-| 1234567
$ LC_ALL=en_US.UTF-8 gawk -f thousands.awk Run in US English UTF locale
-| 1,234,567
For more information about locales and internationalization issues, FIXME: see xxxx.
NOTE: The `'' flag is a nice feature, but its use complicates things: it now becomes difficult to use it in command-line programs. For information on appropriate quoting tricks, FIXME: see XXXX.
printf "%4s", "foo"
prints `•foo'.
The value of width is a minimum width, not a maximum. If the item value requires more than width characters, it can be as wide as necessary. Thus, the following:
printf "%4s", "foobar"
prints `foobar'.
Preceding the width with a minus sign causes the output to be
padded with spaces on the right, instead of on the left.
.prec%e, %E, %f%g, %G%d, %i, %o, %u, %x, %X%sThus, the following:
printf "%.4s", "foobar"
prints `foob'.
The C library printf's dynamic width and prec
capability (for example, "%*.*s") is supported. Instead of
supplying explicit width and/or prec values in the format
string, they are passed in the argument list. For example:
w = 5
p = 3
s = "abcdefg"
printf "%*.*s\n", w, p, s
is exactly equivalent to:
s = "abcdefg"
printf "%5.3s\n", s
Both programs output `••abc'. Earlier versions of awk did not support this capability. If you must use such a version, you may simulate this feature by using concatenation to build up the format string, like so:
w = 5
p = 3
s = "abcdefg"
printf "%" w "." p "s\n", s
This is not particularly easy to read but it does work.
C programmers may be used to supplying additional
`l', `L', and `h'
modifiers in printf format strings. These are not valid in awk.
Most awk implementations silently ignore these modifiers.
If --lint is provided on the command line
(see Options),
gawk warns about their use. If --posix is supplied,
their use is a fatal error.
printfThe following is a simple example of
how to use printf to make an aligned table:
awk '{ printf "%-10s %s\n", $1, $2 }' BBS-list
This command
prints the names of the bulletin boards ($1) in the file
BBS-list as a string of 10 characters that are left-justified. It also
prints the phone numbers ($2) next on the line. This
produces an aligned two-column table of names and phone numbers,
as shown here:
$ awk '{ printf "%-10s %s\n", $1, $2 }' BBS-list
-| aardvark 555-5553
-| alpo-net 555-3412
-| barfly 555-7685
-| bites 555-1675
-| camelot 555-0542
-| core 555-2912
-| fooey 555-1234
-| foot 555-6699
-| macfoo 555-6480
-| sdace 555-3430
-| sabafoo 555-2127
In this case, the phone numbers had to be printed as strings because the numbers are separated by a dash. Printing the phone numbers as numbers would have produced just the first three digits: `555'. This would have been pretty confusing.
It wasn't necessary to specify a width for the phone numbers because they are last on their lines. They don't need to have spaces after them.
The table could be made to look even nicer by adding headings to the
tops of the columns. This is done using the BEGIN pattern
(see BEGIN/END)
so that the headers are only printed once, at the beginning of
the awk program:
awk 'BEGIN { print "Name Number"
print "---- ------" }
{ printf "%-10s %s\n", $1, $2 }' BBS-list
The above example mixed print and printf statements in
the same program. Using just printf statements can produce the
same results:
awk 'BEGIN { printf "%-10s %s\n", "Name", "Number"
printf "%-10s %s\n", "----", "------" }
{ printf "%-10s %s\n", $1, $2 }' BBS-list
Printing each column heading with the same format specification used for the column elements ensures that the headings are aligned just like the columns.
The fact that the same format specification is used three times can be emphasized by storing it in a variable, like this:
awk 'BEGIN { format = "%-10s %s\n"
printf format, "Name", "Number"
printf format, "----", "------" }
{ printf format, $1, $2 }' BBS-list
At this point, it would be a worthwhile exercise to use the
printf statement to line up the headings and table data for the
inventory-shipped example that was covered earlier in the section
on the print statement
(see Print).
print and printfSo far, the output from print and printf has gone
to the standard
output, usually the terminal. Both print and printf can
also send their output to other places.
This is called redirection.
A redirection appears after the print or printf statement.
Redirections in awk are written just like redirections in shell
commands, except that they are written inside the awk program.
There are four forms of output redirection: output to a file, output
appended to a file, output through a pipe to another command, and output
to a coprocess. They are all shown for the print statement,
but they work identically for printf:
print items > output-fileWhen this type of redirection is used, the output-file is erased before the first output is written to it. Subsequent writes to the same output-file do not erase output-file, but append to it. (This is different from how you use redirections in shell scripts.) If output-file does not exist, it is created. For example, here is how an awk program can write a list of BBS names to one file named name-list, and a list of phone numbers to another file named phone-list:
$ awk '{ print $2 > "phone-list"
> print $1 > "name-list" }' BBS-list
$ cat phone-list
-| 555-5553
-| 555-3412
...
$ cat name-list
-| aardvark
-| alpo-net
...
Each output file contains one name or number per line.
print items >> output-fileprint items | commandThe redirection argument command is actually an awk expression. Its value is converted to a string whose contents give the shell command to be run. For example, the following produces two files, one unsorted list of BBS names, and one list sorted in reverse alphabetical order:
awk '{ print $1 > "names.unsorted"
command = "sort -r > names.sorted"
print $1 | command }' BBS-list
The unsorted list is written with an ordinary redirection, while the sorted list is written by piping through the sort utility.
The next example uses redirection to mail a message to the mailing list `bug-system'. This might be useful when trouble is encountered in an awk script run periodically for system maintenance:
report = "mail bug-system"
print "Awk script failed:", $0 | report
m = ("at record number " FNR " of " FILENAME)
print m | report
close(report)
The message is built using string concatenation and saved in the variable
m. It's then sent down the pipeline to the mail program.
(The parentheses group the items to concatenate—see
Concatenation.)
The close function is called here because it's a good idea to close
the pipe as soon as all the intended output has been sent to it.
See Close Files And Pipes,
for more information.
This example also illustrates the use of a variable to represent a file or command—it is not necessary to always use a string constant. Using a variable is generally a good idea, because awk requires that the string value be spelled identically every time.
print items |& commandgetline.
Thus command is a coprocess, which works together with,
but subsidiary to, the awk program.
This feature is a gawk extension, and is not available in POSIX awk. See Two-way I/O, for a more complete discussion.
Redirecting output using `>', `>>', `|', or `|&' asks the system to open a file, pipe, or coprocess only if the particular file or command you specify has not already been written to by your program or if it has been closed since it was last written to.
It is a common error to use `>' redirection for the first print
to a file, and then to use `>>' for subsequent output:
# clear the file
print "Don't panic" > "guide.txt"
...
# append
print "Avoid improbability generators" >> "guide.txt"
This is indeed how redirections must be used from the shell. But in
awk, it isn't necessary. In this kind of case, a program should
use `>' for all the print statements, since the output file
is only opened once.
As mentioned earlier (see Getline Notes), many Many awk implementations limit the number of pipelines that an awk program may have open to just one! In gawk, there is no such limit. gawk allows a program to open as many pipelines as the underlying operating system permits.
A particularly powerful way to use redirection is to build command lines and pipe them into the shell, sh. For example, suppose you have a list of files brought over from a system where all the file names are stored in uppercase, and you wish to rename them to have names in all lowercase. The following program is both simple and efficient:
{ printf("mv %s %s\n", $0, tolower($0)) | "sh" }
END { close("sh") }
The tolower function returns its argument string with all
uppercase characters converted to lowercase
(see String Functions).
The program builds up a list of command lines,
using the mv utility to rename the files.
It then sends the list to the shell for execution.
gawk provides a number of special file names that it interprets internally. These file names provide access to standard file descriptors, process-related information, and TCP/IP networking.
Running programs conventionally have three input and output streams already available to them for reading and writing. These are known as the standard input, standard output, and standard error output. These streams are, by default, connected to your terminal, but they are often redirected with the shell, via the `<', `<<', `>', `>>', `>&', and `|' operators. Standard error is typically used for writing error messages; the reason there are two separate streams, standard output and standard error, is so that they can be redirected separately.
In other implementations of awk, the only way to write an error message to standard error in an awk program is as follows:
print "Serious error detected!" | "cat 1>&2"
This works by opening a pipeline to a shell command that can access the standard error stream that it inherits from the awk process. This is far from elegant, and it is also inefficient, because it requires a separate process. So people writing awk programs often don't do this. Instead, they send the error messages to the terminal, like this:
print "Serious error detected!" > "/dev/tty"
This usually has the same effect but not always: although the standard error stream is usually the terminal, it can be redirected; when that happens, writing to the terminal is not correct. In fact, if awk is run from a background job, it may not have a terminal at all. Then opening /dev/tty fails.
gawk provides special file names for accessing the three standard streams, as well as any other inherited open files. If the file name matches one of these special names when gawk redirects input or output, then it directly uses the stream that the file name stands for. These special file names work for all operating systems that gawk has been ported to, not just those that are POSIX-compliant:
The file names /dev/stdin, /dev/stdout, and /dev/stderr are aliases for /dev/fd/0, /dev/fd/1, and /dev/fd/2, respectively. However, they are more self-explanatory. The proper way to write an error message in a gawk program is to use /dev/stderr, like this:
print "Serious error detected!" > "/dev/stderr"
Note the use of quotes around the file name. Like any other redirection, the value must be a string. It is a common error to omit the quotes, which leads to confusing results.
gawk also provides special file names that give access to information
about the running gawk process. Each of these “files” provides
a single record of information. To read them more than once, they must
first be closed with the close function
(see Close Files And Pipes).
The file names are:
$1getuid system call
(the real user ID number).
$2geteuid system call
(the effective user ID number).
$3getgid system call
(the real group ID number).
$4getegid system call
(the effective group ID number).
If there are any additional fields, they are the group IDs returned by
the getgroups system call.
(Multiple groups may not be supported on all systems.)
These special file names may be used on the command line as data files, as well as for I/O redirections within an awk program. They may not be used as source files with the -f option.
NOTE: The special files that provide process-related information are now considered
obsolete and will disappear entirely
in the next release of gawk.
gawk prints a warning message every time you use one of
these files.
To obtain process-related information, use the PROCINFO array.
See Auto-set.
Starting with version 3.1 of gawk, awk programs can open a two-way TCP/IP connection, acting as either a client or a server. This is done using a special file name of the form:
/inet/protocol/local-port/remote-host/remote-port
The protocol is one of `tcp', `udp', or `raw', and the other fields represent the other essential pieces of information for making a networking connection. These file names are used with the `|&' operator for communicating with a coprocess (see Two-way I/O). This is an advanced feature, mentioned here only for completeness. Full discussion is delayed until TCP/IP Networking.
Here is a list of things to bear in mind when using the special file names that gawk provides:
PROCINFO array.
See Built-in Variables.
dup'ed from file descriptor 4. Most of
the time this does not matter; however, it is important to not
close any of the files related to file descriptors 0, 1, and 2.
Doing so results in unpredictable behavior.
If the same file name or the same shell command is used with getline
more than once during the execution of an awk program
(see Getline),
the file is opened (or the command is executed) the first time only.
At that time, the first record of input is read from that file or command.
The next time the same file or command is used with getline,
another record is read from it, and so on.
Similarly, when a file or pipe is opened for output, the file name or command associated with it is remembered by awk, and subsequent writes to the same file or command are appended to the previous writes. The file or pipe stays open until awk exits.
This implies that special steps are necessary in order to read the same
file again from the beginning, or to rerun a shell command (rather than
reading more output from the same command). The close function
makes these things possible:
close(filename)
or:
close(command)
The argument filename or command can be any expression. Its value must exactly match the string that was used to open the file or start the command (spaces and other “irrelevant” characters included). For example, if you open a pipe with this:
"sort -r names" | getline foo
then you must close it with this:
close("sort -r names")
Once this function call is executed, the next getline from that
file or command, or the next print or printf to that
file or command, reopens the file or reruns the command.
Because the expression that you use to close a file or pipeline must
exactly match the expression used to open the file or run the command,
it is good practice to use a variable to store the file name or command.
The previous example becomes the following:
sortcom = "sort -r names"
sortcom | getline foo
...
close(sortcom)
This helps avoid hard-to-find typographical errors in your awk programs. Here are some of the reasons for closing an output file:
getline.
For example, suppose a program pipes output to the mail program. If it outputs several lines redirected to this pipe without closing it, they make a single message of several lines. By contrast, if the program closes the pipe after each line of output, then each line makes a separate message.
If you use more files than the system allows you to have open,
gawk attempts to multiplex the available open files among
your data files. gawk's ability to do this depends upon the
facilities of your operating system, so it may not always work. It is
therefore both good practice and good portability advice to always
use close on your files when you are done with them.
In fact, if you are using a lot of pipes, it is essential that
you close commands when done. For example, consider something like this:
{
...
command = ("grep " $1 " /some/file | my_prog -q " $3)
while ((command | getline) > 0) {
process output of command
}
# need close(command) here
}
This example creates a new pipeline based on data in each record.
Without the call to close indicated in the comment, awk
creates child processes to run the commands, until it eventually
runs out of file descriptors for more pipelines.
Even though each command has finished (as indicated by the end-of-file
return status from getline), the child process is not
terminated;21
more importantly, the file descriptor for the pipe
is not closed and released until close is called or
awk exits.
close will silently do nothing if given an argument that
does not represent a file, pipe or coprocess that was opened with
a redirection.
Note also that `close(FILENAME)' has no “magic” effects on the implicit loop that reads through the files named on the command line. It is, more likely, a close of a file that was never opened, so awk silently does nothing.
When using the `|&' operator to communicate with a coprocess,
it is occasionally useful to be able to close one end of the two-way
pipe without closing the other.
This is done by supplying a second argument to close.
As in any other call to close,
the first argument is the name of the command or special file used
to start the coprocess.
The second argument should be a string, with either of the values
"to" or "from". Case does not matter.
As this is an advanced feature, a more complete discussion is
delayed until
Two-way I/O,
which discusses it in more detail and gives an example.
close's Return Value
In many versions of Unix awk, the close function
is actually a statement. It is a syntax error to try and use the return
value from close:
(d.c.)
command = "..."
command | getline info
retval = close(command) # syntax error in most Unix awks
gawk treats close as a function.
The return value is −1 if the argument names something
that was never opened with a redirection, or if there is
a system problem closing the file or process.
In these cases, gawk sets the built-in variable
ERRNO to a string describing the problem.
In gawk,
when closing a pipe or coprocess,
the return value is the exit status of the command.22
Otherwise, it is the return value from the system's close or
fclose C functions when closing input or output
files, respectively.
This value is zero if the close succeeds, or −1 if
it fails.
The POSIX standard is very vague; it says that close
returns zero on success and non-zero otherwise. In general,
different implementations vary in what they report when closing
pipes; thus the return value cannot be used portably.
(d.c.)
Expressions are the basic building blocks of awk patterns and actions. An expression evaluates to a value that you can print, test, or pass to a function. Additionally, an expression can assign a new value to a variable or a field by using an assignment operator.
An expression can serve as a pattern or action statement on its own. Most other kinds of statements contain one or more expressions that specify the data on which to operate. As in other languages, expressions in awk include variables, array references, constants, and function calls, as well as combinations of these with various operators.
The simplest type of expression is the constant, which always has the same value. There are three types of constants: numeric, string, and regular expression.
Each is used in the appropriate context when you need a data value that isn't going to change. Numeric constants can have different forms, but are stored identically internally.
A numeric constant stands for a number. This number can be an integer, a decimal fraction, or a number in scientific (exponential) notation.23 Here are some examples of numeric constants that all have the same value:
105
1.05e+2
1050e-1
A string constant consists of a sequence of characters enclosed in double-quotation marks. For example:
"parrot"
represents the string whose contents are `parrot'. Strings in gawk can be of any length, and they can contain any of the possible eight-bit ASCII characters including ASCII nul (character code zero). Other awk implementations may have difficulty with some character codes.
In awk, all numbers are in decimal; i.e., base 10. Many other programming languages allow you to specify numbers in other bases, often octal (base 8) and hexadecimal (base 16). In octal, the numbers go 0, 1, 2, 3, 4, 5, 6, 7, 10, 11, 12, etc. Just as `11', in decimal, is 1 times 10 plus 1, so `11', in octal, is 1 times 8, plus 1. This equals 9 in decimal. In hexadecimal, there are 16 digits. Since the everyday decimal number system only has ten digits (`0'–`9'), the letters `a' through `f' are used to represent the rest. (Case in the letters is usually irrelevant; hexadecimal `a' and `A' have the same value.) Thus, `11', in hexadecimal, is 1 times 16 plus 1, which equals 17 in decimal.
Just by looking at plain `11', you can't tell what base it's in. So, in C, C++, and other languages derived from C, there is a special notation to help signify the base. Octal numbers start with a leading `0', and hexadecimal numbers start with a leading `0x' or `0X':
110110x11This example shows the difference:
$ gawk 'BEGIN { printf "%d, %d, %d\n", 011, 11, 0x11 }'
-| 9, 11, 17
Being able to use octal and hexadecimal constants in your programs is most useful when working with data that cannot be represented conveniently as characters or as regular numbers, such as binary data of various sorts.
gawk allows the use of octal and hexadecimal
constants in your program text. However, such numbers in the input data
are not treated differently; doing so by default would break old
programs.
(If you really need to do this, use the --non-decimal-data
command-line option;
see Nondecimal Data.)
If you have octal or hexadecimal data,
you can use the strtonum function
(see String Functions)
to convert the data into a number.
Most of the time, you will want to use octal or hexadecimal constants
when working with the built-in bit manipulation functions;
see Bitwise Functions,
for more information.
Unlike some early C implementations, `8' and `9' are not valid in octal constants; e.g., gawk treats `018' as decimal 18:
$ gawk 'BEGIN { print "021 is", 021 ; print 018 }'
-| 021 is 17
-| 18
Octal and hexadecimal source code constants are a gawk extension. If gawk is in compatibility mode (see Options), they are not available.
Once a numeric constant has been converted internally into a number, gawk no longer remembers what the original form of the constant was; the internal value is always used. This has particular consequences for conversion of numbers to strings:
$ gawk 'BEGIN { printf "0x11 is <%s>\n", 0x11 }'
-| 0x11 is <17>
A regexp constant is a regular expression description enclosed in
slashes, such as /^beginning and end$/. Most regexps used in
awk programs are constant, but the `~' and `!~'
matching operators can also match computed or “dynamic” regexps
(which are just ordinary strings or variables that contain a regexp).
When used on the righthand side of the `~' or `!~'
operators, a regexp constant merely stands for the regexp that is to be
matched.
However, regexp constants (such as /foo/) may be used like simple expressions.
When a
regexp constant appears by itself, it has the same meaning as if it appeared
in a pattern, i.e., `($0 ~ /foo/)'
(d.c.)
See Expression Patterns.
This means that the following two code segments:
if ($0 ~ /barfly/ || $0 ~ /camelot/)
print "found"
and:
if (/barfly/ || /camelot/)
print "found"
are exactly equivalent. One rather bizarre consequence of this rule is that the following Boolean expression is valid, but does not do what the user probably intended:
# note that /foo/ is on the left of the ~
if (/foo/ ~ $1) print "found foo"
This code is “obviously” testing $1 for a match against the regexp
/foo/. But in fact, the expression `/foo/ ~ $1' actually means
`($0 ~ /foo/) ~ $1'. In other words, first match the input record
against the regexp /foo/. The result is either zero or one,
depending upon the success or failure of the match. That result
is then matched against the first field in the record.
Because it is unlikely that you would ever really want to make this kind of
test, gawk issues a warning when it sees this construct in
a program.
Another consequence of this rule is that the assignment statement:
matches = /foo/
assigns either zero or one to the variable matches, depending
upon the contents of the current input record.
This feature of the language has never been well documented until the
POSIX specification.
Constant regular expressions are also used as the first argument for
the gensub, sub, and gsub functions, and as the
second argument of the match function
(see String Functions).
Modern implementations of awk, including gawk, allow
the third argument of split to be a regexp constant, but some
older implementations do not.
(d.c.)
This can lead to confusion when attempting to use regexp constants
as arguments to user-defined functions
(see User-defined).
For example:
function mysub(pat, repl, str, global)
{
if (global)
gsub(pat, repl, str)
else
sub(pat, repl, str)
return str
}
{
...
text = "hi! hi yourself!"
mysub(/hi/, "howdy", text, 1)
...
}
In this example, the programmer wants to pass a regexp constant to the
user-defined function mysub, which in turn passes it on to
either sub or gsub. However, what really happens is that
the pat parameter is either one or zero, depending upon whether
or not $0 matches /hi/.
gawk issues a warning when it sees a regexp constant used as
a parameter to a user-defined function, since passing a truth value in
this way is probably not what was intended.
Variables are ways of storing values at one point in your program for use later in another part of your program. They can be manipulated entirely within the program text, and they can also be assigned values on the awk command line.
Variables let you give names to values and refer to them later. Variables
have already been used in many of the examples. The name of a variable
must be a sequence of letters, digits, or underscores, and it may not begin
with a digit. Case is significant in variable names; a and A
are distinct variables.
A variable name is a valid expression by itself; it represents the variable's current value. Variables are given new values with assignment operators, increment operators, and decrement operators. See Assignment Ops.
A few variables have special built-in meanings, such as FS (the
field separator), and NF (the number of fields in the current input
record). See Built-in Variables, for a list of the built-in variables.
These built-in variables can be used and assigned just like all other
variables, but their values are also used or changed automatically by
awk. All built-in variables' names are entirely uppercase.
Variables in awk can be assigned either numeric or string values. The kind of value a variable holds can change over the life of a program. By default, variables are initialized to the empty string, which is zero if converted to a number. There is no need to “initialize” each variable explicitly in awk, which is what you would do in C and in most other traditional languages.
Any awk variable can be set by including a variable assignment among the arguments on the command line when awk is invoked (see Other Arguments). Such an assignment has the following form:
variable=text
With it, a variable is set either at the beginning of the awk run or in between input files. When the assignment is preceded with the -v option, as in the following:
-v variable=text
the variable is set at the very beginning, even before the
BEGIN rules are run. The -v option and its assignment
must precede all the file name arguments, as well as the program text.
(See Options, for more information about
the -v option.)
Otherwise, the variable assignment is performed at a time determined by
its position among the input file arguments—after the processing of the
preceding input file argument. For example:
awk '{ print $n }' n=4 inventory-shipped n=2 BBS-list
prints the value of field number n for all input records. Before
the first file is read, the command line sets the variable n
equal to four. This causes the fourth field to be printed in lines from
the file inventory-shipped. After the first file has finished,
but before the second file is started, n is set to two, so that the
second field is printed in lines from BBS-list:
$ awk '{ print $n }' n=4 inventory-shipped n=2 BBS-list
-| 15
-| 24
...
-| 555-5553
-| 555-3412
...
Command-line arguments are made available for explicit examination by
the awk program in the ARGV array
(see ARGC and ARGV).
awk processes the values of command-line assignments for escape
sequences
(see Escape Sequences).
(d.c.)
Strings are converted to numbers and numbers are converted to strings, if the context
of the awk program demands it. For example, if the value of
either foo or bar in the expression `foo + bar'
happens to be a string, it is converted to a number before the addition
is performed. If numeric values appear in string concatenation, they
are converted to strings. Consider the following:
two = 2; three = 3
print (two three) + 4
This prints the (numeric) value 27. The numeric values of
the variables two and three are converted to strings and
concatenated together. The resulting string is converted back to the
number 23, to which 4 is then added.
If, for some reason, you need to force a number to be converted to a
string, concatenate the empty string, "", with that number.
To force a string to be converted to a number, add zero to that string.
A string is converted to a number by interpreting any numeric prefix
of the string as numerals:
"2.5" converts to 2.5, "1e3" converts to 1000, and "25fix"
has a numeric value of 25.
Strings that can't be interpreted as valid numbers convert to zero.
The exact manner in which numbers are converted into strings is controlled
by the awk built-in variable CONVFMT (see Built-in Variables).
Numbers are converted using the sprintf function
with CONVFMT as the format
specifier
(see String Functions).
CONVFMT's default value is "%.6g", which prints a value with
at least six significant digits. For some applications, you might want to
change it to specify more precision.
On most modern machines,
17 digits is enough to capture a floating-point number's
value exactly,
most of the time.24
Strange results can occur if you set CONVFMT to a string that doesn't
tell sprintf how to format floating-point numbers in a useful way.
For example, if you forget the `%' in the format, awk converts
all numbers to the same constant string.
As a special case, if a number is an integer, then the result of converting
it to a string is always an integer, no matter what the value of
CONVFMT may be. Given the following code fragment:
CONVFMT = "%2.2f"
a = 12
b = a ""
b has the value "12", not "12.00".
(d.c.)
Prior to the POSIX standard, awk used the value
of OFMT for converting numbers to strings. OFMT
specifies the output format to use when printing numbers with print.
CONVFMT was introduced in order to separate the semantics of
conversion from the semantics of printing. Both CONVFMT and
OFMT have the same default value: "%.6g". In the vast majority
of cases, old awk programs do not change their behavior.
However, these semantics for OFMT are something to keep in mind if you must
port your new style program to older implementations of awk.
We recommend
that instead of changing your programs, just port gawk itself.
See Print,
for more information on the print statement.
Finally, once again, where you are can matter when it comes to
converting between numbers and strings. In
Locales, we mentioned that the
local character set and language (the locale) can affect how gawk matches
characters. The locale also affects numeric formats. In particular, for awk
programs, it affects the decimal point character. The "C" locale, and most
English-language locales, use the period character (`.') as the decimal point.
However, many (if not most) European and non-English locales use the comma (`,')
as the decimal point character.
The POSIX standard says that awk always uses the period as the decimal
point when reading the awk program source code, and for command-line
variable assignments (see Other Arguments).
However, when interpreting input data, for print and printf output,
and for number to string conversion, the local decimal point character is used.
As of version 3.1.3, gawk fully complies with this aspect
of the standard. Here are some examples indicating the difference in behavior,
on a GNU/Linux system:
$ gawk 'BEGIN { printf "%g\n", 3.1415927 }'
-| 3.14159
$ LC_ALL=en_DK gawk 'BEGIN { printf "%g\n", 3.1415927 }'
-| 3,14159
$ echo 4,321 | gawk '{ print $1 + 1 }'
-| 5
$ echo 4,321 | LC_ALL=en_DK gawk '{ print $1 + 1 }'
-| 5,321
The `en_DK' locale is for English in Denmark, where the comma acts as
the decimal point separator. In the normal "C" locale, gawk
treats `4,321' as `4', while in the Danish locale, it's treated
as the full number, `4.321'.
The awk language uses the common arithmetic operators when evaluating expressions. All of these arithmetic operators follow normal precedence rules and work as you would expect them to.
The following example uses a file named grades, which contains a list of student names as well as three test scores per student (it's a small class):
Pat 100 97 58
Sandy 84 72 93
Chris 72 92 89
This programs takes the file grades and prints the average of the scores:
$ awk '{ sum = $2 + $3 + $4 ; avg = sum / 3
> print $1, avg }' grades
-| Pat 85
-| Sandy 83
-| Chris 84.3333
The following list provides the arithmetic operators in awk, in order from the highest precedence to the lowest:
- x+ x ^ y ** y * y / y % y + y - yUnary plus and minus have the same precedence, the multiplication operators all have the same precedence, and addition and subtraction have the same precedence.
When computing the remainder of x % y,
the quotient is rounded toward zero to an integer and
multiplied by y. This result is subtracted from x;
this operation is sometimes known as “trunc-mod.” The following
relation always holds:
b * int(a / b) + (a % b) == a
One possibly undesirable effect of this definition of remainder is that
x % y is negative if x is negative. Thus:
-17 % 8 = -1
In other awk implementations, the signedness of the remainder may be machine-dependent.
NOTE: The POSIX standard only specifies the use of `^' for exponentiation. For maximum portability, do not use the `**' operator.
It seemed like a good idea at the time.
Brian Kernighan
There is only one string operation: concatenation. It does not have a specific operator to represent it. Instead, concatenation is performed by writing expressions next to one another, with no operator. For example:
$ awk '{ print "Field number one: " $1 }' BBS-list
-| Field number one: aardvark
-| Field number one: alpo-net
...
Without the space in the string constant after the `:', the line runs together. For example:
$ awk '{ print "Field number one:" $1 }' BBS-list
-| Field number one:aardvark
-| Field number one:alpo-net
...
Because string concatenation does not have an explicit operator, it is
often necessary to insure that it happens at the right time by using
parentheses to enclose the items to concatenate. For example, the
following code fragment does not concatenate file and name
as you might expect:
file = "file"
name = "name"
print "something meaningful" > file name
It is necessary to use the following:
print "something meaningful" > (file name)
Parentheses should be used around concatenation in all but the most common contexts, such as on the righthand side of `='. Be careful about the kinds of expressions used in string concatenation. In particular, the order of evaluation of expressions used for concatenation is undefined in the awk language. Consider this example:
BEGIN {
a = "don't"
print (a " " (a = "panic"))
}
It is not defined whether the assignment to a happens
before or after the value of a is retrieved for producing the
concatenated value. The result could be either `don't panic',
or `panic panic'.
The precedence of concatenation, when mixed with other operators, is often
counter-intuitive. Consider this example:
$ awk 'BEGIN { print -12 " " -24 }'
-| -12-24
This “obviously” is concatenating −12, a space, and −24. But where did the space disappear to? The answer lies in the combination of operator precedences and awk's automatic conversion rules. To get the desired result, write the program in the following manner:
$ awk 'BEGIN { print -12 " " (-24) }'
-| -12 -24
This forces awk to treat the `-' on the `-24' as unary. Otherwise, it's parsed as follows:
−12 (" " − 24)
=> −12 (0 − 24)
=> −12 (−24)
=> −12−24
As mentioned earlier, when doing concatenation, parenthesize. Otherwise, you're never quite sure what you'll get.
An assignment is an expression that stores a (usually different)
value into a variable. For example, let's assign the value one to the variable
z:
z = 1
After this expression is executed, the variable z has the value one.
Whatever old value z had before the assignment is forgotten.
Assignments can also store string values. For example, the
following stores
the value "this food is good" in the variable message:
thing = "food"
predicate = "good"
message = "this " thing " is " predicate
This also illustrates string concatenation. The `=' sign is called an assignment operator. It is the simplest assignment operator because the value of the righthand operand is stored unchanged. Most operators (addition, concatenation, and so on) have no effect except to compute a value. If the value isn't used, there's no reason to use the operator. An assignment operator is different; it does produce a value, but even if you ignore it, the assignment still makes itself felt through the alteration of the variable. We call this a side effect.
The lefthand operand of an assignment need not be a variable (see Variables); it can also be a field (see Changing Fields) or an array element (see Arrays). These are all called lvalues, which means they can appear on the lefthand side of an assignment operator. The righthand operand may be any expression; it produces the new value that the assignment stores in the specified variable, field, or array element. (Such values are called rvalues.)
It is important to note that variables do not have permanent types.
A variable's type is simply the type of whatever value it happens
to hold at the moment. In the following program fragment, the variable
foo has a numeric value at first, and a string value later on:
foo = 1
print foo
foo = "bar"
print foo
When the second assignment gives foo a string value, the fact that
it previously had a numeric value is forgotten.
String values that do not begin with a digit have a numeric value of
zero. After executing the following code, the value of foo is five:
foo = "a string"
foo = foo + 5
NOTE: Using a variable as a number and then later as a string can be confusing and is poor programming style. The previous two examples illustrate how awk works, not how you should write your programs!
An assignment is an expression, so it has a value—the same value that is assigned. Thus, `z = 1' is an expression with the value one. One consequence of this is that you can write multiple assignments together, such as:
x = y = z = 5
This example stores the value five in all three variables
(x, y, and z).
It does so because the
value of `z = 5', which is five, is stored into y and then
the value of `y = z = 5', which is five, is stored into x.
Assignments may be used anywhere an expression is called for. For
example, it is valid to write `x != (y = 1)' to set y to one,
and then test whether x equals one. But this style tends to make
programs hard to read; such nesting of assignments should be avoided,
except perhaps in a one-shot program.
Aside from `=', there are several other assignment operators that
do arithmetic with the old value of the variable. For example, the
operator `+=' computes a new value by adding the righthand value
to the old value of the variable. Thus, the following assignment adds
five to the value of foo:
foo += 5
This is equivalent to the following:
foo = foo + 5
Use whichever makes the meaning of your program clearer.
There are situations where using `+=' (or any assignment operator) is not the same as simply repeating the lefthand operand in the righthand expression. For example:
# Thanks to Pat Rankin for this example
BEGIN {
foo[rand()] += 5
for (x in foo)
print x, foo[x]
bar[rand()] = bar[rand()] + 5
for (x in bar)
print x, bar[x]
}
The indices of bar are practically guaranteed to be different, because
rand returns different values each time it is called.
(Arrays and the rand function haven't been covered yet.
See Arrays,
and see Numeric Functions, for more information).
This example illustrates an important fact about assignment
operators: the lefthand expression is only evaluated once.
It is up to the implementation as to which expression is evaluated
first, the lefthand or the righthand.
Consider this example:
i = 1
a[i += 2] = i + 1
The value of a[3] could be either two or four.
table-assign-ops lists the arithmetic assignment operators. In each case, the righthand operand is an expression whose value is converted to a number.
Table 5.1: Arithmetic Assignment Operators
NOTE: Only the `^=' operator is specified by POSIX. For maximum portability, do not use the `**=' operator.
There is a syntactic ambiguity between the `/=' assignment operator and regexp constants whose first character is an `='. (d.c.) This is most notable in commercial awk versions. For example:
$ awk /==/ /dev/null
error--> awk: syntax error at source line 1
error--> context is
error--> >>> /= <<<
error--> awk: bailing out at source line 1
A workaround is:
awk '/[=]=/' /dev/null
gawk does not have this problem, nor do the other freely available versions described in Other Versions.
Increment and decrement operators increase or decrease the value of a variable by one. An assignment operator can do the same thing, so the increment operators add no power to the awk language; however, they are convenient abbreviations for very common operations.
The operator used for adding one is written `++'. It can be used to increment
a variable either before or after taking its value.
To pre-increment a variable v, write `++v'. This adds
one to the value of v—that new value is also the value of the
expression. (The assignment expression `v += 1' is completely
equivalent.)
Writing the `++' after the variable specifies post-increment. This
increments the variable value just the same; the difference is that the
value of the increment expression itself is the variable's old
value. Thus, if foo has the value four, then the expression `foo++'
has the value four, but it changes the value of foo to five.
In other words, the operator returns the old value of the variable,
but with the side effect of incrementing it.
The post-increment `foo++' is nearly the same as writing `(foo
+= 1) - 1'. It is not perfectly equivalent because all numbers in
awk are floating-point—in floating-point, `foo + 1 - 1' does
not necessarily equal foo. But the difference is minute as
long as you stick to numbers that are fairly small (less than 10e12).
Fields and array elements are incremented just like variables. (Use `$(i++)' when you want to do a field reference and a variable increment at the same time. The parentheses are necessary because of the precedence of the field reference operator `$'.)
The decrement operator `--' works just like `++', except that it subtracts one instead of adding it. As with `++', it can be used before the lvalue to pre-decrement or after it to post-decrement. Following is a summary of increment and decrement expressions:
++lvalue++--lvalue--Doctor, doctor! It hurts when I do this!
So don't do that!
Groucho Marx
What happens for something like the following?
b = 6
print b += b++
Or something even stranger?
b = 6
b += ++b + b++
print b
In other words, when do the various side effects prescribed by the postfix operators (`b++') take effect? When side effects happen is implementation defined. In other words, it is up to the particular version of awk. The result for the first example may be 12 or 13, and for the second, it may be 22 or 23.
In short, doing things like this is not recommended and definitely not anything that you can rely upon for portability. You should avoid such things in your own programs.
Many programming languages have a special representation for the concepts
of “true” and “false.” Such languages usually use the special
constants true and false, or perhaps their uppercase
equivalents.
However, awk is different.
It borrows a very simple concept of true and
false from C. In awk, any nonzero numeric value or any
nonempty string value is true. Any other value (zero or the null
string "") is false. The following program prints `A strange
truth value' three times:
BEGIN {
if (3.1415927)
print "A strange truth value"
if ("Four Score And Seven Years Ago")
print "A strange truth value"
if (j = 57)
print "A strange truth value"
}
There is a surprising consequence of the “nonzero or non-null” rule:
the string constant "0" is actually true, because it is non-null.
(d.c.)
The Guide is definitive. Reality is frequently inaccurate.
The Hitchhiker's Guide to the Galaxy
Unlike other programming languages, awk variables do not have a fixed type. Instead, they can be either a number or a string, depending upon the value that is assigned to them.
The 1992 POSIX standard introduced
the concept of a numeric string, which is simply a string that looks
like a number—for example, " +2". This concept is used
for determining the type of a variable.
The type of the variable is important because the types of two variables
determine how they are compared.
In gawk, variable typing follows these rules:
getline input, FILENAME, ARGV elements,
ENVIRON elements, and the
elements of an array created by split that are numeric strings
have the strnum attribute. Otherwise, they have the string
attribute.
Uninitialized variables also have the strnum attribute.
The last rule is particularly important. In the following program,
a has numeric type, even though it is later used in a string
operation:
BEGIN {
a = 12.345
b = a " is a cute number"
print b
}
When two operands are compared, either string comparison or numeric comparison may be used. This depends upon the attributes of the operands, according to the following symmetric matrix:
+——————————————————————–
| STRING NUMERIC STRNUM
———–+——————————————————————–
|
STRING | string string string
|
NUMERIC | string numeric numeric
|
STRNUM | string numeric numeric
———–+——————————————————————–
The basic idea is that user input that looks numeric—and only
user input—should be treated as numeric, even though it is actually
made of characters and is therefore also a string.
Thus, for example, the string constant " +3.14"
is a string, even though it looks numeric,
and is never treated as number for comparison
purposes.
In short, when one operand is a “pure” string, such as a string constant, then a string comparison is performed. Otherwise, a numeric comparison is performed.25
Comparison expressions compare strings or numbers for relationships such as equality. They are written using relational operators, which are a superset of those in C. table-relational-ops describes them.
| Expression | Result
|
|---|---|
x < y | True if x is less than y.
|
x <= y | True if x is less than or equal to y.
|
x > y | True if x is greater than y.
|
x >= y | True if x is greater than or equal to y.
|
x == y | True if x is equal to y.
|
x != y | True if x is not equal to y.
|
x ~ y | True if the string x matches the regexp denoted by y.
|
x !~ y | True if the string x does not match the regexp denoted by y.
|
subscript in array | True if the array array has an element with the subscript subscript.
|
Table 5.2: Relational Operators
Comparison expressions have the value one if true and zero if false.
When comparing operands of mixed types, numeric operands are converted
to strings using the value of CONVFMT
(see Conversion).
Strings are compared
by comparing the first character of each, then the second character of each,
and so on. Thus, "10" is less than "9". If there are two
strings where one is a prefix of the other, the shorter string is less than
the longer one. Thus, "abc" is less than "abcd".
It is very easy to accidentally mistype the `==' operator and leave off one of the `=' characters. The result is still valid awk code, but the program does not do what is intended:
if (a = b) # oops! should be a == b
...
else
...
Unless b happens to be zero or the null string, the if
part of the test always succeeds. Because the operators are
so similar, this kind of error is very difficult to spot when
scanning the source code.
The following table of expressions illustrates the kind of comparison gawk performs, as well as what the result of the comparison is:
1.5 <= 2.0"abc" >= "xyz"1.5 != " +2""1e2" < "3"a = 2; b = "2"a == ba = 2; b = " +2"a == bIn the next example:
$ echo 1e2 3 | awk '{ print ($1 < $2) ? "true" : "false" }'
-| false
the result is `false' because both $1 and $2
are user input. They are numeric strings—therefore both have
the strnum attribute, dictating a numeric comparison.
The purpose of the comparison rules and the use of numeric strings is
to attempt to produce the behavior that is “least surprising,” while
still “doing the right thing.”
String comparisons and regular expression comparisons are very different.
For example:
x == "foo"
has the value one, or is true if the variable x
is precisely `foo'. By contrast:
x ~ /foo/
has the value one if x contains `foo', such as
"Oh, what a fool am I!".
The righthand operand of the `~' and `!~' operators may be
either a regexp constant (/.../) or an ordinary
expression. In the latter case, the value of the expression as a string is used as a
dynamic regexp (see Regexp Usage; also
see Computed Regexps).
In modern implementations of awk, a constant regular
expression in slashes by itself is also an expression. The regexp
/regexp/ is an abbreviation for the following comparison expression:
$0 ~ /regexp/
One special place where /foo/ is not an abbreviation for
`$0 ~ /foo/' is when it is the righthand operand of `~' or
`!~'.
See Using Constant Regexps,
where this is discussed in more detail.
A Boolean expression is a combination of comparison expressions or matching expressions, using the Boolean operators “or” (`||'), “and” (`&&'), and “not” (`!'), along with parentheses to control nesting. The truth value of the Boolean expression is computed by combining the truth values of the component expressions. Boolean expressions are also referred to as logical expressions. The terms are equivalent.
Boolean expressions can be used wherever comparison and matching
expressions can be used. They can be used in if, while,
do, and for statements
(see Statements).
They have numeric values (one if true, zero if false) that come into play
if the result of the Boolean expression is stored in a variable or
used in arithmetic.
In addition, every Boolean expression is also a valid pattern, so you can use one as a pattern to control the execution of rules. The Boolean operators are:
&& boolean2 if ($0 ~ /2400/ && $0 ~ /foo/) print
The subexpression boolean2 is evaluated only if boolean1
is true. This can make a difference when boolean2 contains
expressions that have side effects. In the case of `$0 ~ /foo/ &&
($2 == bar++)', the variable bar is not incremented if there is
no substring `foo' in the record.
|| boolean2 if ($0 ~ /2400/ || $0 ~ /foo/) print
The subexpression boolean2 is evaluated only if boolean1
is false. This can make a difference when boolean2 contains
expressions that have side effects.
! boolean BEGIN { if (! ("HOME" in ENVIRON))
print "no home!" }
(The in operator is described in
Reference to Elements.)
The `&&' and `||' operators are called short-circuit operators because of the way they work. Evaluation of the full expression is “short-circuited” if the result can be determined part way through its evaluation.
Statements that use `&&' or `||' can be continued simply by putting a newline after them. But you cannot put a newline in front of either of these operators without using backslash continuation (see Statements/Lines).
The actual value of an expression using the `!' operator is either one or zero, depending upon the truth value of the expression it is applied to. The `!' operator is often useful for changing the sense of a flag variable from false to true and back again. For example, the following program is one way to print lines in between special bracketing lines:
$1 == "START" { interested = ! interested; next }
interested == 1 { print }
$1 == "END" { interested = ! interested; next }
The variable interested, as with all awk variables, starts
out initialized to zero, which is also false. When a line is seen whose
first field is `START', the value of interested is toggled
to true, using `!'. The next rule prints lines as long as
interested is true. When a line is seen whose first field is
`END', interested is toggled back to false.
NOTE: Thenextstatement is discussed in Next Statement.nexttells awk to skip the rest of the rules, get the next record, and start processing the rules over again at the top. The reason it's there is to avoid printing the bracketing `START' and `END' lines.
A conditional expression is a special kind of expression that has three operands. It allows you to use one expression's value to select one of two other expressions. The conditional expression is the same as in the C language, as shown here:
selector ? if-true-exp : if-false-exp
There are three subexpressions. The first, selector, is always
computed first. If it is “true” (not zero or not null), then
if-true-exp is computed next and its value becomes the value of
the whole expression. Otherwise, if-false-exp is computed next
and its value becomes the value of the whole expression.
For example, the following expression produces the absolute value of x:
x >= 0 ? x : -x
Each time the conditional expression is computed, only one of
if-true-exp and if-false-exp is used; the other is ignored.
This is important when the expressions have side effects. For example,
this conditional expression examines element i of either array
a or array b, and increments i:
x == y ? a[i++] : b[i++]
This is guaranteed to increment i exactly once, because each time
only one of the two increment expressions is executed
and the other is not.
See Arrays,
for more information about arrays.
As a minor gawk extension, a statement that uses `?:' can be continued simply by putting a newline after either character. However, putting a newline in front of either character does not work without using backslash continuation (see Statements/Lines). If --posix is specified (see Options), then this extension is disabled.
A function is a name for a particular calculation.
This enables you to
ask for it by name at any point in the program. For
example, the function sqrt computes the square root of a number.
A fixed set of functions are built-in, which means they are
available in every awk program. The sqrt function is one
of these. See Built-in, for a list of built-in
functions and their descriptions. In addition, you can define
functions for use in your program.
See User-defined,
for instructions on how to do this.
The way to use a function is with a function call expression, which consists of the function name followed immediately by a list of arguments in parentheses. The arguments are expressions that provide the raw materials for the function's calculations. When there is more than one argument, they are separated by commas. If there are no arguments, just write `()' after the function name. The following examples show function calls with and without arguments:
sqrt(x^2 + y^2) one argument
atan2(y, x) two arguments
rand() no arguments
Caution: Do not put any space between the function name and the open-parenthesis! A user-defined function name looks just like the name of a variable—a space would make the expression look like concatenation of a variable with an expression inside parentheses.
With built-in functions, space before the parenthesis is harmless, but
it is best not to get into the habit of using space to avoid mistakes
with user-defined functions. Each function expects a particular number
of arguments. For example, the sqrt function must be called with
a single argument, the number of which to take the square root:
sqrt(argument)
Some of the built-in functions have one or more optional arguments. If those arguments are not supplied, the functions use a reasonable default value. See Built-in, for full details. If arguments are omitted in calls to user-defined functions, then those arguments are treated as local variables and initialized to the empty string (see User-defined).
Like every other expression, the function call has a value, which is computed by the function based on the arguments you give it. In this example, the value of `sqrt(argument)' is the square root of argument. A function can also have side effects, such as assigning values to certain variables or doing I/O. The following program reads numbers, one number per line, and prints the square root of each one:
$ awk '{ print "The square root of", $1, "is", sqrt($1) }'
1
-| The square root of 1 is 1
3
-| The square root of 3 is 1.73205
5
-| The square root of 5 is 2.23607
Ctrl-d
Operator precedence determines how operators are grouped when
different operators appear close by in one expression. For example,
`*' has higher precedence than `+'; thus, `a + b * c'
means to multiply b and c, and then add a to the
product (i.e., `a + (b * c)').
The normal precedence of the operators can be overruled by using parentheses. Think of the precedence rules as saying where the parentheses are assumed to be. In fact, it is wise to always use parentheses whenever there is an unusual combination of operators, because other people who read the program may not remember what the precedence is in this case. Even experienced programmers occasionally forget the exact rules, which leads to mistakes. Explicit parentheses help prevent any such mistakes.
When operators of equal precedence are used together, the leftmost operator groups first, except for the assignment, conditional, and exponentiation operators, which group in the opposite order. Thus, `a - b + c' groups as `(a - b) + c' and `a = b = c' groups as `a = (b = c)'.
The precedence of prefix unary operators does not matter as long as only unary operators are involved, because there is only one way to interpret them: innermost first. Thus, `$++i' means `$(++i)' and `++$x' means `++($x)'. However, when another operator follows the operand, then the precedence of the unary operators can matter. `$x^2' means `($x)^2', but `-x^2' means `-(x^2)', because `-' has lower precedence than `^', whereas `$' has higher precedence. This table presents awk's operators, in order of highest to lowest precedence:
(...)$++ --^ **+ - !* / %+ -< <= == !=> >= >> | |&Note that the I/O redirection operators in print and printf
statements belong to the statement level, not to expressions. The
redirection does not produce an expression that could be the operand of
another operator. As a result, it does not make sense to use a
redirection operator near another operator of lower precedence without
parentheses. Such combinations (for example, `print foo > a ? b : c'),
result in syntax errors.
The correct way to write this statement is `print foo > (a ? b : c)'.
~ !~in&&||?:= += -= *=/= %= ^= **=NOTE: The `|&', `**', and `**=' operators are not specified by POSIX. For maximum portability, do not use them.
As you have already seen, each awk statement consists of a pattern with an associated action. This chapter describes how you build patterns and actions, what kinds of things you can do within actions, and awk's built-in variables.
The pattern-action rules and the statements available for use within actions form the core of awk programming. In a sense, everything covered up to here has been the foundation that programs are built on top of. Now it's time to start building something useful.
Patterns in awk control the execution of rules—a rule is executed when its pattern matches the current input record. The following is a summary of the types of awk patterns:
/regular expression/, pat2BEGINENDRegular expressions are one of the first kinds of patterns presented in this book. This kind of pattern is simply a regexp constant in the pattern part of a rule. Its meaning is `$0 ~ /pattern/'. The pattern matches when the input record matches the regexp. For example:
/foo|bar|baz/ { buzzwords++ }
END { print buzzwords, "buzzwords seen" }
Any awk expression is valid as an awk pattern.
The pattern matches if the expression's value is nonzero (if a
number) or non-null (if a string).
The expression is reevaluated each time the rule is tested against a new
input record. If the expression uses fields such as $1, the
value depends directly on the new input record's text; otherwise, it
depends on only what has happened so far in the execution of the
awk program.
Comparison expressions, using the comparison operators described in
Typing and Comparison,
are a very common kind of pattern.
Regexp matching and nonmatching are also very common expressions.
The left operand of the `~' and `!~' operators is a string.
The right operand is either a constant regular expression enclosed in
slashes (/regexp/), or any expression whose string value
is used as a dynamic regular expression
(see Computed Regexps).
The following example prints the second field of each input record
whose first field is precisely `foo':
$ awk '$1 == "foo" { print $2 }' BBS-list
(There is no output, because there is no BBS site with the exact name `foo'.) Contrast this with the following regular expression match, which accepts any record with a first field that contains `foo':
$ awk '$1 ~ /foo/ { print $2 }' BBS-list
-| 555-1234
-| 555-6699
-| 555-6480
-| 555-2127
A regexp constant as a pattern is also a special case of an expression
pattern. The expression /foo/ has the value one if `foo'
appears in the current input record. Thus, as a pattern, /foo/
matches any record containing `foo'.
Boolean expressions are also commonly used as patterns. Whether the pattern matches an input record depends on whether its subexpressions match. For example, the following command prints all the records in BBS-list that contain both `2400' and `foo':
$ awk '/2400/ && /foo/' BBS-list
-| fooey 555-1234 2400/1200/300 B
The following command prints all records in BBS-list that contain either `2400' or `foo' (or both, of course):
$ awk '/2400/ || /foo/' BBS-list
-| alpo-net 555-3412 2400/1200/300 A
-| bites 555-1675 2400/1200/300 A
-| fooey 555-1234 2400/1200/300 B
-| foot 555-6699 1200/300 B
-| macfoo 555-6480 1200/300 A
-| sdace 555-3430 2400/1200/300 A
-| sabafoo 555-2127 1200/300 C
The following command prints all records in BBS-list that do not contain the string `foo':
$ awk '! /foo/' BBS-list
-| aardvark 555-5553 1200/300 B
-| alpo-net 555-3412 2400/1200/300 A
-| barfly 555-7685 1200/300 A
-| bites 555-1675 2400/1200/300 A
-| camelot 555-0542 300 C
-| core 555-2912 1200/300 C
-| sdace 555-3430 2400/1200/300 A
The subexpressions of a Boolean operator in a pattern can be constant regular
expressions, comparisons, or any other awk expressions. Range
patterns are not expressions, so they cannot appear inside Boolean
patterns. Likewise, the special patterns BEGIN and END,
which never match any input record, are not expressions and cannot
appear inside Boolean patterns.
A range pattern is made of two patterns separated by a comma, in the form `begpat, endpat'. It is used to match ranges of consecutive input records. The first pattern, begpat, controls where the range begins, while endpat controls where the pattern ends. For example, the following:
awk '$1 == "on", $1 == "off"' myfile
prints every record in myfile between `on'/`off' pairs, inclusive.
A range pattern starts out by matching begpat against every input record. When a record matches begpat, the range pattern is turned on and the range pattern matches this record as well. As long as the range pattern stays turned on, it automatically matches every input record read. The range pattern also matches endpat against every input record; when this succeeds, the range pattern is turned off again for the following record. Then the range pattern goes back to checking begpat against each record.
The record that turns on the range pattern and the one that turns it
off both match the range pattern. If you don't want to operate on
these records, you can write if statements in the rule's action
to distinguish them from the records you are interested in.
It is possible for a pattern to be turned on and off by the same
record. If the record satisfies both conditions, then the action is
executed for just that record.
For example, suppose there is text between two identical markers (e.g.,
the `%' symbol), each on its own line, that should be ignored.
A first attempt would be to
combine a range pattern that describes the delimited text with the
next statement
(not discussed yet, see Next Statement).
This causes awk to skip any further processing of the current
record and start over again with the next input record. Such a program
looks like this:
/^%$/,/^%$/ { next }
{ print }
This program fails because the range pattern is both turned on and turned off by the first line, which just has a `%' on it. To accomplish this task, write the program in the following manner, using a flag:
/^%$/ { skip = ! skip; next }
skip == 1 { next } # skip lines with `skip' set
In a range pattern, the comma (`,') has the lowest precedence of all the operators (i.e., it is evaluated last). Thus, the following program attempts to combine a range pattern with another, simpler test:
echo Yes | awk '/1/,/2/ || /Yes/'
The intent of this program is `(/1/,