sed and awk for Real Work

In one line

sed transforms text line by line (a 'find and replace' engine); awk processes text COLUMN by column — together they cover text transformation tasks grep alone can't handle.

0/4 · 0%

Think of it like this

Real-life example: sed is like a proofreader with a strict find-and-replace instruction ('change every occurrence of X to Y'); awk is like an accountant working through a spreadsheet, pulling out specific COLUMNS and doing simple math on them — different jobs, different tools.

Key ideas

01
The single most common sed use: sed 's/old/new/' file.txt replaces the FIRST occurrence of 'old' with 'new' on each line; sed 's/old/new/g' (the trailing g for 'global') replaces EVERY occurrence on each line, not just the first.
02
sed -i edits the file IN PLACE, overwriting it directly — genuinely useful, but genuinely irreversible (there's no confirmation prompt), so it's worth running the command WITHOUT -i first to preview the output before committing to modifying the actual file.
03
awk treats each line as a set of whitespace-separated COLUMNS ($1, $2, $3, ...; $0 means the whole line). awk '{print $1}' file.txt prints just the first column of every line — genuinely powerful for parsing structured text like ls -l output or a CSV-like log format.
04
awk -F: '{print $1}' /etc/passwd uses -F to set a custom field separator (colon, since /etc/passwd is colon-delimited) — this exact pattern is how you'd extract just the usernames from that file, a cleaner alternative to the cut -d: -f1 approach from earlier topics.
05
awk can also filter AND compute: awk '$3 > 1000 {print $1}' /etc/passwd prints the username (column 1) only for lines where the UID (column 3) is greater than 1000 — a genuinely non-trivial filter that would require considerably more code in most general-purpose languages.

In your stack

→
A realistic use: sed 's/DEBUG/INFO/g' application.properties to bulk-change a logging level across a config file before committing, or awk '{print $NF}' access.log to pull out just the LAST column ($NF means 'last field') of an access log — often the response time or status code in a typical log format.

Code & diagrams

sed-and-awk.shmarkdown

sed -i is irreversible — always preview without -i first, on real files.

# sed: preview a substitution WITHOUT modifying the file
echo "hello world" | sed 's/world/there/'
# -> hello there

# Global replacement (all occurrences, not just the first)
echo "aaa" | sed 's/a/b/'
# -> baa (only the first)
echo "aaa" | sed 's/a/b/g'
# -> bbb (all of them)

# awk: extract columns
echo "Amit 28 Engineer" | awk '{print $1}'
# -> Amit
echo "Amit 28 Engineer" | awk '{print $2, $3}'
# -> 28 Engineer

# awk with a custom field separator, mirroring cut's earlier example
awk -F: '{print $1}' /etc/passwd | head -5

# awk with a filter condition
awk -F: '$3 > 1000 {print $1}' /etc/passwd | head -5

Explain it without notes

Why should you always test a sed substitution WITHOUT -i before running it with -i on a real file?

You need to print just the third column of a space-separated file. Would you reach for grep, sed, or awk, and why?

Practice

Use sed (without -i first) to replace every space in a line of text with an underscore, then verify the substitution pattern is correct before ever considering adding -i.

Use awk -F: on /etc/passwd to print both the username (field 1) and the shell (the last field) for every user, in one command.

Trade-offs

↔
sed and awk are genuinely terse and can look like line noise to someone unfamiliar with them — for a one-off interactive task they're unbeatable, but for anything complex enough to need comments and real logic, a proper script in Python or even bash (Phase 6) is usually more maintainable than a dense sed/awk one-liner nobody (including future you) can read six months later.

Done when you can

I can write a basic sed substitution and know why to test it without -i first.
I can use awk to extract specific columns from structured text.
I know when to reach for awk (columns) versus grep (whole-line matching) versus sed (substitution).

Command Palette