Sorting, Counting, and Combining

In one line

sort, uniq, wc, and cut are small enough to feel almost trivial individually, but combined in a pipeline they answer real questions — like 'what are the top 5 most frequent values in this file' — with zero code written.

0/4 · 0%

Key ideas

01
sort file.txt sorts lines alphabetically by default. sort -n sorts NUMERICALLY instead (critical — alphabetical sort puts '10' before '2', since it compares character by character; numeric sort gets this right). sort -r reverses the order, and sort -k2 sorts by the second column specifically.
02
uniq removes ADJACENT duplicate lines — critically, it only catches duplicates that are already NEXT TO each other, which is why uniq is almost always preceded by sort in a real pipeline (sort file.txt | uniq) to guarantee duplicates end up adjacent first.
03
uniq -c counts how many times each line appears (prefixing each unique line with its count) — combined with sort, sort file.txt | uniq -c | sort -rn is a genuinely famous, extremely useful pipeline: it produces a frequency count of every unique line, sorted from MOST to least frequent.
04
wc (word count) with no flags shows lines, words, and bytes. wc -l shows just the LINE count — one of the most frequently used flags in this entire course, since 'how many lines/results are there' is an extremely common question.
05
cut -d',' -f2 extracts a specific column from delimiter-separated text (here, the second column of comma-separated data) — a lighter-weight alternative to awk when you just need a plain column extraction with no filtering or computation involved.
06
Putting it all together: cat access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10 finds the TOP 10 most frequent values in the first column of a log file (commonly IP addresses in a web server access log) — five tiny tools, chained, answering a genuinely useful real-world question.

Code & diagrams

sort-uniq-wc-cut.shmarkdown

The famous 'frequency count' pipeline, built up one piece at a time.

# sort: alphabetical vs numeric
printf "10\n2\n33\n4\n" | sort
# -> 10, 2, 33, 4 (wrong for numbers!)
printf "10\n2\n33\n4\n" | sort -n
# -> 2, 4, 10, 33 (correct)

# uniq only removes ADJACENT duplicates — sort first
printf "b\na\nb\na\n" | uniq
# -> b, a, b, a (no duplicates removed — not adjacent!)
printf "b\na\nb\na\n" | sort | uniq
# -> a, b (now correctly deduplicated)

# uniq -c: count occurrences
printf "a\na\nb\n" | sort | uniq -c
#    2 a
#    1 b

# The famous frequency-count pipeline
cut -d: -f7 /etc/passwd | sort | uniq -c | sort -rn
# (most common login shells on this system, most frequent first)

# wc -l: count lines/results
grep -c "nologin" /etc/passwd   # grep's own built-in count flag
grep "nologin" /etc/passwd | wc -l   # the equivalent piped version

Explain it without notes

Why does uniq on its own often fail to remove duplicates that clearly exist in a file?

Walk through what each stage of awk '{print $1}' file | sort | uniq -c | sort -rn actually contributes, one stage at a time.

Practice

Run the full frequency-count pipeline (sort | uniq -c | sort -rn) against the shells listed in /etc/passwd (via cut -d: -f7) and identify the most common one on your system.

Compare sort versus sort -n on a file containing the numbers 1 through 20 written one per line, and confirm for yourself exactly how the alphabetical ordering gets it wrong.

Trade-offs

↔
These tools are blazingly fast even on files with millions of lines (they're built for exactly this), but they're line/column-oriented — genuinely structured data (real JSON, real CSV with quoted commas inside fields) can trip up a naive cut/awk approach, where a proper parser (jq for JSON, a real CSV library) becomes the more correct tool.

Done when you can

I understand why uniq needs sorted input to work correctly.
I can build the sort | uniq -c | sort -rn frequency-count pipeline from memory.
I know sort -n exists and why plain sort gets numeric ordering wrong.