PHASE 3Beginner ~26 min· topic 7 of 7

Topic 3.7

Real Text Work: Cleaning and Parsing

In one line

Real text is messy: stray spaces, mixed case, punctuation, odd line endings. This topic puts the whole phase together: split text into lines and words, clean it, count words, wrap it with textwrap, use the string constants, and parse a small log file.

Think of it like this

Sorting a pile of handwritten shop receipts. Some are smudged, some use capitals, some have extra spaces. Before you can add anything up, you tidy each one into the same shape, then read it. Text work always has two stages: clean (make the text uniform) and parse (pull out the parts you care about). Skipping the first stage is the usual reason a parser breaks on the 50th line.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Cleaning: Making text uniform: trimming, fixing case, removing unwanted characters.
Parsing: Reading text of a known shape and pulling out its parts as values.
Dictionary: A notebook of key-to-value pairs written with {}. Phase 4 covers it in full.
Triple-quoted string: A string written between """ marks that can span many lines.
Word wrap: Breaking long text into lines of at most a chosen width.
Log: A file or stream of timestamped messages that a program writes about what it is doing.
Delimiter: The character that separates fields, such as a comma or =.
Tokenizing: Splitting text into small pieces (tokens), such as words.

Step by step

01Start with lines

A triple-quoted string stands in for a file here. splitlines() gives one item per line and no newline characters. A common detail: if the text ends with a newline, splitlines() does not produce an extra empty line at the end, unlike split("\n").

main.pywhole filepython

text = """first line
second line
"""

print(text.splitlines())      # ['first line', 'second line']
print(text.split("\n"))       # ['first line', 'second line', '']

02Clean the text

Normalise before you count. Make everything lower case so "The" and "the" are the same word, delete punctuation, then split. maketrans("", "", ".,!?") builds a table that says "map nothing to anything, delete these characters", and translate applies it in one pass.

main.pywhole filepython

text = "The cat sat on the mat.  The dog sat!"

cleaned = text.lower().translate(str.maketrans("", "", ".,!?"))
print(cleaned)            # the cat sat on the mat  the dog sat
print(cleaned.split())    # extra spaces vanish: split() ignores them

03Count the words

The dictionary starts empty. For each word, counts.get(w, 0) returns the current count (or 0 for a new word) and we store one more. Printing in sorted order makes the output the same every time.

main.pywhole filepython

words = "the cat sat on the mat".split()

counts = {}
for w in words:
    counts[w] = counts.get(w, 0) + 1

for w in sorted(counts):
    print(w, counts[w])      # cat 1, mat 1, on 1, sat 1, the 2

04Wrap, shorten and indent with textwrap

textwrap.fill(text, width=30) breaks a long paragraph into lines of at most 30 characters, at word boundaries. shorten collapses whitespace and cuts at a word boundary with a placeholder, perfect for previews. dedent removes the common indentation of a block, which keeps long messages tidy inside indented code.

main.pywhole filepython

import textwrap

para = "Python is a programming language that lets you work quickly."
print(textwrap.fill(para, width=30))
print(textwrap.shorten(para, width=30, placeholder="..."))

05Split a log line into parts

split(None, 3) means "split on whitespace, at most 3 times", so the first three pieces are date, time and level, and everything else stays together as the fourth. Unpacking into four names only works when the line really has at least that many pieces. Real parsers guard against short lines.

main.pywhole filepython

line = "2026-10-09 10:15:07 ERROR user=ravi action=upload"
date, time, level, rest = line.split(None, 3)
print(level)    # ERROR
print(rest)     # user=ravi action=upload

06Parse the key=value fields

rest.split() gives ['user=ravi', 'action=upload']. Each pair goes through partition("="). The underscore _ is a normal variable name that programmers use for "a value I do not need" (here the separator).

main.pywhole filepython

rest = "user=ravi action=upload size=2048"
for pair in rest.split():
    key, _, value = pair.partition("=")
    print(key, "->", value)

07Put it together and expect the unexpected

The complete log parser appears in the examples. Real log files contain blank lines, truncated lines and fields in a different order. Topic 7.1 (exceptions) and Topic 3.6 (regular expressions) give you the tools to deal with those, and Topic 7.9 shows how to write logs properly in the first place.

Try it yourself

1
Add a line with a problem
Add the line "bad line" to the log in the second example and run it. The unpacking date, time, level, rest = ... crashes with a ValueError. Think about how a parser should handle a line it cannot read. Topic 7.1 shows how.
2
Count your own text
Replace the text in the first example with a paragraph from anywhere. Which word is the most common? Does it change if you stop lowercasing?
3
Find the top word
After the counting loop, loop through counts once more and keep the word with the largest count in two variables, best_word and best_count. Start with best_word = "" and best_count = 0. When a word's count is bigger than best_count, update both variables.

Code & diagrams

Clean and count words New tab

A dictionary keeps one count per word. Topic 4.10 shows the shortcut `Counter`.

Expected output

['the', 'cat', 'sat', 'on', 'the', 'mat', 'the', 'dog', 'sat', 'on', 'the', 'log']
total words: 12
cat  1
dog  1
log  1
mat  1
on   2
sat  2
the  4
distinct words: 7

A small log parser New tab

Expected output

10:15:01 INFO  user=asha, action=login
10:15:07 ERROR user=ravi, action=upload, size=2048
  upload size: 2 KB
10:16:30 WARN  user=asha, action=retry
lines: 3 errors: 1

textwrap and the string module New tab

Expected output

Python is a programming
language that lets you work
quickly and integrate systems
effectively.

Python is a programming language that...
> a
> b
'\nindented\n  more\n'
abcdefghijklmnopqrstuvwxyz
ABCDE 0123456789
!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~
True

Cleaning and parsing pipelinediagram

Rendering diagram…

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Unpack a line with too few parts

Parse a log line that has only three words.

main.pywhole filepython

line = "2026-10-09 10:15:07 ERROR"
date, time, level, rest = line.split(None, 3)

terminal

$ python main.py

── what you'll see ──

Traceback (most recent call last):

File "main.py", line 2, in <module>

date, time, level, rest = line.split(None, 3)

^^^^^^^^^^^^^^^^^^^^^^^

ValueError: not enough values to unpack (expected 4, got 3)

Myth vs fact

Myth

Cleaning text is the boring part you can skip.

Fact

Most real-world text bugs and bad data come from skipping it. A word counter that treats The and the as different words is wrong in a way that no test of the counting loop would catch.

Myth

split("\n") and splitlines() are the same.

Fact

splitlines() handles \r\n and other line endings and does not produce a final empty item for a trailing newline.

Myth

A word count equals a token count.

Fact

Language models count sub-word tokens. The same text can have very different word and token counts.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Real files reach your program as bytes. Decode with an explicit encoding first (Topic 3.4, Topic 7.5), then clean. For large files, iterate line by line instead of reading everything, so memory stays flat.
▸
splitlines() also splits on \x0b, \x0c, \x1c to \x1e, \x85, \u2028 and \u2029, not only on \n and \r. That is usually what you want for text, but a log line that legitimately contains a form feed will be cut in two; if records are defined by \n alone, iterate the file object or use split("\n") after normalising \r\n. Pass keepends=True when you must be able to rebuild the original text exactly.
▸
str.translate with a table built by str.maketrans is one pass over the text in C and is much faster than chaining many replace calls for character-level cleaning. For Unicode-aware cleaning, combine unicodedata.normalize (Topic 3.4) with casefold() first.
▸
For text pipelines in AI work, keep the original text and store cleaned versions beside it, with offsets if you need to point back to the source. Chunking and citation (Topic 16.9) depend on being able to find the original span again.

Remember this

1
Lines first. In real programs text arrives in one big string (read from a file, Topic 7.5, or a network reply). text.splitlines() cuts it into lines and handles \n, \r\n (Windows) and old Mac \r endings; text.split("\n") does not handle \r\n and leaves stray \r characters. Here we use a multi-line string, written with triple quotes, in place of a file.
2
Cleaning toolkit. strip() for edges. " ".join(text.split()) collapses any runs of spaces, tabs and newlines into single spaces. lower() or casefold() to ignore case. str.translate(str.maketrans("", "", ".,!?")) deletes every listed character in one pass (the walkthrough explains it). replace for specific fixes. Do the cleaning steps before counting or comparing, and keep the original text if you may need it later.
3
Counting words. Split the cleaned text with split() and count each word. The easiest way to keep a count per word is a dictionary: a notebook that maps a key (the word) to a value (its count), written {}. counts[w] = counts.get(w, 0) + 1 means "look up w, use 0 if it is new, add one". Phase 4 teaches dictionaries properly (Topic 4.4), and the Counter class in Topic 4.10 does this job in one line. sorted(counts) gives the keys in alphabetical order.
4
Standard-library helpers. The standard library is the set of tools that come with Python. The string module holds ready-made constants: string.ascii_lowercase, ascii_uppercase, digits, punctuation and whitespace. The textwrap module reshapes paragraphs: fill wraps text to a width, shorten cuts to a width with a placeholder (...), indent adds a prefix to every line and dedent removes common leading spaces from a block (handy for triple-quoted text inside indented code).
5
A few older padding tools exist too: str.ljust, rjust, center and zfill pad strings, though format specs (Topic 3.3) usually do the same job.
6
Parsing a log. A log line has a fixed shape: date time LEVEL key=value key=value. line.split(None, 3) splits on whitespace but stops after three cuts, leaving the rest of the line as the fourth piece. Then partition("=") splits each key=value pair safely. Always handle the case where a line does not have the shape you expect, because real files have blank lines and corrupted lines. Topic 3.6's regular expressions are the next step up for irregular formats.
7
The AI-era link. Cleaning and splitting text is the first step of nearly every language-model pipeline. Before documents are turned into numbers for search, they are cleaned, split into chunks of limited size, and counted (Topic 16.9). Splitting on whitespace is a crude "tokenizer"; real models split into sub-word tokens (Topic 16.4), so a word count is never a token count.

Explain it without notes

Why should you clean text before counting or comparing it?

What is the difference between splitlines() and split("\n")?

How does line.split(None, 3) differ from line.split(), and why is it useful for logs?

When would you use partition instead of split to read key=value text?

Practice

Count the vowels in "Education is the key".

Find the longest word in "the quick brown foxes jumped" with a loop and two variables.

Wrap "one two three four five six seven" to lines of at most 15 characters.

Trade-offs

↔
Aggressive cleaning against fidelity. Removing punctuation and case makes counting easy but loses information ("US" and "us"). Clean a copy and keep the original.
↔
Hand-written parsers against regular expressions. split and partition are clear for fixed shapes. Once the format has optional parts or varying order, a regular expression with named groups is usually shorter and safer.

Done when you can

Done when you can turn a block of text into lines, words and fields.
Done when you can clean text with strip, lower, split/join and translate.
Done when you can count words with a dictionary and print the result in a stable order.
Done when you can wrap and shorten text with textwrap.
Done when you can parse a date time LEVEL key=value log line safely.

Command Palette

Words you'll meet

Step by step

01Start with lines

02Clean the text

03Count the words

04Wrap, shorten and indent with textwrap

05Split a log line into parts

06Parse the key=value fields

07Put it together and expect the unexpected

Try it yourself

Add a line with a problem

Count your own text

Find the top word

Code & diagrams

Break it on purpose

Unpack a line with too few parts

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can