Reading and Writing Text Files

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

File object

What open() returns: a handle you use to read from or write to a file.

Mode

A short string saying what you plan to do: r read, w write (erases), a append, x create new, b binary.

Encoding

The rule for converting characters to bytes and back. UTF-8 is the standard choice.

Bytes

Raw numbers from 0 to 255. All files are made of bytes; text is bytes plus an encoding.

Buffer

A holding area in memory. Writes collect there and reach the disk later, on flush or close.

Newline

The invisible character that ends a line. Written as \n in Python.

Lazy reading

Reading a little at a time as needed, instead of loading everything.

Mojibake

Garbled text like é that appears when text is decoded with the wrong encoding.

1. 1

   Overwrite by accident

   In the first example change the second mode to "w" and run it. Which lines are left? Change it back to "a" and note the difference. Then try mode "x" on the very first open and run it twice. What does the second run say?

2. 2

   Count bytes

   In the second example change the text to "hello", then to "नमस्ते". Compare the characters and bytes counts. How many bytes does one Hindi letter take in UTF-8?

3. 3

   Find the line

   In the last example make the program print the line number (starting from 1) of the second ERROR line. Hint: use enumerate(f, start=1).

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

with open("missing.txt", encoding="utf-8") as f:
    print(f.read())

with open("old.txt", "wb") as f:
    f.write("café".encode("latin-1"))

with open("old.txt", encoding="utf-8") as f:
    print(f.read())

1. 1

   open(path, mode, encoding=...) returns a file object: a handle you use to read or write. The path is the file's name, such as "notes.txt", optionally with folders in front. A bare name means "in the folder where the program runs". The mode is a short string. "r" reads (the default; the file must exist). "w" writes and erases whatever the file had before, creating it if needed. "a" appends to the end. "x" creates a new file and fails with FileExistsError if it exists, which protects you from overwriting. Adding "+" allows both reading and writing, and "b" means binary (raw bytes) instead of text. The default is text ("t").

2. 2

   Computers store only bytes: numbers from 0 to 255. An encoding is the agreement for turning characters into bytes and back. UTF-8 is the standard that can represent every character in every language and is used almost everywhere on the web. Plain English letters take one byte in UTF-8. Other characters take two to four (é takes two, ₹ takes three). If the writer and the reader use different encodings, you get garbled text (mojibake) or a UnicodeDecodeError.

3. 3

   Always write encoding="utf-8" in open, for both reading and writing. Without it Python uses a default that depends on the operating system and settings (often something else on Windows). The same program then behaves differently on two machines. Python 3.15 makes UTF-8 the default text encoding, which will remove this trap in time. Code that spells out encoding="utf-8" is correct on every version and every machine, so keep writing it.

4. 4

   To read: f.read() returns the whole file as one string, f.readline() returns one line, f.readlines() returns a list of lines. The best way to go through a file is to iterate over the file object: for line in f: (a for loop over the file, Topic 2.4, gives you one line per turn). It reads one line at a time from a buffer, so a 20 GB file uses almost no memory. Each line still ends with "\n". Use line.rstrip("\n") or line.strip() to remove it. This lazy reading is how big AI datasets in .jsonl files are processed.

5. 5

   To write: f.write(text) writes a string (not a number: convert with str() or an f-string) and returns the number of characters written. It does not add a newline, so you write "\n" yourself. f.writelines(list_of_strings) writes many strings, also without newlines. print(text, file=f) writes with a newline added, which is often the neatest way.

6. 6

   Newlines differ between systems: Windows ends lines with \r\n, Linux and macOS with \n. In text mode Python hides this. When reading, every kind of line ending becomes "\n". When writing, "\n" becomes the system's ending. Leave this default alone for ordinary files. When you must keep line endings exactly (a CSV file, Topic 7.7), pass newline="".

7. 7

   And always use with (Topic 7.4) so the file is closed and the data is flushed (pushed out of the buffer onto the disk).

• ↔

  **read() versus iterating.** read() is convenient for small files and when you need the whole text (a regular expression over everything). Iterating scales to any size but gives you only one line at a time.

• ↔

  Strict versus forgiving decoding. The default errors="strict" stops on damaged text so you notice. errors="replace" keeps going and inserts \ufffd characters, which can hide data problems. Choose strict unless losing a few characters is acceptable.

• Done when you can choose between modes r, w, a and x and say what each does to existing content.

• Done when you can read a file by iterating over lines and strip the newline.

• Done when you can explain what an encoding is and why encoding="utf-8" is always written.

• Done when you can explain what happens to \n on reading and writing, and when to use newline="".

• Done when you can handle FileNotFoundError and UnicodeDecodeError sensibly.