PHASE 3Beginner ~28 min· topic 4 of 7

Topic 3.4

Unicode, Bytes and Encodings

In one line

A Python str is a sequence of Unicode code points (numbers that stand for characters), while bytes is a sequence of raw numbers from 0 to 255. Turning text into bytes is encoding and the reverse is decoding; almost every mysterious bug with accents, emoji or Ã© garbage is an encoding mismatch.

Think of it like this

Morse code. Letters are not sent down a telegraph wire; dots and dashes are. Two operators must agree on the code book, or the message turns to nonsense. Computers store and send bytes, not letters. An encoding is the code book that says which bytes mean which characters. Python keeps text as str (people-readable characters) inside your program and uses bytes only at the edges: files, networks, databases.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Character: A letter, digit, symbol or emoji, as a person thinks of it.
Unicode: The worldwide list that gives each character a number.
Code point: The number Unicode gives a character, such as 65 for A. Written U+0041.
Byte: A number from 0 to 255. The unit in which files and networks store data.
Encoding: A code book that turns characters into bytes and back. UTF-8 is the most common.
Encode and decode: encode: str to bytes. decode: bytes to str.
ASCII: The old 128-character English set. UTF-8 keeps its first 128 characters as the same single bytes.
Mojibake: Garbled text such as cafÃ©, caused by decoding bytes with the wrong encoding.
Normalisation: Converting text to one standard form of accents and combined characters so that equal-looking text compares equal.

Step by step

01Characters are numbers underneath

ord gives the number for a character and chr gives the character for a number. Every character on Earth has such a number. The escape \u20b9 in a string means "the character with code point 20b9 in hexadecimal", which is 8377: the rupee sign. (Hexadecimal is base 16; Topic 1.2.)

main.pywhole filepython

print(ord("A"))          # 65
print(chr(65))           # A
print(ord("₹"))          # 8377
print(chr(8377))         # ₹
print("\u20b9")          # ₹

02UTF-8 uses 1 to 4 bytes per character

A computer cannot store the number 8377 as one byte (a byte stops at 255), so encodings define how to spread big numbers over bytes. UTF-8 is designed so that the commonest letters need one byte, and rarer ones need more. That keeps English text small and still covers every language.

UTF-8 uses 1 to 4 bytes per characterdiagram

Rendering diagram…

03encode and decode: str to bytes and back

encode produces a bytes object. When Python prints bytes, it writes printable ASCII as letters and everything else as \xNN escapes, so b'caf\xc3\xa9' is the bytes of café. decode reverses it. Indexing bytes gives integers.

main.pywhole filepython

data = "café".encode("utf-8")
print(data)                 # b'caf\xc3\xa9'
print(len("café"), len(data))   # 4 5
print(data.decode("utf-8"))     # café
print(data[0])                  # 99

encode and decode: str to bytes and backdiagram

Rendering diagram…

04The wrong code book gives garbage

Bytes carry no label saying which encoding wrote them. If you read UTF-8 bytes as Latin-1 (an old 1-byte-per-character encoding), every byte becomes a character, and the two bytes of é turn into two odd characters, Ã©. If you ever see Ã©, â€™ or Ã¼ in text, a UTF-8 file was read with the wrong encoding.

The cure is to decode with the encoding that wrote the bytes. Where you cannot know, UTF-8 is the best first guess.

main.pywhole filepython

bad = "café".encode("utf-8").decode("latin-1")
print(bad)                                    # cafÃ©
print(bad.encode("latin-1").decode("utf-8"))  # café  (repaired)

05When the bytes are not valid

Some byte sequences are simply not valid UTF-8. With the default errors="strict", decoding stops with UnicodeDecodeError and tells you which byte and where. For data you must keep going with (a log line with garbage in it), choose errors="replace", which substitutes � (U+FFFD, the replacement character). Use ignore rarely: it silently loses data.

Encoding can fail too: asking for ASCII on ₹ raises UnicodeEncodeError unless you pass errors="replace" or "ignore".

main.pywhole filepython

print(ascii(b"\xff\xfeabc".decode("utf-8", errors="replace")))
# '\ufffd\ufffdabc'
print("café ₹5".encode("ascii", errors="replace"))   # b'caf? ?5'

06One character, many code points

The accented é can be stored as one code point or as e followed by a combining accent mark. Both display as é, but they have different lengths and compare as not equal. This breaks searches, dictionary lookups and duplicate detection. The standard fix is to normalise both sides to the same form, usually "NFC" (compose), before comparing.

The line import unicodedata brings in a ready-made toolbox from Python's standard library (Phase 6 explains import). unicodedata.normalize is the tool for this job.

Emoji add a second layer: a flag is two regional-indicator code points; a family emoji joins three people with invisible "zero-width joiner" code points. So len is not the number of things you see.

main.pywhole filepython

import unicodedata

a = "é"            # one code point
b = "e\u0301"      # e + combining accent
print(len(a), len(b), a == b)                 # 1 2 False

nfc = unicodedata.normalize
print(nfc("NFC", a) == nfc("NFC", b))         # True

07Where bytes appear in real work

You meet bytes whenever data leaves your program: reading a file in binary mode, an HTTP response body, a hash digest, an image, a compressed block. The rule that keeps programs sane is the Unicode sandwich: decode bytes to str as early as possible, do all your work on str, and encode back to bytes as late as possible.

A note for AI work: model text limits are counted in tokens, which are pieces of text (not characters and not bytes), and token counts for the same text differ between models and between languages (Topic 16.4). Emoji and non-English scripts commonly cost more tokens than the same number of English letters.

Try it yourself

1
Count characters and bytes
Add "नमस्ते" (Hindi "hello") to the first example. Predict how many bytes each code point takes before you run it. Then compare len of the text with len of its encoded bytes.
2
Create mojibake on purpose, then repair it
Encode any word with an accent as UTF-8 and decode it as "latin-1". Then reverse the two steps. This is exactly how you diagnose files that were saved with one encoding and opened with another.
3
Two e-acutes
In the third example, change "NFC" to "NFD" and look at the lengths. NFD splits characters into base plus accent; NFC combines them. Either form works for comparing, as long as both sides use the same one.

Code & diagrams

Code points and UTF-8 byte counts New tab

One character, one code point, 1 to 4 bytes. Add your own character to the list.

Expected output

A code point 65 0x41 bytes 1 b'A'
é code point 233 0xe9 bytes 2 b'\xc3\xa9'
₹ code point 8377 0x20b9 bytes 3 b'\xe2\x82\xb9'
日 code point 26085 0x65e5 bytes 3 b'\xe6\x97\xa5'
😀 code point 128512 0x1f600 bytes 4 b'\xf0\x9f\x98\x80'
A ₹ 😀
₹ 😀

str, bytes and the wrong code book New tab

Expected output

7 10
b'caf\xc3\xa9 \xe2\x82\xb95'
café ₹5
wrong codec: cafÃ©
'\ufffd\ufffdabc'
b'caf? ?5'
b'caf 5'
<class 'str'> <class 'bytes'>
99 b'caf'

Normalisation and lookalike characters New tab

Expected output

é é 1 2 False
1 1 True
LATIN SMALL LETTER E WITH ACUTE | COMBINING ACUTE ACCENT
2 5 1
STRASSE strasse

The Unicode sandwichdiagram

Rendering diagram…

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Decode bytes that are not UTF-8

Run data = b"\xff\xfe" and print(data.decode("utf-8")).

main.pywhole filepython

data = b"\xff\xfe"
print(data.decode("utf-8"))

terminal

$ python main.py

── what you'll see ──

Traceback (most recent call last):

File "main.py", line 2, in <module>

print(data.decode("utf-8"))

~~~~~~~~~~~^^^^^^^^^

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

Break #2

Mix str and bytes

Run print("abc" + b"def").

main.pywhole filepython

print("abc" + b"def")

terminal

$ python main.py

── what you'll see ──

Traceback (most recent call last):

File "main.py", line 1, in <module>

print("abc" + b"def")

~~~~~~^~~~~~~~

TypeError: can only concatenate str (not "bytes") to str

Myth vs fact

Myth

Unicode is an encoding.

Fact

Unicode is the list of characters and their numbers (code points). UTF-8, UTF-16 and UTF-32 are encodings that turn those numbers into bytes.

Myth

One character is one byte.

Fact

Only in ASCII. In UTF-8 a character takes 1 to 4 bytes, so len(text) and len(text.encode()) usually differ.

Myth

len(text) is the number of characters a person sees.

Fact

It counts code points. Combining accents, flags and family emoji use several code points for one visible symbol.

Myth

Two strings that look identical are equal.

Fact

They may use different code points (composed and decomposed accents). Normalise both with the same form before comparing.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
CPython stores a str with a fixed width per string chosen by its widest character (PEP 393): 1, 2 or 4 bytes per code point. A string of plain ASCII or Latin-1 is compact, but adding one emoji makes the whole string use 4 bytes per character. It is invisible in your code, and it explains odd memory numbers from sys.getsizeof.
▸
Encodings fail in production in a few predictable places: opening files without encoding=, reading HTTP bodies without checking the charset, and mixing data from a Windows spreadsheet export (often a Windows code page or UTF-8 with a BOM, which "utf-8-sig" handles) with UTF-8 data. Make encoding explicit at every boundary.
▸
Unicode also has security consequences: lookalike characters (a Cyrillic а instead of a Latin a) enable spoofed names and links. Normalising input (NFKC is the usual choice for usernames) and restricting identifiers to a known alphabet are the standard defences. Normalisation does not make a Cyrillic а equal to a Latin a; they stay different characters, so only an allow-list of scripts, or a confusables check, catches that. For identifiers in Python itself, the language NFKC-normalises names.
▸
Know the four normal forms. NFC composes (e + accent becomes é) and is the right default for storing and comparing text. NFD decomposes. NFKC and NFKD also fold "compatibility" characters, so the ligature ﬁ becomes fi and the full-width Ａ becomes A; this loses information, which is acceptable for search keys and usernames but not for stored text. Python has no built-in grapheme-cluster iterator: for "user-visible characters" (truncating a tweet-like field without cutting an emoji) use the third-party regex package (\X) or grapheme. Also remember that JSON, HTTP and the web are UTF-8, but a lone surrogate (Python produces these with surrogateescape for file names that are not valid UTF-8, mostly on Linux) can make .encode("utf-8") raise unless you pass errors="surrogateescape" or "replace".

Remember this

1
Unicode is a giant list that gives every character in every writing system a unique number called a code point: A is 65, é is 233, ₹ is 8377, 😀 is 128512. Code points are written U+0041 or in Python as "\u0041" (four hex digits) and "\U0001F600" (eight hex digits). Hex digits are base 16: 0 to 9 and then a to f (Topic 1.2). ord("A") gives the code point of a character and chr(65) goes back. A Python str is exactly a sequence of these code points, so len(text) counts code points.
2
UTF-8 is the encoding almost everything uses today: web pages, JSON, source code, most files. It stores each code point in 1 to 4 bytes: plain English letters take 1 byte (identical to old ASCII), é takes 2, ₹ and most Asian characters take 3, emoji take 4. So a string's number of characters and number of bytes can differ: "café ₹5" has 7 characters but 10 bytes in UTF-8.
3
text.encode("utf-8") turns a str into bytes; data.decode("utf-8") turns bytes back into a str. A bytes literal (bytes typed straight into your code) is written with a b before the quotes (b"abc"), may only contain ASCII characters directly (use \xNN for other values), and indexing it gives integers: b"abc"[0] is 97, not a letter. str and bytes are different types that never mix: "abc" + b"def" is a TypeError.
4
Decoding goes wrong when you use a different encoding than the one that wrote the bytes. If bytes that are not valid in the chosen encoding appear, Python raises UnicodeDecodeError. If they happen to be valid but mean something else, you get mojibake: UTF-8 bytes of café read as Latin-1 (an old encoding that uses one byte per character) turn into cafÃ©. The errors= argument chooses what to do with bad data: "strict" (the default, raises), "replace" (puts � or ?), "ignore" (drops it).
5
A code point is still not always what a person sees as one character. An é can be one code point (U+00E9) or two (e plus a combining accent U+0301); they look identical but are not ==. Normalisation (unicodedata.normalize("NFC", s)) converts text to one standard form so comparisons work. Flags and family emoji are built from several code points joined together: len("🇮🇳") is 2 and the family emoji 👨‍👩‍👧 has len 5. A "character as the user sees it" is called a grapheme cluster, and the standard library has no function for it.
6
Python source files are UTF-8, and sys.getdefaultencoding() is always "utf-8", but the default encoding used for opening text files depends on the operating system and has been a classic source of bugs on Windows. The safe habit, which Topic 7.5 turns into a rule, is to always write open(path, encoding="utf-8"). (Python 3.15 makes UTF-8 the default for text files.)

Explain it without notes

What is the difference between a character, a code point and a byte?

Why can len(text) differ from len(text.encode("utf-8"))?

Why can two strings that print the same be unequal, and how do you fix it?

Practice

Print how many characters and how many UTF-8 bytes are in the text "Hectal ₹".

Make mojibake by decoding the UTF-8 bytes of "café" as Latin-1, print it, then repair it.

Print the letter that is 3 places after "a" using ord and chr, and the letter 25 places before "z".

Trade-offs

↔
UTF-8 against fixed-width encodings. UTF-8 is compact for English and universal, but you cannot jump to the n-th character in the bytes without decoding. Python's str pays extra memory to give you instant indexing by code point.
↔
Strict against lenient decoding. strict fails loudly and protects data quality; replace keeps a pipeline running but hides corruption. Pick one on purpose and log when you use the lenient mode.

Done when you can

Done when you can explain code point, byte and encoding in your own words.
Done when you can predict the byte length of an ASCII, accented, rupee and emoji character in UTF-8.
Done when you can encode and decode between str and bytes, and explain why they never mix.
Done when you can recognise mojibake and say how it happened.
Done when you can explain why len is not what the reader sees, and normalise before comparing.

Command Palette

Words you'll meet

Step by step

01Characters are numbers underneath

02UTF-8 uses 1 to 4 bytes per character

03encode and decode: str to bytes and back

04The wrong code book gives garbage

05When the bytes are not valid

06One character, many code points

07Where bytes appear in real work

Try it yourself

Count characters and bytes

Create mojibake on purpose, then repair it

Two e-acutes

Code & diagrams

Break it on purpose

Decode bytes that are not UTF-8

Mix str and bytes

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can