Topic 7.8
TOML, YAML, Pickle and Binary Data
In one line
TOML and YAML are friendly formats for configuration files, pickle saves Python objects but must never be opened from an untrusted source, and struct and bytes let you read and write raw binary data.
Think of it like this
Four kinds of paper. A settings card you stick on the fridge (TOML): short, tidy, one fixed layout. A handwritten recipe (YAML): flexible, easy for a person to write, with room for mistakes. A sealed parcel that only your own family should open (pickle): it can hold almost anything, but you must never open a parcel from a stranger. And a sheet of raw numbers that only a machine reads (binary data). This topic is about choosing and using the right one.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- TOML
- A small, strict configuration format with key = value lines and [tables].
- YAML
- A configuration format that uses indentation. Flexible and easy to read, but guesses types.
- safe_load
- The PyYAML function that reads only plain data and refuses to create arbitrary Python objects.
- Pickle
- Python's format for saving objects. Unpickling untrusted data can run code.
- bytes
- An unchangeable sequence of byte values 0-255. bytearray is the changeable version.
- Endianness
- Which end of a multi-byte number is stored first: big-endian (most significant first) or little-endian.
- struct
- A module that converts between Python values and fixed-layout binary data.
- Deserialisation attack
- Sending crafted data that makes a program misbehave when it loads it.
Step by step
01TOML: read a config
This is a TOML file (named app.toml, not Python code). [server] starts a table (a group of settings), and ports is an array (a list). Booleans and numbers are real types, not strings. Names are case sensitive, and strings must be quoted.
title = "Hectal demo"
debug = false
[server]
host = "localhost"
ports = [8000, 8001]
[database]
url = "sqlite:///app.db"
timeout = 2.502Parse TOML in Python
tomllib.loads takes a string. For a file, use tomllib.load with a file opened in binary mode ("rb", which gives raw bytes, Topic 7.5). TOML is defined as UTF-8, so the parser does the decoding itself. The result is a nested dictionary, so config["server"]["ports"] goes into the [server] table, then into ports.
import tomllib
with open("app.toml", "rb") as f:
config = tomllib.load(f)
print(config["server"]["ports"]) # [8000, 8001]
print(config["database"]["timeout"]) # 2.503YAML: safe_load and the type-guessing trap
yaml.safe_load returns dictionaries and lists. Look at what it does to the unquoted values below: no is a boolean, and 3.10 is the number 3.1, losing the zero. Quote anything that must stay as text.
import yaml
text = """
name: demo
country: no
python: 3.10
quoted: "3.10"
steps:
- build
- test
"""
print(yaml.safe_load(text))04Pickle: what it is and the rule
Pickle is a round trip for Python objects: whatever goes in comes out, including sets and tuples, which JSON would change. That is its attraction. The rule is on the right: loading is only safe for data you made.
import pickle
original = {"tags": {"a", "b"}, "point": (1, 2)}
blob = pickle.dumps(original) # bytes
copy = pickle.loads(blob)
print(copy == original) # True05bytes and integers
A bytes value indexes to numbers 0–255. to_bytes and from_bytes convert integers, and you must name the byte order. Changing the order changes the bytes.
n = 1000
print(n.to_bytes(2, "big")) # b'\x03\xe8'
print(n.to_bytes(2, "little")) # b'\xe8\x03'
print(int.from_bytes(b"\x03\xe8", "big")) # 100006struct: a fixed binary layout
A format string describes the fields: < little-endian, I 4-byte unsigned integer, H 2-byte unsigned integer, f 4-byte float. The same format string both packs and unpacks. File formats and network headers are defined this way.
import struct
record = struct.pack("<IHf", 7, 300, 1.5)
print(len(record), record.hex())
print(struct.unpack("<IHf", record)) # (7, 300, 1.5)Try it yourself
- 1
Break the type
In the TOML example change
timeout = 2.5totimeout = "2.5"and printtype(...). Then trytimeout = 2and see how an integer differs from a float. - 2
Quote the trap
In the YAML example write
country: "no"andpython: "3.10". Compare the two printed results. Then trycountry: NOandcountry: n. - 3
Flip the order
In the bytes example change the struct format from
<IHfto>IHfand compare the hex output. Which bytes moved?
Code & diagrams
Note the types: false is a bool, 2.5 a float, the array a list. Table [server] becomes a nested dictionary.
Expected output
{'title': 'Hectal demo', 'debug': False, 'server': {'host': 'localhost', 'ports': [8000, 8001]}, 'database': {'url': 'sqlite:///app.db', 'timeout': 2.5}}
8001 floatcountry: no becomes False, and python: 3.10 becomes 3.1. The quoted value stays a string.
Expected output
{'name': 'demo', 'country': False, 'python': 3.1, 'quoted': '3.10', 'steps': ['build', 'test']}
name: demo
country: false
python: 3.1
quoted: '3.10'
steps:
- build
- testSets and tuples survive, unlike with JSON. This is safe only because we made the bytes ourselves a moment ago.
Expected output
pickle same? True tuple
json changes: listExpected output
b'\x03\xe8' b'\xe8\x03'
1000
10 070000002c010000c03f
(7, 300, 1.5)
b'Abc' [104, 105]# pyproject.toml is TOML: it describes a project, not a program.
[project]
name = "hectal-demo"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = ["httpx", "pydantic>=2"]Real binary formats (images, audio, network packets) start with a fixed header: a magic number that says what the file is, a version, then sizes. `Struct` describes the layout once, `unpack_from` reads at an offset, and the checks reject files that are too short or of the wrong kind. Money is stored in paise as whole numbers, never as floats.
Expected output
header size: 12
30 4843544c0101000300000012
(1, 1, [(1, 2000), (2, 3500), (3, 1250)])
rejected: file too short for a header
rejected: not a Hectal file: b'XXXX'
struct.error: unpack requires a buffer of 6 bytes
8 5Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Open a TOML file in text mode
Pass a text-mode file to tomllib.load.
import tomllib
with open("app.toml", "w", encoding="utf-8") as f:
f.write('name = "demo"\n')
with open("app.toml", encoding="utf-8") as f:
print(tomllib.load(f))Break #2
safe_load refuses Python objects
Feed safe_load a document that asks for a Python-specific type.
import yaml
print(yaml.safe_load("point: !!python/tuple [1, 2]"))Myth vs fact
Myth
Pickle is safe if the file looks harmless.
Fact
The danger is in how the file is loaded, not in what it looks like. A pickle can contain instructions that run during loads. Only unpickle data you made yourself.
Myth
yaml.load and yaml.safe_load are the same.
Fact
Not at all. safe_load creates plain data only. Using a full loader on untrusted input has led to real security holes. Always use safe_load.
Myth
YAML is better than TOML because it is more powerful.
Fact
More power means more surprises: implicit types, indentation errors, anchors. TOML is smaller and stricter, which is a feature for settings files.
Myth
Binary data is just text in a strange form.
Fact
Bytes have no meaning until you know the layout or encoding. Text is bytes plus an agreed encoding; other binary data has its own layout (images, audio, tensors).
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Defensive rules for deserialisation: parse data formats (JSON, TOML, YAML
safe_load) rather than object formats (pickle), validate the result against an expected schema (a written description of the fields and types you accept; Pydantic, covered later in the course, does this), and put limits on file size and nesting depth. - ▸
If you must store pickles internally, sign them with an HMAC (a secret-key checksum that proves nobody changed the data) and verify the signature before loading, or keep them in a place only your service can write. This protects against tampering, but it does not make loading pickles from strangers safe.
- ▸
tomllibis deliberately read-only. Writing is rare in tools (config is written by humans), and the standard library did not want to take on formatting and comment preservation. Usetomli-wfor simple output, ortomlkitto edit a file and keep its comments. - ▸
Machine-learning model files have been a real route for malicious pickles. Prefer formats that store only numbers (
safetensors), and load weights from sources you trust. When a library offersweights_only=True-style options (PyTorch'storch.loadhas one), turn them on. - ▸
Why pickle is dangerous, precisely: an object can define
__reduce__, which tells pickle "to rebuild me, call this function with these arguments". The loader obeys, and the function can be any importable callable. Subclassingpickle.Unpicklerand overridingfind_classto allow only a short list of safe classes narrows the risk, but the Python documentation itself recommends a data-only format instead. Pickles also break when you rename a class or move its module, and they are not guaranteed to load across Python versions or to work for open files, sockets or locks. - ▸
In
struct, the format prefix matters. Without one (or with@) Python uses the platform's native byte order, native sizes and alignment padding, sostruct.calcsize("@BI")is 8 where"<BI"is 5. For file formats and network data always start the format with<,>or!(network order, big-endian) so the layout is identical on every machine. For fixed-size arrays of numbers,array.array,memoryviewand NumPy (Topic 13.9) are faster than callingstructonce per value. - ▸
PyYAML's
yaml.loadhas needed an explicitLoader=argument since version 6, precisely because the old default could construct arbitrary objects. Evensafe_loadfollows YAML 1.1 rules:0123is read as the octal number 83, and2024-01-05becomes adate. A document with deeply nested aliases ("billion laughs") can also use a lot of memory, so limit the size of YAML you accept from outside.
Remember this
- 1
TOML (Tom's Obvious Minimal Language) is a configuration format with a clear, small grammar:
key = valuelines,[tables]for groups, strings in quotes, real numbers, booleans (true/false), dates, and arrays. Python's ownpyproject.toml(Topic 6.7) is TOML. Python 3.11 added thetomllibmodule to read it:tomllib.loads(text)parses a string.tomllib.load(f)parses a file opened in binary mode ("rb", Topic 7.5). The result is ordinary dictionaries, lists, strings, numbers and booleans.tomllibcan only read. To write TOML you need a third-party library (one you install withpip, Topic 6.5) such astomli-w. - 2
YAML is another configuration format. It uses indentation instead of quotes and brackets, which makes it pleasant to write by hand. Kubernetes, Docker Compose and GitHub Actions files (see the Docker and CI/CD courses) are YAML. Python reads it with the third-party package PyYAML (
import yaml). The one rule: **useyaml.safe_loadandyaml.safe_dump**. Never use plainyaml.loadon data you did not write yourself, because YAML has features that can create arbitrary Python objects. Also know that YAML guesses types. Unquotedno,offandyesbecome booleans, and3.10becomes the float3.1. Quote values that must stay text. - 3
Pickle is Python's own format for saving almost any object (lists, sets, your own classes) exactly as it is.
pickle.dumps(obj)returns bytes andpickle.loads(data)rebuilds the object. It is convenient and fast, but it has a serious danger. A pickle is not just data. It is a list of instructions for rebuilding objects, and those instructions can call any function. Unpickling data from an untrusted source can run code of the sender's choice on your machine. The rule has no exceptions: only unpickle data you created yourself and stored where nobody else can change it. - 4
What to use instead of pickle: JSON (Topic 7.7) for data you exchange or store for others, CSV, Parquet (a compact format for big tables, Phase 14) or a database for tables. For machine learning models there are specialised safe formats, such as
safetensors, which exists because pickled model files were an attack route. Many machine-learning libraries save models with pickle internally. Treat a downloaded model file from an unknown place with the same caution as a downloaded program. - 5
Binary data is bytes that are not text.
bytesis an unchangeable sequence of numbers 0–255 (writtenb"\x00\x01", where\x01means the byte with value 1 in hexadecimal).bytearrayis the changeable version.int.to_bytes(length, "big")andint.from_bytes(data, "big")convert whole numbers. A number such as 1000 needs two bytes, so the writer and reader must agree which comes first. That order is called endianness (big or little). - 6
The
structmodule packs and unpacks fixed layouts.struct.pack("<IH", 1, 2)produces a 4-byte little-endian unsigned integer followed by a 2-byte one. Think of it as filling in a form with boxes of fixed size. - 7
Which format when? Configuration written by people: TOML for tools and projects, YAML where the ecosystem expects it. Data exchanged between programs: JSON. Tables: CSV for small, Parquet or a database for big. Python-only caches that you fully control: pickle at most. Network protocols and file headers:
struct. In AI projects expect to read YAML for experiment and pipeline settings, TOML for project metadata, JSON for API messages and bytes for images, audio and model weights.
Explain it without notes
Why is it dangerous to unpickle data from an untrusted source? What should you use instead?
What is the difference between yaml.load and yaml.safe_load, and what typing surprises can YAML give?
When would you choose TOML over YAML or JSON for configuration?
What is endianness, and where does it matter?
Practice
Parse the TOML text [app]\nname = "demo"\nretries = 3 with tomllib.loads. Print the value of retries plus 1 (it should print 4). Hint: the value sits inside the app table.
Use yaml.safe_load on a YAML list of three numbers written one per line, each starting with - (for example - 10). Print their sum. Hint: the result is a normal Python list.
Pack the numbers 1, 2, 3 as three little-endian 2-byte unsigned integers with struct.pack("<HHH", ...). Print the .hex() of the result, then unpack it back and print the tuple.
Trade-offs
- ↔
Human-friendly versus predictable. YAML is easiest to write and easiest to get wrong (implicit types, indentation). TOML trades some flexibility for predictable parsing.
- ↔
Convenience versus safety. Pickle stores any object with no code, but it is unsafe across trust boundaries and tied to Python versions and your own class definitions. JSON needs more work but is safe, portable and readable.
Done when you can
Done when you can read a TOML file with
tomllib.loadin binary mode.Done when you can load YAML with
safe_loadand explain why plainloadis avoided.Done when you can explain, in your own words, why unpickling untrusted data is dangerous and what to use instead.
Done when you can convert integers to and from bytes and name the byte order.
Done when you can pack and unpack a small record with
struct.