Topic 11.7
Coverage, TDD and Property-Based Testing
In one line
Coverage shows which lines your tests never ran, test-driven development (red, green, refactor) writes the test before the code, and property-based testing lets a tool invent hundreds of inputs to break a rule you state. Together they find bugs your hand-picked examples miss.
Think of it like this
You read a textbook and highlight every page you have read. The highlighting shows which pages you skipped. It does not show whether you understood the pages you read. Code coverage is that highlighter for your tests: it marks the lines of your program that ran while the tests ran, and shows the lines that never ran. Lines no test touches cannot be protected by any test.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Code coverage
- The share of your program's lines (or branches) that ran during the tests.
- Branch coverage
- Coverage that also tracks whether each
iftook both its true and false paths. - Test-driven development (TDD)
- Writing a failing test first, then the code to pass it, then cleaning up.
- Red, green, refactor
- The three steps of TDD: fail, pass, tidy.
- Regression test
- A test written to make sure a fixed bug never returns.
- Example-based test
- A test with fixed inputs and a fixed expected output.
- Property
- A rule that must be true for every valid input.
- Property-based test
- A test that states a property and lets a tool generate many inputs to check it.
- Hypothesis
- The Python library for property-based testing.
- Shrinking
- Reducing a failing input to the smallest one that still fails.
Step by step
01Measure coverage
Here shop.py has a function that raises an error for a bad percentage, but the test file only tests the normal path. coverage run -m pytest runs the tests under measurement; coverage report -m prints the table. The Missing column says line 7 never ran, and line 7 is the raise.
02Use the number to find missing tests
The percentage is the least useful part. The useful part is the Missing list: it tells you exactly which behaviour has no test. Line 7 is the error for a bad percent. Write that test (pytest.raises(ValueError) from Topic 11.2) and the line turns green.
03pytest-cov, branches and a threshold
With pytest-cov one command does everything. --cov-branch tracks branches too, so the if on line 6 shows as a partial branch (BrPart): its true side ran but not the other. --cov-fail-under=90 turns the number into a gate: pytest prints an error and exits non-zero when coverage is lower. Output trimmed.
04How a coverage tool sees your code
The runnable example installs a tracer (a function Python calls at each step) with sys.settrace. Python calls it for every line that runs inside the function. After the calls, the example compares the executed lines with all executable lines (which dis.findlinestarts lists). Lines in the second set but not the first are the "missing" ones. Real tools do the same with more care and speed.
05Coverage is not correctness
The second runnable example has a test that touches every line of average and passes, so coverage is 100 %. Yet average([]) divides by zero. The test never asked the hard question. Coverage tells you where you did not look; it cannot tell you that looking was enough.
06Red, green, refactor
In the TDD example, the test table comes first. The first slugify is the simplest idea and fails two cases (red). The second uses a regular expression (Topic 3.6) and passes all four (green). The third is the same logic written more cleanly, and the same tests prove nothing broke (refactor).
Keep each cycle short, minutes not hours. If a step feels big, the test was too big: split it.
07Bug first: reproduce, then fix
When a user reports a bug, do not start by changing code. Write a test that fails for exactly that report. When it fails the way the user described, you have understood the bug. Then fix the code until the test passes, and keep the test forever. This is the cheapest and most reliable use of TDD.
08Property-based testing, by hand
The last runnable example does what Hypothesis does, in 15 lines: generate many random lists (with a seed so the output repeats), call the function, and check rules. my_sort passes the example test [3, 1, 2], but it uses set, which silently drops duplicates. A random list with a repeated number finds it in two tries.
09Hypothesis: strategies, properties and shrinking
With Hypothesis you describe the inputs with a strategy: st.lists(st.integers()) means "any list of integers, including empty and huge". The decorated test is called many times. When a call fails, Hypothesis shrinks the input. Here it reports the smallest list that breaks the rule: two zeros. (Hypothesis is not available in the in-browser Python, so this is a normal block. Output trimmed.)
from hypothesis import given, strategies as st
def my_sort(items):
return sorted(set(items)) # bug: drops duplicates
@given(st.lists(st.integers()))
def test_sort_keeps_length(items):
assert len(my_sort(items)) == len(items)Try it yourself
- 1
Reach 100 % in the tracer
In the first example, change the second list to also include 10 and confirm that the missing list becomes empty. Then add a new line to
grade(for example,if score < 0: return "invalid") and see which line is never run. - 2
Write the test the coverage missed
In the second example, decide what
average([])should do (raiseValueError, or return 0). Write a test that expects your choice, watch it fail, and fixaverage. That is the TDD cycle on a real gap. - 3
Add a property
In the property example, add a third rule: the sorted result must contain exactly the same numbers as the input (compare
sorted(data)withresult). Which rule breaks first for the buggymy_sort?
Code & diagrams
The first run calls `grade` with 95 and 70 and never reaches the last line. Add 10 to the second list and watch coverage reach 100 %.
Expected output
calls 95 and 70
lines run (relative to def): [1, 2, 3, 4]
lines never run: [5]
coverage: 80%
calls 95, 70 and 10
lines run (relative to def): [1, 2, 3, 4, 5]
lines never run: []
coverage: 100%Every line of `average` ran, so a coverage tool would show 100 %. Decide what `average([])` should do, then add a test for it.
Expected output
test passed: every line of average() ran, so line coverage is 100%
but average([]) still crashes: division by zero
average([-1, 1, 0.5]) = 0.16666666666666666 - is that what the caller wants for a mix?The test table is written first. Add a new row, for example `("Done!", "done")`, and see whether all three versions still pass.
Expected output
step 1 (red): first attempt
ok slugify('Hello World') -> 'hello-world'
FAIL slugify(' Hello, World! ') -> '--hello,---world!--'
FAIL slugify('Python 3.13 is out') -> 'python-3.13-is-out'
ok slugify('') -> ''
2 failing
step 2 (green): use a regular expression
ok slugify('Hello World') -> 'hello-world'
ok slugify(' Hello, World! ') -> 'hello-world'
ok slugify('Python 3.13 is out') -> 'python-3-13-is-out'
ok slugify('') -> ''
0 failing
step 3 (refactor): same behaviour, cleaner code
ok slugify('Hello World') -> 'hello-world'
ok slugify(' Hello, World! ') -> 'hello-world'
ok slugify('Python 3.13 is out') -> 'python-3-13-is-out'
ok slugify('') -> ''
0 failingThe seed makes the 'random' lists the same every run. Change `random.seed(7)` to another number and see whether a different input finds the bug.
Expected output
example test passed
property broken after 2 tries: length changed
input: [0, 0, 4, 0, 2]
output: [0, 2, 4]Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
A test with no assert
Write a test that calls the function but never checks the result.
def apply_discount(amount, percent):
return amount - amount * percent / 10 # bug: should be / 100
def test_discount():
apply_discount(200, 10) # runs the code, checks nothing
test_discount()
print("test passed, every line ran: 100% coverage")
print("but the real answer is", apply_discount(200, 10), "instead of 180.0")Break #2
Report coverage before measuring anything
Run coverage report in a folder where no measurement has been recorded.
coverage erase
coverage reportMyth vs fact
Myth
100 % coverage means the code is fully tested.
Fact
It means every line ran at least once. Tests without asserts, missing edge cases and missing code all hide behind it.
Myth
TDD means writing all the tests before any code.
Fact
TDD works in tiny loops: one failing test, the smallest code to pass it, a bit of cleanup, then the next test.
Myth
Property-based testing replaces example tests.
Fact
It adds to them. Examples document specific behaviour and regressions; properties explore the space around them. Use both.
Myth
Random inputs make the tests flaky.
Fact
Hypothesis stores failures and replays them, and a seed or a printed example reproduces any run. A failing property gives a stable, minimal example.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Measure branch coverage, not just lines. An
ifwithoutelse(or aforloop that is never empty, or never entered) shows as fully covered by lines even when one outcome never happened. Branch coverage catches that, at a small speed cost. It works line to line, so it cannot see inside one line:x = a if cond else b,a or bandif x: ywritten on a single line still count as one step. - ▸
coverage.py's default engine is a C tracer built on the same hook as
sys.settrace; on Python 3.12+ it can usesys.monitoring(setCOVERAGE_CORE=sysmon), which is cheaper but, before 3.14, does not support branch measurement. Configure it inpyproject.tomlunder[tool.coverage.run](source,branch = true) and[tool.coverage.report](exclude_also,fail_under). Mark code that truly cannot run in tests with# pragma: no cover, and keep those marks rare and reviewed, because each one is a hole you chose not to see. Code in subprocesses and worker processes needs extra setup (concurrency,parallel) or it is silently not counted. - ▸
Use coverage as a diff gate more than a global number: require that the lines you changed in a pull request are covered (tools such as Codecov and
diff-coverdo this). It keeps new code tested without forcing a cleanup of old code. - ▸
Mutation testing is the next step beyond coverage: a tool makes small changes to your code (turn
>into>=) and checks whether any test fails. If no test notices, your tests did not really check that line. It is slow, so teams apply it to the most important modules (Python tools includemutmutandcosmic-ray). - ▸
Hypothesis runs 100 examples per test by default (
max_examples) and gives each test a time limit per example (deadline, 200 ms), so a slow or flaky function shows up asDeadlineExceeded. A function-scoped pytest fixture is created once for all examples of a@giventest, not once per example, so state leaks between examples (Hypothesis raises a health-check error for this); build fresh state inside the test body, or usest.builds(...). In CI,settings.register_profile("ci", derandomize=True)makes runs repeatable. - ▸
Hypothesis stores failing examples in a
.hypothesis/folder and replays them first on the next run. Keep that folder out of version control, but use@example(...)to pin an important failing input into the test file. Usesettings(max_examples=...)to control the cost in CI.
Remember this
- 1
To measure it, install the
coveragepackage, orpytest-cov, which plugs it into pytest:pytest --cov=shop --cov-report=term-missing. The report shows, per file, the number of statements (lines that do something), how many were missed, the percentage, and the line numbers that never ran. - 2
Add
--cov-branchfor branch coverage, which also notices anifwhose "false" side was never taken.--cov-fail-under=90makes the run fail when coverage drops below a number, which CI (a server that runs your checks on every change, Topic 11.8) can enforce. - 3
How does it work? The tool asks Python to report every line as it runs. Debuggers use the same mechanism (
sys.settrace; Python 3.12 added a faster one,sys.monitoring). The tool records the set of lines seen, then compares it with the set of lines that exist. The runnable example does the same in a few lines, so the idea is not magic. - 4
What coverage does not prove is the part beginners miss. First, a line that ran is not a line that was checked: a test with no
assertstill gives 100 % coverage (see the Break it). Second, a line that ran with one input may fail with another: 100 % coverage andaverage([])still crashes. Third, coverage cannot see code that is missing: the validation you forgot to write. - 5
Treat coverage as a map of untested code, not as a score. Chasing 100 % produces tests that exist only to colour lines green. A common healthy target is a high number that the team never games, such as 80 to 90 %, with extra care on the risky modules.
- 6
Test-driven development (TDD) reverses the order: write the test first, then the code. The cycle has three steps and a colour for each. Red: write a small test for behaviour that does not exist yet and watch it fail (this proves the test can fail). Green: write the simplest code that makes it pass. Refactor: clean up the code while the tests stay green.
- 7
Then repeat with the next small behaviour. The runnable example walks through all three.
- 8
TDD is a tool, not a religion. It shines when the behaviour is clear (parsers, rules, calculations) and when fixing a bug: first write a test that reproduces the bug and see it fail, then fix the code. That test then guards against the bug returning (a regression test, Topic 11.1). It is awkward for exploratory work where you do not yet know what you want; sketch first, then add tests.
- 9
Property-based testing changes what a test says. Instead of "for input 3, the answer is 9" (an example), you state a rule that must hold for every input, such as "sorting never changes the length", and let a tool generate hundreds of random inputs to try to break it. In Python the library is Hypothesis.
- 10
When Hypothesis finds a failing input it shrinks it to the smallest one that still fails, so you debug
[0, 0]and not a list of 37 random numbers. It also remembers failures and replays them next time. - 11
Good properties are easy to find once you know the shapes. Round trip:
decode(encode(x)) == x. Idempotent: doing it twice equals doing it once (sorted(sorted(x)) == sorted(x)). Invariant: something stays true (length, sum, no duplicates added). Oracle: compare a clever fast version with a slow obvious one. - 12
In AI work these are ideal for the plain-Python parts: "parsing any valid JSON reply gives back the original data", and "the reply parser never raises anything except
ValueError, whatever string you give it".
Explain it without notes
What does code coverage measure, and what does it not tell you?
What is the difference between line coverage and branch coverage?
Describe red, green, refactor and why you watch the test fail first.
What is property-based testing, and what is shrinking?
Give three useful kinds of properties and an example of each.
Practice
Write a function clamp(value, low, high) using TDD: first the table of cases (below, inside, above, equal to the limits), then the implementation. Print which cases pass.
Write a property test by hand (seeded random) that checks reversed_twice(x) == x for 100 random lists using list(reversed(list(reversed(x)))).
Use sys.settrace as in the example to show that a function with an if has a line that a single call never reaches.
Trade-offs
- ↔
High coverage targets versus meaningful tests. A hard 100 % rule invites tests that only touch lines. A moderate target plus code review of test quality works better.
- ↔
TDD versus exploration. TDD gives clean, tested design for well-understood behaviour. For experiments, explore first, then write the tests for what you keep.
- ↔
Property tests find more but cost more. They need thought to write good properties and can be slower. Use them for parsers, serialisers, algorithms and anything with a clear rule.
Done when you can
Done when you can run coverage with
pytest --covand read theMissingcolumn.Done when you can explain three things coverage does not prove.
Done when you can follow a red, green, refactor cycle on a small function.
Done when you can turn a bug report into a failing test first.
Done when you can state a property (round trip, idempotent, invariant) and explain shrinking.