PHASE 11Intermediate ~31 min· topic 1 of 8

Topic 11.1

Why Test, and What a Test Is

In one line

A test is a small program that runs your code with a chosen input and checks the result against the answer you expect. Tests are a safety net: they let you change code and find out in seconds whether something broke.

Think of it like this

Before a long drive, you check the lights, the brakes and the tyres. Each check is a tiny question with a known right answer: "do the brake lights come on when I press the pedal?" You do it before the trip, not on the motorway. A test is the same: a small, automatic check with a known right answer, run before your code reaches real users.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Test: A small program that runs your code with a chosen input and checks the result.
Assert (assertion): A check that something is true. If it is false, the program stops with an error.
AssertionError: The error Python raises when an assert condition is false.
Arrange, act, assert: The three parts of a test: set up, call the code, check the result.
Unit test: A test of one small piece of code, such as a single function.
Integration test: A test that checks several pieces working together.
End-to-end test: A test that drives the whole system the way a user would.
Regression: A bug that comes back, or something that used to work and now does not.
Boundary value: An input at the edge of a rule: zero, empty, exactly on the limit, just below or above it.
Test runner: A program that finds your tests, runs them and reports which passed.

Step by step

01Checking by eye is not a test

The usual way to check code is to run it, print something and look. That works once. It does not work next week, when you have forgotten what the right answer was, or when a colleague changes the code. A test writes the expected answer down and checks it for you, every time.

Compare the two ways of checking the same function. The first needs a human to read and judge. The second needs nobody.

main.pywhole filepython

def total(prices):
    return sum(prices)


# By eye: you must remember that 42 is right
print(total([30, 12]))

# As a check: the right answer is written down, and the computer judges
assert total([30, 12]) == 42

02A test is a function with asserts

Put the check inside a function named for what it checks. The name should say the situation and the expected result, so a failure reads like a sentence: "test exactly 500 ships free" failed.

The function takes no arguments and returns nothing. It passes if it finishes, and fails if an assert inside it raises AssertionError.

main.pywhole filepython

def shipping_fee(order_total):
    if order_total >= 500:
        return 0
    return 40


def test_exactly_500_ships_free():
    # arrange
    order_total = 500
    # act
    fee = shipping_fee(order_total)
    # assert
    assert fee == 0, f"expected 0, got {fee}"

A test is a function with assertsdiagram

Rendering diagram…

03What assert does, exactly

assert condition, message means: "if condition is not true, raise AssertionError(message)". The message is optional but worth writing: it is what you read at 2 a.m. when the test fails.

Any value works as the condition, using the same truthy and falsy rules as if (Topic 1.4): 0, "", [] and None fail, most other values pass. Python can run with the -O flag ("optimise"), which removes every assert. So assert is for tests and for checking your own assumptions, never for validating input from users or files. For that, raise a real exception (Topic 7.2).

terminal

$ python -c "assert False, 'boom'"

python -O -c "assert False, 'boom'

print('assert was skipped under -O')"

── expected output ──

Traceback (most recent call last):

File "<string>", line 1, in <module>

AssertionError: boom

assert was skipped under -O

04A runner finds and runs the tests

Writing tests is only useful if running them is effortless. A test runner finds every test, runs each one separately, so that one failure does not hide the others, and prints a summary. The tiny runner below does it in a few lines. It looks through the program's names (globals() is a dictionary of every name defined at the top of the file, Topic 5.5), keeps the functions whose name starts with test_, and calls each inside try (Topic 7.1).

This is the same idea pytest uses, with a lot more care. Notice that the second test fails but the third still runs. Tests must not depend on each other or on their order.

main.pywhole filepython

tests = [fn for name, fn in sorted(globals().items()) if name.startswith("test_")]
passed = 0
for test in tests:
    try:
        test()
        print("PASS", test.__name__)
        passed += 1
    except AssertionError as error:
        print("FAIL", test.__name__, "-", error)
print(f"{passed} of {len(tests)} passed")

05Pick inputs at the edges

Run the first runnable example below. The code says "free shipping above 500", but the rule is "from 500 up". The ordinary tests (120 and 900) both pass. Only the boundary test, exactly 500, catches the bug.

A useful habit: for each rule, write one test just below the boundary, one on it, and one just above. Also try the empty case and the single-item case. Most bugs hide in those few inputs.

06Tests as documentation and as a net

A good test suite also documents behaviour. If you want to know what shipping_fee(500) does, the test says so with a real, checked example. Unlike a comment, a test cannot quietly go out of date: it fails when the code and the claim disagree.

And it is a net under every future change. The usual cycle is: change code, run tests, fix whatever turned red, repeat. The faster the tests run, the more often you run them, so keep unit tests quick.

Tests as documentation and as a netdiagram

Rendering diagram…

Try it yourself

1
Fix the boundary bug
In the first example, change > to >= in shipping_fee and run it. All three tests pass. You just did the core loop of testing: a red test showed the bug, a small fix turned it green.
2
Add the missing edge tests
Add test_just_below_500_pays_fee (order 499, expected 40) and test_zero_order_pays_fee (order 0, expected 40). Predict whether they pass before you run. Each new test is one more piece of the net.
3
Break it on purpose
Change the expected value in one test so it is wrong, and read the message. Then delete the message after the comma and run again. Which version would you rather read when a test fails?

Code & diagrams

Three tests and a tiny runner (one fails) New tab

The shipping rule is 'free from 500 up' but the code uses `>`. Which test catches it? Change `>` to `>=` and run again.

Expected output

PASS test_big_order_ships_free
FAIL test_exactly_500_ships_free - expected 0, got 40
PASS test_small_order_pays_fee
2 of 3 passed

How assert behaves: pass, fail with a message, stop the program New tab

The last assert has no message and is not caught, so the traceback ends the program. Fix the expected value to 15 and run again.

Expected output

first assert passed
caught AssertionError: average of 1, 2 should be 2
empty list raises ZeroDivisionError: division by zero
Traceback (most recent call last):
  File "main.py", line 21, in <module>
    assert average([10, 20]) == 20
           ^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

The test pyramiddiagram

Rendering diagram…

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

An assert that can never fail

Wrap the condition and the message together in round brackets, as many people do with print.

main.pywhole filepython

def total(prices):
    return sum(prices)


assert (total([1, 2]) == 4, "total should be 3")
print("the program kept going: this assert never fails")

terminal

$ python main.py

── what you'll see ──

main.py:5: SyntaxWarning: assertion is always true, perhaps remove parentheses?

assert (total([1, 2]) == 4, "total should be 3")

the program kept going: this assert never fails

Break #2

Compare floats with ==

Write a test that expects 0.1 + 0.2 to equal 0.3.

main.pywhole filepython

def test_float_sum():
    assert 0.1 + 0.2 == 0.3, "0.1 + 0.2 should be 0.3"


test_float_sum()

terminal

$ python main.py

── what you'll see ──

Traceback (most recent call last):

File "main.py", line 5, in <module>

test_float_sum()

~~~~~~~~~~~~~~^^

File "main.py", line 2, in test_float_sum

assert 0.1 + 0.2 == 0.3, "0.1 + 0.2 should be 0.3"

^^^^^^^^^^^^^^^^

AssertionError: 0.1 + 0.2 should be 0.3

Myth vs fact

Myth

Testing is only for big companies or big projects.

Fact

A five-line function with one assert already pays for itself the first time you change the function. Small projects change often and have nobody else to catch mistakes.

Myth

If my tests pass, my code has no bugs.

Fact

Tests only prove that the cases you wrote work. A bug in a case you did not think of passes silently. Passing tests lower the risk; they do not remove it.

Myth

Writing tests slows me down.

Fact

Writing them costs minutes; debugging an untested change costs hours. The slowdown is real only on the first day; after that you move faster because you dare to change things.

Myth

I should test every line, including print statements and simple getters.

Fact

Test behaviour that matters: rules, edge cases, error handling, and bugs you have already met. Tests of trivial code add noise and break when you rename things.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Good tests check behaviour, not implementation. A test that asserts which internal helper was called breaks when you refactor, even though nothing visible changed. Test what the function returns or what effect it has, and your tests survive rewrites, which is what a safety net is for.
▸
Keep tests deterministic and independent. A test that depends on today's date, a random number, the order of other tests or the network will fail one day for no reason (a "flaky" test). Teams stop trusting flaky suites and then stop running them. Control time, randomness and I/O with fakes (Topic 11.4).
▸
The pyramid has a reason: a unit test pinpoints a bug to one function in milliseconds, while an end-to-end test says only that "checkout is broken" and takes seconds or minutes. Put most checks low, and use higher tests for the paths whose failure would cost real money.
▸
For code that calls a language model, do not assert on exact model text. Assert on the structure you control: the prompt you built, the JSON you parse, the retry and timeout behaviour. Quality of the answers is measured separately with an evaluation set, run less often because it is slow and costs money.

Remember this

1
A test calls your code with an input you chose and compares the result with the answer you expected. If they match, the test passes. If not, it fails and tells you what went wrong. In Phase 0 you saw that some bugs are silent: the program runs fine and gives a wrong answer (apple + apple). Tests are how you catch exactly those bugs, because only a test knows what the answer was supposed to be.
2
Almost every test has three parts, called arrange, act, assert.
3
Arrange: set up the input and anything the code needs. Act: call the code you are testing, once. Assert: check that the result is what you expected. Keeping the three parts visible makes a test easy to read and quick to fix when it fails.
4
Python's built-in tool for the last part is the assert statement: assert condition, message. If the condition is true, nothing happens. If it is false, Python raises (Topic 7.2) an AssertionError with your message. A test function is then just an ordinary function (Topic 5.1) that contains asserts. No library is needed for the idea; pytest (Topic 11.2) only makes it more comfortable.
5
Tests give you a safety net. Once your code has tests, you can fix a bug, speed something up, or rewrite a function, run the tests, and know within seconds whether you broke something that used to work. A bug that comes back after being fixed is called a regression, and a test written when you fixed it makes sure it can never return unnoticed. Without tests, people become afraid to touch old code. That fear is the most expensive bug of all.
6
Choose the inputs that break things, not only the easy ones. Bugs live at boundaries: zero, one, empty, the largest allowed value, exactly on the limit, one below the limit. The free-shipping bug in this topic is a classic: the rule says "free from 500 up" and the code says "above 500", so only an order of exactly 500 shows it.
7
Tests come in sizes, usually drawn as a test pyramid. At the bottom are many fast unit tests, each checking one small piece (a function). In the middle are fewer integration tests, which check that pieces work together (your code plus a real database). At the top are a few end-to-end tests, which drive the whole system like a user. Lower tests are fast, cheap and precise; higher tests are slow and break for more reasons but prove more. Write mostly unit tests, then add a few higher ones for the important paths.
8
In AI work the same rule applies, with one twist: a language model's answer can change from run to run, so you cannot assert on the exact text it returns. You test the Python around the model instead. That is the code that builds the prompt (the text you send to the model), parses the reply (reads the text and pulls out its parts) and handles errors. You check it with a stand-in for the model (Topic 11.4). The checks of answer quality (called evals) are a separate job.

Explain it without notes

What is a test, and what are arrange, act and assert?

What does the assert statement do, and why should you not use it to validate user input?

What is a regression, and how do tests protect against one?

Explain the test pyramid. Why are there more unit tests than end-to-end tests?

How would you choose inputs for a function that has a rule like "free shipping from 500 up"?

Practice

Write is_adult(age) that returns True from 18 up. Write three tests: below, on and above the boundary, and run them with a loop like the one in this topic.

This function should return the largest number in a list, but the test with [-5, -2, -9] fails. Find the bug: def biggest(nums): top = 0 then a loop that sets top = n when n > top, then return top. Write the failing test first.

Write a test for average([]). Decide what it should do (raise ZeroDivisionError, or return 0?), write the test for your decision, then make the function match.

Trade-offs

↔
Tests cost time to write and to maintain. Every test is code that can break. Write tests for behaviour that matters and that will not change weekly; skip tests that only repeat the code line by line.
↔
Fast tests are used, slow tests are skipped. More realistic tests (real database, real network) catch more problems but run slowly and fail more randomly. The pyramid is the compromise: realism where it counts, speed everywhere else.
↔
Tests give confidence, not proof. A green suite means the cases you thought of work. That is still far more than nothing, and the next topics add tools (coverage, property-based testing) to find the cases you did not think of.

Done when you can

Done when you can write a test with arrange, act and assert for a small function.
Done when you can explain what assert does and why python -O makes it unsafe for validation.
Done when you can choose boundary inputs for a rule such as "from 500 up".
Done when you can explain the difference between unit, integration and end-to-end tests.
Done when you can say why a test of a language-model call checks the surrounding Python and not the model's exact words.

Command Palette

Words you'll meet

Step by step

01Checking by eye is not a test

02A test is a function with asserts

03What assert does, exactly

04A runner finds and runs the tests

05Pick inputs at the edges

06Tests as documentation and as a net

Try it yourself

Fix the boundary bug

Add the missing edge tests

Break it on purpose

Code & diagrams

Break it on purpose

An assert that can never fail

Compare floats with ==

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can