unittest and doctest

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

`unittest`

The testing framework in Python's standard library.

`TestCase`

The base class for unittest tests. Each test_* method is one test.

`setUp` / `tearDown`

Methods that run before and after every test method.

Assertion method

A self.assertXxx method that checks something and fails the test with a clear message.

Test loader / runner / result

The parts that find tests, run them, and record what passed or failed.

`doctest`

A module that runs the examples written inside docstrings.

Docstring

A string right under a def or class that documents it (Topic 5.9).

Directive

A special comment such as +ELLIPSIS that changes how a doctest compares output.

Legacy suite

An older, existing set of tests written in another style than the one you now prefer.

1. 1

   Fix the failing test

   In the first example, change the expected value in test_wrong_on_purpose to 90 and run. Check that tests run: 4 and was successful: True.

2. 2

   Add a setUp-dependent test

   Add test_spend_everything that spends 100 and checks the balance is 0. Then add a second test that spends 100 as well. Both pass, because setUp gives each its own wallet. Predict what would happen if the wallet were created once outside the class.

3. 3

   Break a doctest

   In the passing doctest example change -2 to -3. Run it and read the report: the expected text, the text it got, and the line number.

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

import io
import unittest


class Demo(unittest.TestCase):
    def test_one(self):
        self.assertEqual(1 + 1, 2)

    def cannot_overspend(self):          # no test prefix: never collected
        self.assertEqual(1 + 1, 3)


suite = unittest.defaultTestLoader.loadTestsFromTestCase(Demo)
result = unittest.TextTestRunner(stream=io.StringIO()).run(suite)
print("tests run:", result.testsRun, "- successful:", result.wasSuccessful())

1. 1

   unittest follows the xUnit design that came from Java's JUnit. You write a class (Topic 8.1) that inherits from unittest.TestCase (Topic 8.5). Each method whose name starts with test is one test. Inside, you use methods such as self.assertEqual(a, b) instead of a bare assert.

2. 2

   Many older projects and libraries (and Python's own test suite) are written this way, so you will meet it even if you prefer pytest.

3. 3

   setUp runs before every test method and tearDown runs after it (even when the test fails). Use them to give each test a fresh object, like the Wallet(100) in the example. Each test method runs on its own new instance of the class, which keeps tests apart.

4. 4

   setUpClass and tearDownClass (class methods, Topic 8.3) run once for the whole class. They are the unittest version of a wider-scope fixture (Topic 11.3).

5. 5

   The assertion methods tell you what failed. assertEqual(a, b) fails with 90 != 100. assertTrue, assertFalse, assertIs, assertIsNone, assertIn and assertAlmostEqual (for floats) cover the common cases. For errors use with self.assertRaises(ValueError): (a with block, Topic 7.4), or assertRaisesRegex to match the message.

6. 6

   self.subTest(...) runs several cases in one method and reports each failing one separately. @unittest.skip and @unittest.expectedFailure mark tests that should not count.

7. 7

   Run a suite with python -m unittest (it discovers files named test*.py), add -v to list each test, or name one: python -m unittest test_wallet.WalletTests.test_cannot_overspend. A script can call unittest.main(). The exit code (Topic 11.2) is non-zero if anything failed, so CI can use it.

8. 8

   Underneath, a loader builds a suite of tests, a runner runs them, and a result object counts passes, failures and errors. The runnable example uses these directly.

9. 9

   doctest searches docstrings (the text in triple quotes right under a def, Topic 5.9) for text that looks like an interactive session (Topic 0.5). A line starting with >>> is code, and the lines under it are the expected output. It runs the code and compares the printed output text exactly.

10. 10

    python -m doctest -v module.py or doctest.testmod() run them, and pytest --doctest-modules runs them inside pytest. They are perfect for short, pure examples that also serve as documentation. They are poor for anything with setup, randomness, or large output.

11. 11

    Because doctest compares text, you sometimes need directives. # doctest: +ELLIPSIS lets ... match any text (hiding a memory address, the number that says where an object sits in the computer, which changes every run). +NORMALIZE_WHITESPACE ignores differences in spaces and line breaks. <BLANKLINE> stands for an empty line in expected output.

12. 12

    Without directives, a doctest that prints a dictionary or a memory address fails for reasons that have nothing to do with correctness.

13. 13

    Reading a legacy unittest suite is mostly translation. self.assertEqual(a, b) is assert a == b; setUp is a fixture; assertRaises is pytest.raises. The good news is that **pytest runs TestCase classes unchanged**, so you can adopt pytest as the runner for an old suite today and write new tests in pytest style next to it. You do not need to rewrite working tests.

• ↔

  unittest: no install, more ceremony. It is always available and explicit, but needs classes and long assertion method names. pytest is shorter but is a dependency.

• ↔

  doctest: documentation that cannot rot, but fragile. Examples stay correct because they run, but exact text matching makes them break on harmless formatting changes. Keep them small.

• ↔

  Rewriting old tests: low value, high risk. A working test suite that still runs has value. Convert when you change the code under it, not for fashion.

• Done when you can write a TestCase with setUp and assertion methods and run it.

• Done when you can explain the difference between a failure and an error.

• Done when you can write a docstring with >>> examples and run it with doctest.

• Done when you can use +ELLIPSIS and <BLANKLINE> in a doctest.

• Done when you can read a unittest-based test file and say how its pieces map to pytest.