Adding Tests#

Ensure you have installed the development dependencies by following the instructions in the installation guide. To run all tests, use the following command:

invoke test

PyTest#

Tests can be added to the tests directory. PyTest will automatically discover and run these tests. They should be named test_*.py, and the test functions should be named test_*. See the PyTest documentation for more information.

Use PyTest style tests for parameterised tests, tests that require fixtures, and tests that require setup.

These tests can be run with:

pytest

Or to run a specific test:

pytest tests/test_*.py

Or to run with the same configuration as continuous integration:

invoke test.pytest

Testing PyTorch-Optional Code#

PyTorch is an optional extra (see the PyTorch note in Setup). CI runs the suite twice: once with -m "not torch" and no PyTorch installed, once with -m "torch" and --extra torch-cpu.

Mark any test that needs torch with @pytest.mark.torch, or pytest.param(..., marks=pytest.mark.torch) for one case of a parametrized test, so -m "not torch" deselects it.
A module that needs torch just to be collected – e.g. one that imports capymoa.ocl at module scope – needs more than a marker, since collection happens before marker-based deselection. Assign pytest.markskip("torch") (defined in tests/conftest.py) to pytestmark before the torch-touching import:
```
import pytest

pytestmark = pytest.markskip("torch")

import torch
from capymoa.ocl.util._buffer_list import BufferList
```
- markskip returns pytest.mark.torch when -m selects it (so the assignment marks the whole module, same as pytestmark = pytest.mark.torch written by hand), or raises a module-level skip when it doesn’t.
- It never checks whether the dependency is actually importable: when torch tests are wanted (-m "torch", or no -m filter at all) but torch isn’t installed, the import right after markskip fails loudly instead of being silently skipped.
For a file that mixes torch and non-torch cases, call markskip("torch") bare (discarding the return value) inside the constructor for just the torch-only case instead of guarding the whole module – see _make_finetune in tests/test_classifiers.py.

Doctest#

Doctest allows you to write tests directly in the docstrings of your code, making it easier to keep documentation up-to-date. The tests are written as examples in a Python interactive shell.

Use doctest style tests to document code with simple tested examples.

Here’s an example of a function with a doctest:

def hello_world():
    """
    >>> hello_world()
    Hello, World!
    """
    print("Hello, World!")

You can run this test with:

pytest --doctest-modules path/to/your/module.py

Alternatively, you can run all unit tests with the same configuration as continuous integration:

invoke test.doctest

Notebooks#

We use nbmake to test that all notebooks in the notebooks directory run without error. The notebooks are stored as Jupytext py:percent scripts rather than .ipynb files (see Notebooks), and nbmake only understands .ipynb, so invoke docs.nb regenerates a matching .ipynb file next to each .py notebook, then runs nbmake with --overwrite so it writes the real outputs back into that .ipynb file – the same file invoke docs.build later renders:

invoke docs.nb

# Often the examples take too long to run regularly as tests. To speed up
# testing some notebooks use the NB_FAST environment variable to run the
# notebook faster by using smaller datasets or fewer iterations. This is
# the default; use --slow to run the notebooks with their full-size
# datasets:
invoke docs.nb --slow

invoke test.nb is a deprecated alias for the same thing and will be removed in a future release.

For more about NB_FAST read the notebooks documentation in Documentation.

Code Coverage#

Code coverage measures how many statements of code is executed while running tests. It identifies unused and untested code. We encourage contributors to use it to write more robust programs, but don’t have a target percentage.

To generate code coverage reports add --cov=capymoa and --cov-report=html to the pytest command:

pytest --cov=capymoa --cov-report=html

Alternatively, CapyMOA’s invoke testing tasks can generate coverage reports with:

invoke test --coverage