diff options
| -rw-r--r-- | INSTALL.rst.txt | 6 | ||||
| -rw-r--r-- | doc/TESTS.rst.txt | 128 | ||||
| -rw-r--r-- | doc/source/dev/development_environment.rst | 25 | ||||
| -rw-r--r-- | doc/source/reference/routines.testing.rst | 14 | ||||
| -rw-r--r-- | doc/source/reference/testing.rst | 7 | ||||
| -rw-r--r-- | numpy/lib/tests/test_format.py | 2 |
6 files changed, 95 insertions, 87 deletions
diff --git a/INSTALL.rst.txt b/INSTALL.rst.txt index a81468d9f..d3ed7197e 100644 --- a/INSTALL.rst.txt +++ b/INSTALL.rst.txt @@ -29,12 +29,12 @@ Building NumPy requires the following software installed: 2) Cython >= 0.19 (for development versions of numpy, not for released versions) -3) nose__ (optional) 1.0 or later +3) pytest__ (optional) 1.15 or later This is required for testing numpy, but not for using it. -Python__ https://www.python.org -nose__ https://nose.readthedocs.io +Python__ http://www.python.org +pytest__ http://pytest.readthedocs.io .. note:: diff --git a/doc/TESTS.rst.txt b/doc/TESTS.rst.txt index 170519110..5fe0be1f1 100644 --- a/doc/TESTS.rst.txt +++ b/doc/TESTS.rst.txt @@ -1,19 +1,19 @@ -.. -*- rest -*- - NumPy/SciPy Testing Guidelines ============================== .. contents:: + Introduction '''''''''''' -SciPy uses the `Nose testing system -<https://nose.readthedocs.io>`__, with some -minor convenience features added. Nose is an extension of the unit -testing framework offered by `unittest.py -<https://docs.python.org/library/unittest.html>`__. Our goal is that -every module and package in SciPy should have a thorough set of unit +Until the 1.15 release, NumPy used the `nose`_ testing framework, it now uses +the `pytest`_ framework. The older framework is still maintained in order to +support downstream projects that use the old numpy framework, but all tests +for NumPy should use pytest. + +Our goal is that every module and package in SciPy and NumPy +should have a thorough set of unit tests. These tests should exercise the full functionality of a given routine as well as its robustness to erroneous or unexpected input arguments. Long experience has shown that by far the best time to @@ -33,9 +33,13 @@ To run SciPy's full test suite, use the following:: >>> import scipy >>> scipy.test() +or from the command line:: + + $ python runtests.py + SciPy uses the testing framework from NumPy (specifically -``numpy.testing``), so all the SciPy examples shown here are also -applicable to NumPy. So NumPy's full test suite can be run as +:ref:`numpy-testing`), so all the SciPy examples shown here are also +applicable to NumPy. NumPy's full test suite can be run as follows:: >>> import numpy @@ -57,7 +61,11 @@ messages about which modules don't have tests:: Finally, if you are only interested in testing a subset of SciPy, for example, the ``integrate`` module, use the following:: ->>> scipy.integrate.test() + >>> scipy.integrate.test() + +or from the command line:: + + $python runtests.py -t scipy/integrate/tests The rest of this page will give you a basic idea of how to add unit tests to modules in SciPy. It is extremely important for us to have @@ -76,7 +84,7 @@ Writing your own tests Every Python module, extension module, or subpackage in the SciPy package directory should have a corresponding ``test_<name>.py`` file. -Nose examines these files for test methods (named test*) and test +Pytest examines these files for test methods (named test*) and test classes (named Test*). Suppose you have a SciPy module ``scipy/xxx/yyy.py`` containing a @@ -121,13 +129,16 @@ Sometimes it is convenient to run ``test_yyy.py`` by itself, so we add at the bottom. -Labeling tests with nose ------------------------- +Labeling tests +-------------- + +As an alternative to ``pytest.mark.<label>``, there are a number of labels you +can use. Unlabeled tests like the ones above are run in the default ``scipy.test()`` run. If you want to label your test as slow - and therefore reserved for a full ``scipy.test(label='full')`` run, you -can label it with a nose decorator:: +can label it with a decorator:: # numpy.testing module includes 'import decorators as dec' from numpy.testing import dec, assert_ @@ -143,11 +154,26 @@ Similarly for methods:: def test_simple(self): assert_(zzz() == 'Hello from zzz') +Available labels are: + +- ``slow``: marks a test as taking a long time +- ``setastest(tf)``: work-around for test discovery when the test name is + non conformant +- ``skipif(condition, msg=None)``: skips the test when ``eval(condition)`` is + ``True`` +- ``knownfailureif(fail_cond, msg=None)``: will avoid running the test if + ``eval(fail_cond)`` is ``True``, useful for tests that conditionally segfault +- ``deprecated(conditional=True)``: filters deprecation warnings emitted in the + test +- ``paramaterize(var, input)``: an alternative to + `pytest.mark.paramaterized + <https://docs.pytest.org/en/latest/parametrize.html>`_ + Easier setup and teardown functions / methods --------------------------------------------- -Nose looks for module level setup and teardown functions by name; -thus:: +Testing looks for module-level or class-level setup and teardown functions by +name; thus:: def setup(): """Module-level setup""" @@ -158,64 +184,25 @@ thus:: print 'doing teardown' -You can add setup and teardown functions to functions and methods with -nose decorators:: - - import nose - # import all functions from numpy.testing that are needed - from numpy.testing import assert_, assert_array_almost_equal + class TestMe(object): + def setup(): + """Class-level setup""" + print 'doing setup' - def setup_func(): - """A trivial setup function.""" - global helpful_variable - helpful_variable = 'pleasant' - print "In setup_func" + def teardown(): + """Class-level teardown""" + print 'doing teardown' - def teardown_func(): - """A trivial teardown function.""" - global helpful_variable - del helpful_variable - print "In teardown_func" - @nose.with_setup(setup_func, teardown_func) - def test_with_extras(): - # This test uses the setup/teardown functions. - global helpful_variable - print " In test_with_extras" - print " Helpful is %s" % helpful_variable +Setup and teardown functions to functions and methods are known as "fixtures", +and their use is not encouraged. Parametric tests ---------------- -One very nice feature of nose is allowing easy testing across a range -of parameters - a nasty problem for standard unit tests. It does this -with test generators:: - - def check_even(n, nn): - """A check function to be used in a test generator.""" - assert_(n % 2 == 0 or nn % 2 == 0) - - def test_evens(): - for i in range(0,4,2): - yield check_even, i, i*3 - -Note that ``check_even`` is not itself a test (no 'test' in the name), -but ``test_evens`` is a generator that returns a series of tests, using -``check_even``, across a range of inputs. - -A problem with generator tests can be that if a test is failing, it's -hard to see for which parameters. To avoid this problem, ensure that: - - - No computation related to the features tested is done in the - ``test_*`` generator function, but delegated to a corresponding - ``check_*`` function (can be inside the generator, to share namespace). - - The generators are used *solely* for loops over parameters. - - Those parameters are *not* arrays. - -.. warning:: - - Parametric tests cannot be implemented on classes derived from - TestCase. +One very nice feature of testing is allowing easy testing across a range +of parameters - a nasty problem for standard unit tests. Use the +``dec.paramaterize`` decorator. Doctests -------- @@ -384,3 +371,8 @@ occasionally with no code changes is not helpful. Make the random data deterministic by setting the random number seed before generating it. Use either Python's ``random.seed(some_number)`` or NumPy's ``numpy.random.seed(some_number)``, depending on the source of random numbers. + + +.. _nose: https://nose.readthedocs.io/en/latest/ +.. _pytest: https://pytest.readthedocs.io +.. _parameterization: https://docs.pytest.org/en/latest/parametrize.html diff --git a/doc/source/dev/development_environment.rst b/doc/source/dev/development_environment.rst index f4c6f3ec7..0c9f307f8 100644 --- a/doc/source/dev/development_environment.rst +++ b/doc/source/dev/development_environment.rst @@ -3,7 +3,6 @@ Setting up and using your development environment ================================================= - Recommended development setup ----------------------------- @@ -22,7 +21,7 @@ do one of:: $ python runtests.py -v $ python runtests.py -v -s random - $ python runtests.py -v -t numpy/core/tests/test_iter.py:test_iter_c_order + $ python runtests.py -v -t numpy/core/tests/test_nditer.py::test_iter_c_order $ python runtests.py --ipython $ python runtests.py --python somescript.py $ python runtests.py --bench @@ -35,10 +34,15 @@ any) found on current PYTHONPATH. When specifying a target using ``-s``, ``-t``, or ``--python``, additional arguments may be forwarded to the target embedded by ``runtests.py`` by passing the extra arguments after a bare ``--``. For example, to run a test method with -the ``--pdb`` flag forwarded to nose, run the following:: +the ``--pdb`` flag forwarded to the target, run the following:: $ python runtests.py -t numpy/tests/test_scripts.py:test_f2py -- --pdb +When using pytest as a target (the default), you can +`match test names using python operators`_ by passing the ``-k`` argument to pytest:: + + $ python runtests.py -v -t numpy/core/tests/test_multiarray.py -- -k "MatMul and not vector" + Using ``runtests.py`` is the recommended approach to running tests. There are also a number of alternatives to it, for example in-place build or installing to a virtualenv. See the FAQ below for details. @@ -123,9 +127,8 @@ Or a similar way from the command line:: $ python -c "import numpy as np; np.test()" -Tests can also be run with ``nosetests numpy``, however then the NumPy-specific -``nose`` plugin is not found which causes tests marked as ``KnownFailure`` to -be reported as errors. +Tests can also be run with ``pytest numpy``, however then the NumPy-specific +plugin is not found which causes strange side effects Running individual test files can be useful; it's much faster than running the whole test suite or that of a whole module (example: ``np.random.test()``). @@ -141,10 +144,9 @@ run the test suite with Python 3.4, use:: $ tox -e py34 -For more extensive info on running and writing tests, see -https://github.com/numpy/numpy/blob/master/doc/TESTS.rst.txt . +For more extensive information, see :ref:`testing-guidelines` -*Note: do not run the tests from the root directory of your numpy git repo, +*Note: do not run the tests from the root directory of your numpy git repo without ``runtests.py``, that will result in strange test errors.* @@ -202,14 +204,11 @@ typically packaged as ``python-dbg``) is highly recommended. .. _DebuggingWithGdb: https://wiki.python.org/moin/DebuggingWithGdb - .. _tox: https://tox.readthedocs.io/ - .. _virtualenv: http://www.virtualenv.org/ - .. _virtualenvwrapper: http://www.doughellmann.com/projects/virtualenvwrapper/ - .. _Waf: https://code.google.com/p/waf/ +.. _`match test names using python operators`: https://docs.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests Understanding the code & getting started ---------------------------------------- diff --git a/doc/source/reference/routines.testing.rst b/doc/source/reference/routines.testing.rst index ad95bb399..5a52a40d6 100644 --- a/doc/source/reference/routines.testing.rst +++ b/doc/source/reference/routines.testing.rst @@ -1,3 +1,5 @@ +.. _numpy-testing: + Test Support (:mod:`numpy.testing`) =================================== @@ -6,8 +8,9 @@ Test Support (:mod:`numpy.testing`) Common test support for all numpy test scripts. This single module should provide all the common functionality for numpy -tests in a single location, so that test scripts can just import it and -work right away. +tests in a single location, so that :ref:`test scripts +<development-environment>` can just import it and work right away. For +background, see the :ref:`testing-guidelines` Asserts @@ -50,3 +53,10 @@ Test Running run_module_suite rundocs suppress_warnings + +Guidelines +---------- + +.. toctree:: + + testing diff --git a/doc/source/reference/testing.rst b/doc/source/reference/testing.rst new file mode 100644 index 000000000..72780dd44 --- /dev/null +++ b/doc/source/reference/testing.rst @@ -0,0 +1,7 @@ +.. _testing-guidelines: + +Testing Guidelines +================== + +.. include:: ../../TESTS.rst.txt + :start-line: 6 diff --git a/numpy/lib/tests/test_format.py b/numpy/lib/tests/test_format.py index 38a9b8000..fd227595a 100644 --- a/numpy/lib/tests/test_format.py +++ b/numpy/lib/tests/test_format.py @@ -479,7 +479,7 @@ def test_long_str(): @pytest.mark.slow def test_memmap_roundtrip(): - # Fixme: test crashes nose on windows. + # Fixme: used to crash on windows if not (sys.platform == 'win32' or sys.platform == 'cygwin'): for arr in basic_arrays + record_arrays: if arr.dtype.hasobject: |