diff options
Diffstat (limited to 'doc')
134 files changed, 2398 insertions, 931 deletions
diff --git a/doc/C_STYLE_GUIDE.rst.txt b/doc/C_STYLE_GUIDE.rst.txt index 07f4b99df..4e2f27fbb 100644 --- a/doc/C_STYLE_GUIDE.rst.txt +++ b/doc/C_STYLE_GUIDE.rst.txt @@ -1,211 +1,3 @@ -=================== -NumPy C Style Guide -=================== -The NumPy C coding conventions are based on Python PEP-0007 by Guido van -Rossum with a few added strictures. There are many C coding conventions and -it must be emphasized that the primary goal of the NumPy conventions isn't -to choose the 'best', about which there is certain to be disagreement, but -to achieve uniformity. Because the NumPy conventions are very close to -those in PEP-0007, that PEP is used as a template below with the NumPy -additions and variations in the appropriate spots. - -Introduction ------------- - -This document gives coding conventions for the C code comprising -the C implementation of NumPy. Note, rules are there to be broken. -Two good reasons to break a particular rule: - -1. When applying the rule would make the code less readable, even - for someone who is used to reading code that follows the rules. - -2. To be consistent with surrounding code that also breaks it - (maybe for historic reasons) -- although this is also an - opportunity to clean up someone else's mess. - - -C dialect ---------- - -* Use C99 (that is, the standard defined by ISO/IEC 9899:1999). - -* Don't use GCC extensions (e.g. don't write multi-line strings - without trailing backslashes). Preferably break long strings - up onto separate lines like so:: - - "blah blah" - "blah blah" - - This will work with MSVC, which otherwise chokes on very long - strings. - -* All function declarations and definitions must use full - prototypes (i.e. specify the types of all arguments). - -* No compiler warnings with major compilers (gcc, VC++, a few others). - Note: NumPy still produces compiler warnings that need to be addressed. - - -Code lay-out ------------- - -* Use 4-space indents and no tabs at all. - -* No line should be longer than 80 characters. If this and the - previous rule together don't give you enough room to code, your code is - too complicated, consider using subroutines. - -* No line should end in whitespace. If you think you need - significant trailing whitespace, think again, somebody's editor might - delete it as a matter of routine. - -* Function definition style: function name in column 1, outermost - curly braces in column 1, blank line after local variable declarations:: - - static int - extra_ivars(PyTypeObject *type, PyTypeObject *base) - { - int t_size = PyType_BASICSIZE(type); - int b_size = PyType_BASICSIZE(base); - - assert(t_size >= b_size); /* type smaller than base! */ - ... - return 1; - } - - If the transition to C++ goes through it is possible that this form will - be relaxed so that short class methods meant to be inlined can have the - return type on the same line as the function name. However, that is yet to - be determined. - -* Code structure: one space between keywords like ``if``, ``for`` and - the following left parenthesis; no spaces inside the parenthesis; braces - around all ``if`` branches and no statements on the same line as the - ``if``. They should be formatted as shown:: - - if (mro != NULL) { - one_line_statement; - } - else { - ... - } - - - for (i = 0; i < n; i++) { - one_line_statement; - } - - - while (isstuff) { - dostuff; - } - - - do { - stuff; - } while (isstuff); - - - switch (kind) { - /* Boolean kind */ - case 'b': - return 0; - /* Unsigned int kind */ - case 'u': - ... - /* Anything else */ - default: - return 3; - } - - -* The return statement should *not* get redundant parentheses:: - - return Py_None; /* correct */ - return(Py_None); /* incorrect */ - -* Function and macro call style: ``foo(a, b, c)``, no space before - the open paren, no spaces inside the parens, no spaces before - commas, one space after each comma. - -* Always put spaces around the assignment, Boolean and comparison - operators. In expressions using a lot of operators, add spaces - around the outermost (lowest priority) operators. - -* Breaking long lines: if you can, break after commas in the - outermost argument list. Always indent continuation lines - appropriately, e.g., :: - - PyErr_SetString(PyExc_TypeError, - "Oh dear, you messed up."); - - Here appropriately means at least two tabs. It isn't necessary to - line everything up with the opening parenthesis of the function - call. - -* When you break a long expression at a binary operator, the - operator goes at the end of the previous line, e.g., :: - - if (type > tp_dictoffset != 0 && - base > tp_dictoffset == 0 && - type > tp_dictoffset == b_size && - (size_t)t_size == b_size + sizeof(PyObject *)) { - return 0; - } - - Note that the terms in the multi-line Boolean expression are indented so - as to make the beginning of the code block clearly visible. - -* Put blank lines around functions, structure definitions, and - major sections inside functions. - -* Comments go before the code they describe. Multi-line comments should - be like so:: - - /* - * This would be a long - * explanatory comment. - */ - - Trailing comments should be used sparingly. Instead of :: - - if (yes) { // Success! - - do :: - - if (yes) { - // Success! - -* All functions and global variables should be declared static - when they aren't needed outside the current compilation unit. - -* Declare external functions and variables in a header file. - - -Naming conventions ------------------- - -* There has been no consistent prefix for NumPy public functions, but - they all begin with a prefix of some sort, followed by an underscore, and - are in camel case: ``PyArray_DescrAlignConverter``, ``NpyIter_GetIterNext``. - In the future the names should be of the form ``Npy*_PublicFunction``, - where the star is something appropriate. - -* Public Macros should have a ``NPY_`` prefix and then use upper case, - for example, ``NPY_DOUBLE``. - -* Private functions should be lower case with underscores, for example: - ``array_real_get``. Single leading underscores should not be used, but - some current function names violate that rule due to historical accident. - Those functions should be renamed at some point. - - -Function documentation ----------------------- - -NumPy doesn't have a C function documentation standard at this time, but -needs one. Most numpy functions are not documented in the code and that -should change. One possibility is Doxygen with a plugin so that the same -NumPy style used for Python functions can also be used for documenting -C functions, see the files in doc/cdoc/. +The "NumPy C Style Guide" at this page has been supserseded by +"NEP 45 — C Style Guide" at https://numpy.org/neps/nep-0045-c_style_guide.html diff --git a/doc/DISTUTILS.rst.txt b/doc/DISTUTILS.rst.txt index 2957bb907..15073f754 100644 --- a/doc/DISTUTILS.rst.txt +++ b/doc/DISTUTILS.rst.txt @@ -345,7 +345,7 @@ expression, '<...>' will be replaced first with item1, and then with item2, and so forth until N repeats are accomplished. Once a named repeat specification has been introduced, the same repeat rule may be used **in the current block** by referring only to the name -(i.e. <rule1>. +(i.e. <rule1>). Short repeat rule diff --git a/doc/HOWTO_RELEASE.rst.txt b/doc/HOWTO_RELEASE.rst.txt index 79a6830c4..6af8d9ca6 100644 --- a/doc/HOWTO_RELEASE.rst.txt +++ b/doc/HOWTO_RELEASE.rst.txt @@ -352,36 +352,30 @@ Also create a new version hash in cversions.txt and a corresponding version define NPY_x_y_API_VERSION in numpyconfig.h -Trigger the wheel builds on travis-ci and Appveyor --------------------------------------------------- -See the `numpy wheels` repository. +Trigger the wheel builds +------------------------ +See the `MacPython/numpy wheels` repository. In that repository edit the files: -- ``.travis.yml``; -- ``appveyor.yml``. +- ``azure/posix.yml`` +- ``azure/windows.yml``. In both cases, set the ``BUILD_COMMIT`` variable to the current release tag - -e.g. ``v1.11.1``. +e.g. ``v1.19.0``:: -Make sure that the release tag has been pushed. - -Trigger a build by doing a commit of your edits to ``.travis.yml`` and -``appveyor.yml`` to the repository:: - - cd /path/to/numpy-wheels - # Edit .travis.yml, appveyor.yml - git commit - git push + $ gvim azure/posix.yml azure/windows.yml + $ git commit -a + $ git push upstream HEAD -The wheels, once built, appear at a Rackspace container pointed at by: - -- http://wheels.scipy.org -- https://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a43.ssl.cf2.rackcdn.com +Make sure that the release tag has been pushed. -The HTTP address may update first, and you should wait 15 minutes after the -build finishes before fetching the binaries. +Trigger a build by pushing a commit of your edits to the repository. Note that +you can do this on a branch, but it must be pushed upstream to the +``MacPython/numpy-wheels`` repository to trigger uploads since only +that repo has the appropriate tokens to allow uploads. +The wheels, once built, appear at https://anaconda.org/multibuild-wheels-staging/numpy Make the release ---------------- @@ -423,7 +417,7 @@ https://github.com/MacPython/terryfy. Here is the recommended incantation for downloading all the Windows, Manylinux, OSX wheels and uploading to PyPI. :: NPY_WHLS=~/wheelhouse # local directory to cache wheel downloads - CDN_URL=https://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a43.ssl.cf2.rackcdn.com + CDN_URL=https://anaconda.org/multibuild-wheels-staging/numpy/files wheel-uploader -u $CDN_URL -w $NPY_WHLS -v -s -t win numpy 1.11.1rc1 wheel-uploader -u $CDN_URL -w warehouse -v -s -t macosx numpy 1.11.1rc1 wheel-uploader -u $CDN_URL -w warehouse -v -s -t manylinux1 numpy 1.11.1rc1 diff --git a/doc/Makefile b/doc/Makefile index 199a22c34..dd63702de 100644 --- a/doc/Makefile +++ b/doc/Makefile @@ -29,7 +29,7 @@ ALLSPHINXOPTS = -WT --keep-going -d build/doctrees $(PAPEROPT_$(PAPER)) \ .PHONY: help clean html web pickle htmlhelp latex changes linkcheck \ dist dist-build gitwash-update version-check html-build latex-build \ - merge-doc + merge-doc show #------------------------------------------------------------------------------ @@ -46,6 +46,7 @@ help: @echo " gitwash-update GITWASH=path/to/gitwash update gitwash developer docs" @echo " upload USERNAME=... RELEASE=... to upload built docs to docs.scipy.org" @echo " merge-doc TAG=... to clone numpy/doc and archive documentation into it" + @echo " show to show the html output in a browser" clean: -rm -rf build/* @@ -103,11 +104,11 @@ dist: build/dist.tar.gz build/dist.tar.gz: make $(DIST_VARS) real-dist -real-dist: dist-build html-build html-scipyorg +real-dist: dist-build html-build test -d build/latex || make latex-build make -C build/latex all-pdf -rm -rf build/dist - cp -r build/html-scipyorg build/dist + cp -r build/html build/dist cd build/html && zip -9r ../dist/numpy-html.zip . cp build/latex/numpy-ref.pdf build/dist cp build/latex/numpy-user.pdf build/dist @@ -185,7 +186,7 @@ html-scipyorg: mkdir -p build/html build/doctrees $(SPHINXBUILD) -t scipyorg -b html $(ALLSPHINXOPTS) build/html-scipyorg $(FILES) @echo - @echo "Build finished. The HTML pages are in build/html." + @echo "Build finished. The HTML pages are in build/html-scipyorg." pickle: generate version-check mkdir -p build/pickle build/doctrees @@ -253,3 +254,7 @@ info: @echo "Running Texinfo files through makeinfo..." make -C build/texinfo info @echo "makeinfo finished; the Info files are in build/texinfo." + +show: + @python -c "import webbrowser; webbrowser.open_new_tab('file://$(PWD)/build/html/index.html')" + diff --git a/doc/RELEASE_WALKTHROUGH.rst.txt b/doc/RELEASE_WALKTHROUGH.rst.txt index bae4b838d..0ff9ff933 100644 --- a/doc/RELEASE_WALKTHROUGH.rst.txt +++ b/doc/RELEASE_WALKTHROUGH.rst.txt @@ -1,7 +1,10 @@ -This file contains a walkthrough of the NumPy 1.14.5 release on Linux. +This file contains a walkthrough of the NumPy 1.14.5 release on Linux, modified +for building on azure and uploading to anaconda.org The commands can be copied into the command line, but be sure to replace 1.14.5 by the correct version. +This should be read together with the general directions in `releasing`. + Release Walkthrough ==================== @@ -90,7 +93,7 @@ Build wheels ------------ Trigger the wheels build by pointing the numpy-wheels repository at this -commit. This can take a while. The numpy-wheels repository is cloned from +commit. This can take up to an hour. The numpy-wheels repository is cloned from `<https://github.com/MacPython/numpy-wheels>`_. Start with a pull as the repo may have been accessed and changed by someone else and a push will fail:: @@ -99,25 +102,26 @@ may have been accessed and changed by someone else and a push will fail:: $ git branch <new version> # only when starting new numpy version $ git checkout v1.14.x # v1.14.x already existed for the 1.14.4 release -Edit the ``.travis.yml`` and ``.appveyor.yml`` files to make sure they have the -correct version, and put in the commit hash for the ``REL`` commit created -above for ``BUILD_COMMIT``, see the _example from `v1.14.3`:: +Edit the ``azure/posix.yml`` and ``azure/windows.yml`` files to make sure they +have the correct version, and put in the commit hash for the ``REL`` commit +created above for ``BUILD_COMMIT``, see an _example:: - $ gvim .travis.yml .appveyor.yml + $ gvim azure/posix.yml azure/windows.yml $ git commit -a $ git push upstream HEAD Now wait. If you get nervous at the amount of time taken -- the builds can take -several hours-- you can check the build progress by following the links -provided at `<https://github.com/MacPython/numpy-wheels>`_ to check the travis -and appveyor build status. Check if all the needed wheels have been built and -uploaded before proceeding. There should currently be 22 of them at -`<https://wheels.scipy.org>`_, 4 for Mac, 8 for Windows, and 10 for Linux. -Note that sometimes builds, like tests, fail for unrelated reasons and you will -need to restart them. +a while -- you can check the build progress by following the links +provided at `<https://github.com/MacPython/numpy-wheels>`_ to check the +build status. Check if all the needed wheels have been built and +uploaded before proceeding. There should currently be 21 of them at +`<https://anaconda.org/multibuild-wheels-staging/numpy/files>`_, 3 for Mac, 6 +for Windows, and 12 for Linux. -.. example_: https://github.com/MacPython/numpy-wheels/commit/fed9c04629c155e7804282eb803d81097244598d +.. example_: https://github.com/MacPython/numpy-wheels/pull/80/commits/cbf4af4 +Note that sometimes builds, like tests, fail for unrelated reasons and +you will need to restart them. Download wheels --------------- @@ -131,7 +135,7 @@ upload later using ``twine``:: $ cd ../terryfy $ git pull upstream master - $ CDN_URL=https://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a43.ssl.cf2.rackcdn.com + $ CDN_URL=https://anaconda.org/multibuild-wheels-staging/numpy/files $ NPY_WHLS=../numpy/release/installers $ ./wheel-uploader -u $CDN_URL -n -v -w $NPY_WHLS -t win numpy 1.14.5 $ ./wheel-uploader -u $CDN_URL -n -v -w $NPY_WHLS -t manylinux1 numpy 1.14.5 diff --git a/doc/TESTS.rst.txt b/doc/TESTS.rst.txt index 007840b39..4ab39c586 100644 --- a/doc/TESTS.rst.txt +++ b/doc/TESTS.rst.txt @@ -12,7 +12,7 @@ the `pytest`_ framework. The older framework is still maintained in order to support downstream projects that use the old numpy framework, but all tests for NumPy should use pytest. -Our goal is that every module and package in SciPy and NumPy +Our goal is that every module and package in NumPy should have a thorough set of unit tests. These tests should exercise the full functionality of a given routine as well as its robustness to erroneous or unexpected input @@ -28,26 +28,30 @@ is found in a routine, you should write a new test for that specific case and add it to the test suite to prevent that bug from creeping back in unnoticed. -To run SciPy's full test suite, use the following:: +.. note:: - >>> import scipy - >>> scipy.test() + SciPy uses the testing framework from :mod:`numpy.testing`, + so all of the NumPy examples shown below are also applicable to SciPy -or from the command line:: +Testing NumPy +''''''''''''' - $ python runtests.py +NumPy can be tested in a number of ways, choose any way you feel comfortable. + +Running tests from inside Python +-------------------------------- -SciPy uses the testing framework from :mod:`numpy.testing`, so all -the SciPy examples shown here are also applicable to NumPy. NumPy's full test -suite can be run as follows:: +You can test an installed NumPy by `numpy.test`, for example, +To run NumPy's full test suite, use the following:: >>> import numpy - >>> numpy.test() + >>> numpy.test(label='slow') -The test method may take two or more arguments; the first, ``label`` is a -string specifying what should be tested and the second, ``verbose`` is an -integer giving the level of output verbosity. See the docstring for -numpy.test for details. The default value for ``label`` is 'fast' - which +The test method may take two or more arguments; the first ``label`` is a +string specifying what should be tested and the second ``verbose`` is an +integer giving the level of output verbosity. See the docstring +`numpy.test` +for details. The default value for ``label`` is 'fast' - which will run the standard tests. The string 'full' will run the full battery of tests, including those identified as being slow to run. If ``verbose`` is 1 or less, the tests will just show information messages about the tests @@ -55,38 +59,43 @@ that are run; but if it is greater than 1, then the tests will also provide warnings on missing tests. So if you want to run every test and get messages about which modules don't have tests:: - >>> scipy.test(label='full', verbose=2) # or scipy.test('full', 2) + >>> numpy.test(label='full', verbose=2) # or numpy.test('full', 2) -Finally, if you are only interested in testing a subset of SciPy, for -example, the ``integrate`` module, use the following:: +Finally, if you are only interested in testing a subset of NumPy, for +example, the ``core`` module, use the following:: - >>> scipy.integrate.test() + >>> numpy.core.test() -or from the command line:: +Running tests from the command line +----------------------------------- - $python runtests.py -t scipy/integrate/tests +If you want to build NumPy in order to work on NumPy itself, use +``runtests.py``.To run NumPy's full test suite:: -The rest of this page will give you a basic idea of how to add unit -tests to modules in SciPy. It is extremely important for us to have -extensive unit testing since this code is going to be used by -scientists and researchers and is being developed by a large number of -people spread across the world. So, if you are writing a package that -you'd like to become part of SciPy, please write the tests as you -develop the package. Also since much of SciPy is legacy code that was -originally written without unit tests, there are still several modules -that don't have tests yet. Please feel free to choose one of these -modules and develop tests for it as you read through -this introduction. + $ python runtests.py + +Testing a subset of NumPy:: + + $python runtests.py -t numpy/core/tests + +For detailed info on testing, see :ref:`testing-builds` + +Other methods of running tests +------------------------------ + +Run tests using your favourite IDE such as `vscode`_ or `pycharm`_ Writing your own tests '''''''''''''''''''''' -Every Python module, extension module, or subpackage in the SciPy +If you are writing a package that you'd like to become part of NumPy, +please write the tests as you develop the package. +Every Python module, extension module, or subpackage in the NumPy package directory should have a corresponding ``test_<name>.py`` file. -Pytest examines these files for test methods (named test*) and test -classes (named Test*). +Pytest examines these files for test methods (named ``test*``) and test +classes (named ``Test*``). -Suppose you have a SciPy module ``scipy/xxx/yyy.py`` containing a +Suppose you have a NumPy module ``numpy/xxx/yyy.py`` containing a function ``zzz()``. To test this function you would create a test module called ``test_yyy.py``. If you only need to test one aspect of ``zzz``, you can simply add a test function:: @@ -100,7 +109,7 @@ a test class:: from numpy.testing import assert_, assert_raises # import xxx symbols - from scipy.xxx.yyy import zzz + from numpy.xxx.yyy import zzz class TestZzz: def test_simple(self): @@ -119,6 +128,11 @@ that makes it hard to identify the test from the output of running the test suite with ``verbose=2`` (or similar verbosity setting). Use plain comments (``#``) if necessary. +Also since much of NumPy is legacy code that was +originally written without unit tests, there are still several modules +that don't have tests yet. Please feel free to choose one of these +modules and develop tests for it. + Labeling tests -------------- @@ -126,8 +140,8 @@ As an alternative to ``pytest.mark.<label>``, there are a number of labels you can use. Unlabeled tests like the ones above are run in the default -``scipy.test()`` run. If you want to label your test as slow - and -therefore reserved for a full ``scipy.test(label='full')`` run, you +``numpy.test()`` run. If you want to label your test as slow - and +therefore reserved for a full ``numpy.test(label='full')`` run, you can label it with a decorator:: # numpy.testing module includes 'import decorators as dec' @@ -211,10 +225,10 @@ for numpy.lib:: >>> np.lib.test(doctests=True) The doctests are run as if they are in a fresh Python instance which -has executed ``import numpy as np``. Tests that are part of a SciPy +has executed ``import numpy as np``. Tests that are part of a NumPy subpackage will have that subpackage already imported. E.g. for a test -in ``scipy/linalg/tests/``, the namespace will be created such that -``from scipy import linalg`` has already executed. +in ``numpy/linalg/tests/``, the namespace will be created such that +``from numpy import linalg`` has already executed. ``tests/`` @@ -223,15 +237,15 @@ in ``scipy/linalg/tests/``, the namespace will be created such that Rather than keeping the code and the tests in the same directory, we put all the tests for a given subpackage in a ``tests/`` subdirectory. For our example, if it doesn't already exist you will -need to create a ``tests/`` directory in ``scipy/xxx/``. So the path -for ``test_yyy.py`` is ``scipy/xxx/tests/test_yyy.py``. +need to create a ``tests/`` directory in ``numpy/xxx/``. So the path +for ``test_yyy.py`` is ``numpy/xxx/tests/test_yyy.py``. -Once the ``scipy/xxx/tests/test_yyy.py`` is written, its possible to +Once the ``numpy/xxx/tests/test_yyy.py`` is written, its possible to run the tests by going to the ``tests/`` directory and typing:: python test_yyy.py -Or if you add ``scipy/xxx/tests/`` to the Python path, you could run +Or if you add ``numpy/xxx/tests/`` to the Python path, you could run the tests interactively in the interpreter like this:: >>> import test_yyy @@ -256,20 +270,20 @@ section of your setup.py:: ... def configuration(parent_package='', top_path=None): ... - config.add_data_dir('tests') + config.add_subpackage('tests') return config ... Now you can do the following to test your module:: - >>> import scipy - >>> scipy.xxx.test() + >>> import numpy + >>> numpy.xxx.test() -Also, when invoking the entire SciPy test suite, your tests will be +Also, when invoking the entire NumPy test suite, your tests will be found and run:: - >>> import scipy - >>> scipy.test() + >>> import numpy + >>> numpy.test() # your tests are included and run automatically! Tips & Tricks @@ -370,7 +384,14 @@ failures without requiring a fixed seed, reporting *minimal* examples for each failure, and better-than-naive-random techniques for triggering bugs. +Documentation for ``numpy.test`` +-------------------------------- + +.. autofunction:: numpy.test + .. _nose: https://nose.readthedocs.io/en/latest/ .. _pytest: https://pytest.readthedocs.io .. _parameterization: https://docs.pytest.org/en/latest/parametrize.html .. _Hypothesis: https://hypothesis.readthedocs.io/en/latest/ +.. _vscode: https://code.visualstudio.com/docs/python/testing#_enable-a-test-framework +.. _pycharm: https://www.jetbrains.com/help/pycharm/testing-your-first-python-application.html diff --git a/doc/changelog/1.18.3-changelog.rst b/doc/changelog/1.18.3-changelog.rst new file mode 100644 index 000000000..6ed2d4851 --- /dev/null +++ b/doc/changelog/1.18.3-changelog.rst @@ -0,0 +1,24 @@ + +Contributors +============ + +A total of 6 people contributed to this release. People with a "+" by their +names contributed a patch for the first time. + +* Charles Harris +* Max Balandat + +* @Mibu287 + +* Pan Jan + +* Sebastian Berg +* @panpiort8 + + +Pull requests merged +==================== + +A total of 5 pull requests were merged for this release. + +* `#15916 <https://github.com/numpy/numpy/pull/15916>`__: BUG: Fix eigh and cholesky methods of numpy.random.multivariate_normal +* `#15929 <https://github.com/numpy/numpy/pull/15929>`__: BUG,MAINT: Remove incorrect special case in string to number... +* `#15930 <https://github.com/numpy/numpy/pull/15930>`__: BUG: Guarantee array is in valid state after memory error occurs... +* `#15954 <https://github.com/numpy/numpy/pull/15954>`__: BUG: Check that `pvals` is 1D in `_generator.multinomial`. +* `#16017 <https://github.com/numpy/numpy/pull/16017>`__: BUG: Alpha parameter must be 1D in `generator.dirichlet` diff --git a/doc/changelog/1.18.4-changelog.rst b/doc/changelog/1.18.4-changelog.rst new file mode 100644 index 000000000..f3524b5f5 --- /dev/null +++ b/doc/changelog/1.18.4-changelog.rst @@ -0,0 +1,23 @@ + +Contributors +============ + +A total of 4 people contributed to this release. People with a "+" by their +names contributed a patch for the first time. + +* Charles Harris +* Matti Picus +* Sebastian Berg +* Warren Weckesser + +Pull requests merged +==================== + +A total of 6 pull requests were merged for this release. + +* `#16055 <https://github.com/numpy/numpy/pull/16055>`__: BLD: add i686 for 1.18 builds +* `#16090 <https://github.com/numpy/numpy/pull/16090>`__: BUG: random: ``Generator.integers(2**32)`` always returned 0. +* `#16091 <https://github.com/numpy/numpy/pull/16091>`__: BLD: fix path to libgfortran on macOS +* `#16109 <https://github.com/numpy/numpy/pull/16109>`__: REV: Reverts side-effect changes to casting +* `#16114 <https://github.com/numpy/numpy/pull/16114>`__: BLD: put openblas library in local directory on windows +* `#16132 <https://github.com/numpy/numpy/pull/16132>`__: DOC: Change import error "howto" to link to new troubleshooting... diff --git a/doc/neps/Makefile b/doc/neps/Makefile index 799e86888..4bbe4b42a 100644 --- a/doc/neps/Makefile +++ b/doc/neps/Makefile @@ -18,7 +18,7 @@ help: .PHONY: help Makefile index index: - python tools/build_index.py + python3 tools/build_index.py # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). diff --git a/doc/neps/index.rst.tmpl b/doc/neps/index.rst.tmpl index 4c5b7766f..0299f8671 100644 --- a/doc/neps/index.rst.tmpl +++ b/doc/neps/index.rst.tmpl @@ -29,6 +29,9 @@ Meta-NEPs (NEPs about NEPs or Processes) nep-template + +{% if has_provisional %} + Provisional NEPs (provisionally accepted; interface may change) --------------------------------------------------------------- @@ -39,6 +42,8 @@ Provisional NEPs (provisionally accepted; interface may change) {{ tags['Title'] }} <{{ tags['Filename'] }}> {% endfor %} +{% endif %} + Accepted NEPs (implementation in progress) ------------------------------------------ diff --git a/doc/neps/nep-0000.rst b/doc/neps/nep-0000.rst index 56a332626..dcc7f4cf8 100644 --- a/doc/neps/nep-0000.rst +++ b/doc/neps/nep-0000.rst @@ -1,3 +1,5 @@ +.. _NEP00: + =========================== NEP 0 — Purpose and Process =========================== diff --git a/doc/neps/nep-0001-npy-format.rst b/doc/neps/nep-0001-npy-format.rst index 4eded02ff..3a28247ab 100644 --- a/doc/neps/nep-0001-npy-format.rst +++ b/doc/neps/nep-0001-npy-format.rst @@ -1,3 +1,5 @@ +.. _NEP01: + ============================================= NEP 1 — A Simple File Format for NumPy Arrays ============================================= diff --git a/doc/neps/nep-0002-warnfix.rst b/doc/neps/nep-0002-warnfix.rst index 207dfa3d4..dfccd5ab8 100644 --- a/doc/neps/nep-0002-warnfix.rst +++ b/doc/neps/nep-0002-warnfix.rst @@ -1,3 +1,5 @@ +.. _NEP02: + ================================================================================= NEP 2 — A proposal to build numpy without warning with a big set of warning flags ================================================================================= diff --git a/doc/neps/nep-0003-math_config_clean.rst b/doc/neps/nep-0003-math_config_clean.rst index ebd32b124..ff5a325fc 100644 --- a/doc/neps/nep-0003-math_config_clean.rst +++ b/doc/neps/nep-0003-math_config_clean.rst @@ -1,3 +1,5 @@ +.. _NEP03: + ===================================================== NEP 3 — Cleaning the math configuration of numpy.core ===================================================== diff --git a/doc/neps/nep-0004-datetime-proposal3.rst b/doc/neps/nep-0004-datetime-proposal3.rst index b32964e88..78b139dc5 100644 --- a/doc/neps/nep-0004-datetime-proposal3.rst +++ b/doc/neps/nep-0004-datetime-proposal3.rst @@ -1,3 +1,5 @@ +.. _NEP04: + ========================================================================= NEP 4 — A (third) proposal for implementing some date/time types in NumPy ========================================================================= diff --git a/doc/neps/nep-0005-generalized-ufuncs.rst b/doc/neps/nep-0005-generalized-ufuncs.rst index 366e26ffd..43459a555 100644 --- a/doc/neps/nep-0005-generalized-ufuncs.rst +++ b/doc/neps/nep-0005-generalized-ufuncs.rst @@ -1,3 +1,5 @@ +.. _NEP05: + ======================================= NEP 5 — Generalized Universal Functions ======================================= diff --git a/doc/neps/nep-0006-newbugtracker.rst b/doc/neps/nep-0006-newbugtracker.rst index 8dc7a1d8e..cb13f7882 100644 --- a/doc/neps/nep-0006-newbugtracker.rst +++ b/doc/neps/nep-0006-newbugtracker.rst @@ -1,3 +1,5 @@ +.. _NEP06: + =================================================== NEP 6 — Replacing Trac with a different bug tracker =================================================== diff --git a/doc/neps/nep-0007-datetime-proposal.rst b/doc/neps/nep-0007-datetime-proposal.rst index 5547a4306..8f6c52737 100644 --- a/doc/neps/nep-0007-datetime-proposal.rst +++ b/doc/neps/nep-0007-datetime-proposal.rst @@ -1,3 +1,5 @@ +.. _NEP07: + ================================================================== NEP 7 — A proposal for implementing some date/time types in NumPy ================================================================== diff --git a/doc/neps/nep-0008-groupby_additions.rst b/doc/neps/nep-0008-groupby_additions.rst index 3189fcf41..89d454914 100644 --- a/doc/neps/nep-0008-groupby_additions.rst +++ b/doc/neps/nep-0008-groupby_additions.rst @@ -1,3 +1,5 @@ +.. _NEP08: + ============================================================= NEP 8 — A proposal for adding groupby functionality to NumPy ============================================================= diff --git a/doc/neps/nep-0009-structured_array_extensions.rst b/doc/neps/nep-0009-structured_array_extensions.rst index 8b81a308d..cd6c3f6c3 100644 --- a/doc/neps/nep-0009-structured_array_extensions.rst +++ b/doc/neps/nep-0009-structured_array_extensions.rst @@ -1,3 +1,5 @@ +.. _NEP09: + =================================== NEP 9 — Structured array extensions =================================== diff --git a/doc/neps/nep-0010-new-iterator-ufunc.rst b/doc/neps/nep-0010-new-iterator-ufunc.rst index fd7b3e52c..87617f414 100644 --- a/doc/neps/nep-0010-new-iterator-ufunc.rst +++ b/doc/neps/nep-0010-new-iterator-ufunc.rst @@ -1,3 +1,5 @@ +.. _NEP10: + ============================================== NEP 10 — Optimizing Iterator/UFunc Performance ============================================== @@ -1291,7 +1293,7 @@ Construction and Destruction Returns ``NPY_SUCCEED`` or ``NPY_FAIL``. -``int NpyIter_HasInnerLoop(NpyIter *iter`` +``int NpyIter_HasInnerLoop(NpyIter *iter)`` Returns 1 if the iterator handles the inner loop, or 0 if the caller needs to handle it. This is controlled diff --git a/doc/neps/nep-0011-deferred-ufunc-evaluation.rst b/doc/neps/nep-0011-deferred-ufunc-evaluation.rst index a7143c6ee..c40ca56d7 100644 --- a/doc/neps/nep-0011-deferred-ufunc-evaluation.rst +++ b/doc/neps/nep-0011-deferred-ufunc-evaluation.rst @@ -1,3 +1,5 @@ +.. _NEP11: + ================================== NEP 11 — Deferred UFunc Evaluation ================================== diff --git a/doc/neps/nep-0012-missing-data.rst b/doc/neps/nep-0012-missing-data.rst index dbcf1b579..0adcea69a 100644 --- a/doc/neps/nep-0012-missing-data.rst +++ b/doc/neps/nep-0012-missing-data.rst @@ -1,3 +1,5 @@ +.. _NEP12: + ============================================ NEP 12 — Missing Data Functionality in NumPy ============================================ @@ -313,7 +315,7 @@ The following works in the current draft implementation:: For floating point numbers, Inf and NaN are separate concepts from missing values. If a division by zero occurs in an array with default missing value support, an unmasked Inf or NaN will be produced. To -mask those values, a further 'a[np.logical_not(a.isfinite(a)] = np.NA' +mask those values, a further 'a[np.logical_not(a.isfinite(a))] = np.NA' can achieve that. For the bitpattern approach, the parameterized dtype('NA[f8,InfNan]') described in a later section can be used to get these semantics without the extra manipulation. @@ -926,7 +928,7 @@ to access the array elements. This python indexing still goes through the Python API, so the NA handling and error checking in numpy still can work like normal and fail if the inputs have NAs which cannot fit in the output array. In this case it fails when trying to convert the NA into an integer -to set in in the output. +to set in the output. The next version of the code introduces more efficient indexing. This operates based on Python's buffer protocol. This causes Cython to call diff --git a/doc/neps/nep-0013-ufunc-overrides.rst b/doc/neps/nep-0013-ufunc-overrides.rst index 0888c7559..5c28e9446 100644 --- a/doc/neps/nep-0013-ufunc-overrides.rst +++ b/doc/neps/nep-0013-ufunc-overrides.rst @@ -1,3 +1,5 @@ +.. _NEP13: + ========================================== NEP 13 — A Mechanism for Overriding Ufuncs ========================================== diff --git a/doc/neps/nep-0014-dropping-python2.7-proposal.rst b/doc/neps/nep-0014-dropping-python2.7-proposal.rst index 3adf3b407..dfc09d254 100644 --- a/doc/neps/nep-0014-dropping-python2.7-proposal.rst +++ b/doc/neps/nep-0014-dropping-python2.7-proposal.rst @@ -1,3 +1,5 @@ +.. _NEP14: + ============================================= NEP 14 — Plan for dropping Python 2.7 support ============================================= diff --git a/doc/neps/nep-0015-merge-multiarray-umath.rst b/doc/neps/nep-0015-merge-multiarray-umath.rst index 576a21e23..1efceb957 100644 --- a/doc/neps/nep-0015-merge-multiarray-umath.rst +++ b/doc/neps/nep-0015-merge-multiarray-umath.rst @@ -1,3 +1,5 @@ +.. _NEP15: + ===================================== NEP 15 — Merging multiarray and umath ===================================== diff --git a/doc/neps/nep-0016-abstract-array.rst b/doc/neps/nep-0016-abstract-array.rst index 7551b11b9..63ad600e9 100644 --- a/doc/neps/nep-0016-abstract-array.rst +++ b/doc/neps/nep-0016-abstract-array.rst @@ -1,3 +1,5 @@ +.. _NEP16: + ============================================================= NEP 16 — An abstract base class for identifying "duck arrays" ============================================================= diff --git a/doc/neps/nep-0016-benchmark.py b/doc/neps/nep-0016-benchmark.py index ec8e44726..af6059114 100644 --- a/doc/neps/nep-0016-benchmark.py +++ b/doc/neps/nep-0016-benchmark.py @@ -1,3 +1,5 @@ +.. _NEP16: + import perf import abc import numpy as np diff --git a/doc/neps/nep-0017-split-out-maskedarray.rst b/doc/neps/nep-0017-split-out-maskedarray.rst index 7ef949763..151c5ad1a 100644 --- a/doc/neps/nep-0017-split-out-maskedarray.rst +++ b/doc/neps/nep-0017-split-out-maskedarray.rst @@ -1,3 +1,5 @@ +.. _NEP17: + ================================ NEP 17 — Split Out Masked Arrays ================================ diff --git a/doc/neps/nep-0018-array-function-protocol.rst b/doc/neps/nep-0018-array-function-protocol.rst index fb9b838b5..0dcb0ff7e 100644 --- a/doc/neps/nep-0018-array-function-protocol.rst +++ b/doc/neps/nep-0018-array-function-protocol.rst @@ -1,3 +1,5 @@ +.. _NEP18: + ==================================================================== NEP 18 — A dispatch mechanism for NumPy's high level array functions ==================================================================== @@ -7,7 +9,7 @@ NEP 18 — A dispatch mechanism for NumPy's high level array functions :Author: Marten van Kerkwijk <mhvk@astro.utoronto.ca> :Author: Hameer Abbasi <hameerabbasi@yahoo.com> :Author: Eric Wieser <wieser.eric@gmail.com> -:Status: Provisional +:Status: Final :Type: Standards Track :Created: 2018-05-29 :Updated: 2019-05-25 diff --git a/doc/neps/nep-0019-rng-policy.rst b/doc/neps/nep-0019-rng-policy.rst index 4f766fa2d..077997f43 100644 --- a/doc/neps/nep-0019-rng-policy.rst +++ b/doc/neps/nep-0019-rng-policy.rst @@ -1,3 +1,5 @@ +.. _NEP19: + ======================================= NEP 19 — Random Number Generator Policy ======================================= diff --git a/doc/neps/nep-0020-gufunc-signature-enhancement.rst b/doc/neps/nep-0020-gufunc-signature-enhancement.rst index a7a992cf1..90ed930b4 100644 --- a/doc/neps/nep-0020-gufunc-signature-enhancement.rst +++ b/doc/neps/nep-0020-gufunc-signature-enhancement.rst @@ -1,3 +1,5 @@ +.. _NEP20: + =============================================================== NEP 20 — Expansion of Generalized Universal Function Signatures =============================================================== diff --git a/doc/neps/nep-0021-advanced-indexing.rst b/doc/neps/nep-0021-advanced-indexing.rst index 8e525b0cb..7751d309b 100644 --- a/doc/neps/nep-0021-advanced-indexing.rst +++ b/doc/neps/nep-0021-advanced-indexing.rst @@ -1,3 +1,5 @@ +.. _NEP21: + ================================================== NEP 21 — Simplified and explicit advanced indexing ================================================== @@ -48,7 +50,7 @@ NumPy arrays currently support a flexible range of indexing operations: For clarity, we will refer to these existing rules as "legacy indexing". This is only a high-level summary; for more details, see NumPy's documentation -and and `Examples` below. +and `Examples` below. Outer indexing ~~~~~~~~~~~~~~ diff --git a/doc/neps/nep-0022-ndarray-duck-typing-overview.rst b/doc/neps/nep-0022-ndarray-duck-typing-overview.rst index 077166453..47b81d9e7 100644 --- a/doc/neps/nep-0022-ndarray-duck-typing-overview.rst +++ b/doc/neps/nep-0022-ndarray-duck-typing-overview.rst @@ -1,3 +1,5 @@ +.. _NEP22: + =========================================================== NEP 22 — Duck typing for NumPy arrays – high level overview =========================================================== @@ -23,7 +25,7 @@ Detailed description -------------------- Traditionally, NumPy’s ``ndarray`` objects have provided two things: a -high level API for expression operations on homogenously-typed, +high level API for expression operations on homogeneously-typed, arbitrary-dimensional, array-structured data, and a concrete implementation of the API based on strided in-RAM storage. The API is powerful, fairly general, and used ubiquitously across the scientific diff --git a/doc/neps/nep-0023-backwards-compatibility.rst b/doc/neps/nep-0023-backwards-compatibility.rst index 92974ad6e..158f46273 100644 --- a/doc/neps/nep-0023-backwards-compatibility.rst +++ b/doc/neps/nep-0023-backwards-compatibility.rst @@ -1,3 +1,5 @@ +.. _NEP23: + ======================================================= NEP 23 — Backwards compatibility and deprecation policy ======================================================= diff --git a/doc/neps/nep-0024-missing-data-2.rst b/doc/neps/nep-0024-missing-data-2.rst index f4414e0a0..8e63629f3 100644 --- a/doc/neps/nep-0024-missing-data-2.rst +++ b/doc/neps/nep-0024-missing-data-2.rst @@ -1,3 +1,5 @@ +.. _NEP24: + ============================================================= NEP 24 — Missing Data Functionality - Alternative 1 to NEP 12 ============================================================= diff --git a/doc/neps/nep-0025-missing-data-3.rst b/doc/neps/nep-0025-missing-data-3.rst index 6343759e8..19be3cf1e 100644 --- a/doc/neps/nep-0025-missing-data-3.rst +++ b/doc/neps/nep-0025-missing-data-3.rst @@ -1,3 +1,5 @@ +.. _NEP25: + ====================================== NEP 25 — NA support via special dtypes ====================================== diff --git a/doc/neps/nep-0026-missing-data-summary.rst b/doc/neps/nep-0026-missing-data-summary.rst index 78fe999df..f25ce9b91 100644 --- a/doc/neps/nep-0026-missing-data-summary.rst +++ b/doc/neps/nep-0026-missing-data-summary.rst @@ -1,3 +1,5 @@ +.. _NEP26: + ==================================================== NEP 26 — Summary of Missing Data NEPs and discussion ==================================================== @@ -8,7 +10,7 @@ NEP 26 — Summary of Missing Data NEPs and discussion :Created: 2012-04-22 *Context*: this NEP was written as summary of the large number of discussions -and proposals (`NEP 12`_, `NEP 24`_, `NEP 25`_), regarding missing data +and proposals (:ref:`NEP12`, :ref:`NEP24`, :ref:`NEP25`), regarding missing data functionality. The debate about how NumPy should handle missing data, a subject with @@ -349,7 +351,7 @@ dtypes can arrange for certain bitpatterns to be given NA semantics. One option is to copy numpy.ma closely, but with a more optimized implementation. (Or to simply optimize the existing implementation.) -One option is that described in `NEP 12`_, for which an implementation +One option is that described in `NEP12`, for which an implementation of mask-based missing data exists. This system is roughly: * There is both bitpattern and mask-based missing data, and both @@ -364,7 +366,7 @@ of mask-based missing data exists. This system is roughly: a bitpattern NA to an array which supports both requires accessing the data by "peeking under the mask". -Another option is that described in `NEP 24`_, which is to implement +Another option is that described in `NEP24`, which is to implement bitpattern dtypes with NA semantics for the "statistical missing data" use case, and to also implement a totally independent API for masked arrays with ignore semantics and all mask manipulation done explicitly @@ -704,14 +706,14 @@ risk of reducing developer contribution. References and Footnotes ------------------------ -`NEP 12`_ describes Mark's NA-semantics/mask implementation/view based mask +:ref:`NEP12` describes Mark's NA-semantics/mask implementation/view based mask handling API. -`NEP 24`_ ("the alterNEP") was Nathaniel's initial attempt at separating MISSING +:ref:`NEP24` ("the alterNEP") was Nathaniel's initial attempt at separating MISSING and IGNORED handling into bit-patterns versus masks, though there's a bunch he would change about the proposal at this point. -`NEP 25`_ ("miniNEP 2") was a later attempt by Nathaniel to sketch out an +:ref:`NEP25` ("miniNEP 2") was a later attempt by Nathaniel to sketch out an implementation strategy for NA dtypes. A further discussion overview page can be found at: @@ -722,9 +724,3 @@ Copyright --------- This document has been placed in the public domain. - -.. _NEP 12: http://www.numpy.org/neps/nep-0012-missing-data.html - -.. _NEP 24: http://www.numpy.org/neps/nep-0024-missing-data-2.html - -.. _NEP 25: http://www.numpy.org/neps/nep-0025-missing-data-3.html diff --git a/doc/neps/nep-0027-zero-rank-arrarys.rst b/doc/neps/nep-0027-zero-rank-arrarys.rst index 430397235..2d152234c 100644 --- a/doc/neps/nep-0027-zero-rank-arrarys.rst +++ b/doc/neps/nep-0027-zero-rank-arrarys.rst @@ -1,3 +1,5 @@ +.. _NEP27: + ========================= NEP 27 — Zero Rank Arrays ========================= diff --git a/doc/neps/nep-0028-website-redesign.rst b/doc/neps/nep-0028-website-redesign.rst index dcd182d55..e27da91c6 100644 --- a/doc/neps/nep-0028-website-redesign.rst +++ b/doc/neps/nep-0028-website-redesign.rst @@ -1,3 +1,5 @@ +.. _NEP28: + =================================== NEP 28 — numpy.org website redesign =================================== @@ -131,7 +133,7 @@ Possible options for static site generators Unlike the previous options, Docusaurus doesn't come with themes, and thus we would not want to use this for our landing page. This is an excellent docs option written in React. Docusaurus natively has support for i18n (via - Crowdin_, document versioning, and document search. + Crowdin_), document versioning, and document search. Both Jekyll and Hugo are excellent options that should be supported into the future and are good choices for NumPy. Docusaurus has several bonus features @@ -153,7 +155,7 @@ significant drain on the time of maintainers. 2. *Github Pages.* Github Pages also has a 100GB bandwidth limit, and is unclear if additional bandwidth can be purchased. It is also unclear where sites are deployed, and should be assumed sites aren't deployed globally. Github Pages has an easy to - use CI & DNS, similar to to Netlify. HTTPS is supported. + use CI & DNS, similar to Netlify. HTTPS is supported. 3. *Cloudflare.* An excellent option, additional CI is likely needed for the same ease of deployment. diff --git a/doc/neps/nep-0029-deprecation_policy.rst b/doc/neps/nep-0029-deprecation_policy.rst index bdc0baa39..9b6022a81 100644 --- a/doc/neps/nep-0029-deprecation_policy.rst +++ b/doc/neps/nep-0029-deprecation_policy.rst @@ -1,3 +1,5 @@ +.. _NEP0029: + ================================================================================== NEP 29 — Recommend Python and Numpy version support as a community policy standard ================================================================================== diff --git a/doc/neps/nep-0030-duck-array-protocol.rst b/doc/neps/nep-0030-duck-array-protocol.rst index 635f10165..11a297132 100644 --- a/doc/neps/nep-0030-duck-array-protocol.rst +++ b/doc/neps/nep-0030-duck-array-protocol.rst @@ -1,3 +1,5 @@ +.. _NEP30: + ====================================================== NEP 30 — Duck Typing for NumPy Arrays - Implementation ====================================================== @@ -33,30 +35,32 @@ For the purpose above, NEP 22 introduced the concept of duck typing to NumPy arrays. The suggested solution described in the NEP allows libraries to avoid coercion of a NumPy-like array to a pure NumPy array where necessary, while still allowing that NumPy-like array libraries that do not wish to implement -the protocol to coerce arrays to a pure Numpy array via ``np.asarray``. +the protocol to coerce arrays to a pure NumPy array via ``np.asarray``. Usage Guidance ~~~~~~~~~~~~~~ -Code that uses np.duckarray is meant for supporting other ndarray-like objects +Code that uses ``np.duckarray`` is meant for supporting other ndarray-like objects that "follow the NumPy API". That is an ill-defined concept at the moment -- every known library implements the NumPy API only partly, and many deviate intentionally in at least some minor ways. This cannot be easily remedied, so -for users of ``__duckarray__`` we recommend the following strategy: check if the -NumPy functionality used by the code that follows your use of ``__duckarray__`` +for users of ``np.duckarray`` we recommend the following strategy: check if the +NumPy functionality used by the code that follows your use of ``np.duckarray`` is present in Dask, CuPy and Sparse. If so, it's reasonable to expect any duck array to work here. If not, we suggest you indicate in your docstring what kinds of duck arrays are accepted, or what properties they need to have. To exemplify the usage of duck arrays, suppose one wants to take the ``mean()`` of an array-like object ``arr``. Using NumPy to achieve that, one could write -``np.asarray(arr).mean()`` to achieve the intended result. However, libraries -may expect ``arr`` to be a NumPy-like array, and at the same time, the array may -or may not be an object compliant to the NumPy API (either in full or partially) -such as a CuPy, Sparse or a Dask array. In the case where ``arr`` is already an -object compliant to the NumPy API, we would simply return it (and prevent it -from being coerced into a pure NumPy array), otherwise, it would then be coerced -into a NumPy array. +``np.asarray(arr).mean()`` to achieve the intended result. If ``arr`` is not +a NumPy array, this would create an actual NumPy array in order to call +``.mean()``. However, if the array is an object that is compliant with the NumPy +API (either in full or partially) such as a CuPy, Sparse or a Dask array, then +that copy would have been unnecessary. On the other hand, if one were to use the new +``__duckarray__`` protocol: ``np.duckarray(arr).mean()``, and ``arr`` is an object +compliant with the NumPy API, it would simply be returned rather than coerced +into a pure NumPy array, avoiding unnecessary copies and potential loss of +performance. Implementation -------------- @@ -64,9 +68,9 @@ Implementation The implementation idea is fairly straightforward, requiring a new function ``duckarray`` to be introduced in NumPy, and a new method ``__duckarray__`` in NumPy-like array classes. The new ``__duckarray__`` method shall return the -downstream array-like object itself, such as the ``self`` object. If appropriate, -an ``__array__`` method may be implemented that returns a NumPy array or possibly -raise a ``TypeError`` with a helpful message. +downstream array-like object itself, such as the ``self`` object, while the +``__array__`` method raises ``TypeError``. Alternatively, the ``__array__`` +method could create an actual NumPy array and return that. The new NumPy ``duckarray`` function can be implemented as follows: @@ -91,8 +95,8 @@ a complete implementation would look like the following: return self def __array__(self): - return TypeError("NumPyLikeArray can not be converted to a numpy array. " - "You may want to use np.duckarray.") + raise TypeError("NumPyLikeArray can not be converted to a NumPy " + "array. You may want to use np.duckarray() instead.") The implementation above exemplifies the simplest case, but the overall idea is that libraries will implement a ``__duckarray__`` method that returns the @@ -114,7 +118,7 @@ An example of how the ``__duckarray__`` protocol could be used to write a seen below. The example here was chosen not only to demonstrate the usage of the ``duckarray`` function, but also to demonstrate its dependency on the NumPy API, demonstrated by checks on the array's ``shape`` attribute. Note that the -example is merely a simplified version of NumPy's actually implementation of +example is merely a simplified version of NumPy's actual implementation of ``stack`` working on the first axis, and it is assumed that Dask has implemented the ``__duckarray__`` method. diff --git a/doc/neps/nep-0031-uarray.rst b/doc/neps/nep-0031-uarray.rst index 737d2b456..47d4bdd37 100644 --- a/doc/neps/nep-0031-uarray.rst +++ b/doc/neps/nep-0031-uarray.rst @@ -1,3 +1,5 @@ +.. _NEP31: + ============================================================ NEP 31 — Context-local and global overrides of the NumPy API ============================================================ diff --git a/doc/neps/nep-0032-remove-financial-functions.rst b/doc/neps/nep-0032-remove-financial-functions.rst index a78b11fea..1c3722f46 100644 --- a/doc/neps/nep-0032-remove-financial-functions.rst +++ b/doc/neps/nep-0032-remove-financial-functions.rst @@ -1,3 +1,5 @@ +.. _NEP32: + ================================================== NEP 32 — Remove the financial functions from NumPy ================================================== diff --git a/doc/neps/nep-0034.rst b/doc/neps/nep-0034-infer-dtype-is-object.rst index 2e0ae46f4..a424ab4a3 100644 --- a/doc/neps/nep-0034.rst +++ b/doc/neps/nep-0034-infer-dtype-is-object.rst @@ -1,3 +1,5 @@ +.. _NEP34: + =========================================================== NEP 34 — Disallow inferring ``dtype=object`` from sequences =========================================================== diff --git a/doc/neps/nep-0035-array-creation-dispatch-with-array-function.rst b/doc/neps/nep-0035-array-creation-dispatch-with-array-function.rst new file mode 100644 index 000000000..2d7539952 --- /dev/null +++ b/doc/neps/nep-0035-array-creation-dispatch-with-array-function.rst @@ -0,0 +1,188 @@ +.. _NEP35: + +=========================================================== +NEP 35 — Array Creation Dispatching With __array_function__ +=========================================================== + +:Author: Peter Andreas Entschev <pentschev@nvidia.com> +:Status: Draft +:Type: Standards Track +:Created: 2019-10-15 +:Updated: 2019-10-15 +:Resolution: + +Abstract +-------- + +We propose the introduction of a new keyword argument ``like=`` to all array +creation functions to permit dispatching of such functions by the +``__array_function__`` protocol, addressing one of the protocol shortcomings, +as described by NEP-18 [1]_. + +Detailed description +-------------------- + +The introduction of the ``__array_function__`` protocol allowed downstream +library developers to use NumPy as a dispatching API. However, the protocol +did not -- and did not intend to -- address the creation of arrays by downstream +libraries, preventing those libraries from using such important functionality in +that context. + +Other NEPs have been written to address parts of that limitation, such as the +introduction of the ``__duckarray__`` protocol in NEP-30 [2]_, and the +introduction of an overriding mechanism called ``uarray`` by NEP-31 [3]_. + +The purpose of this NEP is to address that shortcoming in a simple and +straighforward way: introduce a new ``like=`` keyword argument, similar to how +the ``empty_like`` family of functions work. When array creation functions +receive such an argument, they will trigger the ``__array_function__`` protocol, +and call the downstream library's own array creation function implementation. +The ``like=`` argument, as its own name suggests, shall be used solely for the +purpose of identifying where to dispatch. In contrast to the way +``__array_function__`` has been used so far (the first argument identifies where +to dispatch), and to avoid breaking NumPy's API with regards to array creation, +the new ``like=`` keyword shall be used for the purpose of dispatching. + +Usage Guidance +~~~~~~~~~~~~~~ + +The new ``like=`` keyword is solely intended to identify the downstream library +where to dispatch and the object is used only as reference, meaning that no +modifications, copies or processing will be performed on that object. + +We expect that this functionality will be mostly useful to library developers, +allowing them to create new arrays for internal usage based on arrays passed +by the user, preventing unnecessary creation of NumPy arrays that will +ultimately lead to an additional conversion into a downstream array type. + +Implementation +-------------- + +The implementation requires introducing a new ``like=`` keyword to all existing +array creation functions of NumPy. As examples of functions that would add this +new argument (but not limited to) we can cite those taking array-like objects +such as ``array`` and ``asarray``, functions that create arrays based on +numerical ranges such as ``range`` and ``linspace``, as well as the ``empty`` +family of functions, even though that may be redundant, since there exists +already specializations for those with the naming format ``empty_like``. As of +the writing of this NEP, a complete list of array creation functions can be +found in [4]_. + +This newly proposed keyword shall be removed by the ``__array_function__`` +mechanism from the keyword dictionary before dispatching. The purpose for this +is twofold: + +1. The object will have no use in the downstream library's implementation; and +2. Simplifies adoption of array creation by those libraries already opting-in + to implement the ``__array_function__`` protocol, thus removing the + requirement to explicitly opt-in for all array creation functions. + +Downstream libraries thus shall _NOT_ include the ``like=`` keyword to their +array creation APIs, which is a NumPy-exclusive keyword. + +Function Dispatching +~~~~~~~~~~~~~~~~~~~~ + +There are two different cases to dispatch: Python functions, and C functions. +To permit ``__array_function__`` dispatching, one possible implementation is to +decorate Python functions with ``overrides.array_function_dispatch``, but C +functions have a different requirement, which we shall describe shortly. + +The example below shows a suggestion on how the ``asarray`` could be decorated +with ``overrides.array_function_dispatch``: + +.. code:: python + + def _asarray_decorator(a, dtype=None, order=None, like=None): + return (like,) + + @set_module('numpy') + @array_function_dispatch(_asarray_decorator) + def asarray(a, dtype=None, order=None, like=None): + return array(a, dtype, copy=False, order=order) + +Note in the example above that the implementation remains unchanged, the only +difference is the decoration, which uses the new ``_asarray_decorator`` function +to instruct the ``__array_function__`` protocol to dispatch if ``like`` is not +``None``. + +We will now look at a C function example, and since ``asarray`` is anyway a +specialization of ``array``, we will use the latter as an example now. As +``array`` is a C function, currently all NumPy does regarding its Python source +is to import the function and adjust its ``__module__`` to ``numpy``. The +function will now be decorated with a specialization of +``overrides.array_function_from_dispatcher``, which shall take care of adjusting +the module too. + +.. code:: python + + array_function_nodocs_from_c_func_and_dispatcher = functools.partial( + overrides.array_function_from_dispatcher, + module='numpy', docs_from_dispatcher=False, verify=False) + + @array_function_nodocs_from_c_func_and_dispatcher(_multiarray_umath.array) + def array(a, dtype=None, copy=True, order='K', subok=False, ndmin=0, like=None): + return (like,) + +There are two downsides to the implementation above for C functions: + +1. It creates another Python function call; and +2. To follow current implementation standards, documentation should be attached + directly to the Python source code. + +Alternatively for C functions, the implementation of ``like=`` could be moved +into the C implementation itself. This is not the primary suggestion here due +to its inherent complexity which would be difficult too long to describe in its +entirety here, and too tedious for the reader. However, we leave that as an +option open for discussion. + +Usage +----- + +The purpose of this NEP is to keep things simple. Similarly, we can exemplify +the usage of ``like=`` in a simple way. Imagine you have an array of ones +created by a downstream library, such as CuPy. What you need now is a new array +that can be created using the NumPy API, but that will in fact be created by +the downstream library, a simple way to achieve that is shown below. + +.. code:: python + + x = cupy.ones(2) + np.array([1, 3, 5], like=x) # Returns cupy.ndarray + +As a second example, we could also create an array of evenly spaced numbers +using a Dask identity matrix as reference: + +.. code:: python + + x = dask.array.eye(3) + np.linspace(0, 2, like=x) # Returns dask.array + + +Compatibility +------------- + +This proposal does not raise any backward compatibility issues within NumPy, +given that it only introduces a new keyword argument to existing array creation +functions. + +Downstream libraries will benefit from the ``like=`` argument automatically, +that is, without any explicit changes in their codebase. The only requirement +is that they already implement the ``__array_function__`` protocol, as +described by NEP-18 [2]_. + +References and Footnotes +------------------------ + +.. [1] `NEP-18 - A dispatch mechanism for NumPy's high level array functions <https://numpy.org/neps/nep-0018-array-function-protocol.html>`_. + +.. [2] `NEP 30 — Duck Typing for NumPy Arrays - Implementation <https://numpy.org/neps/nep-0030-duck-array-protocol.html>`_. + +.. [3] `NEP 31 — Context-local and global overrides of the NumPy API <https://github.com/numpy/numpy/pull/14389>`_. + +.. [4] `Array creation routines <https://docs.scipy.org/doc/numpy-1.17.0/reference/routines.array-creation.html>`_. + +Copyright +--------- + +This document has been placed in the public domain. diff --git a/doc/neps/nep-0037-array-module.rst b/doc/neps/nep-0037-array-module.rst index 10ff7260b..1e868324d 100644 --- a/doc/neps/nep-0037-array-module.rst +++ b/doc/neps/nep-0037-array-module.rst @@ -1,3 +1,5 @@ +.. _NEP37: + =================================================== NEP 37 — A dispatch protocol for NumPy-like modules =================================================== @@ -13,8 +15,8 @@ Abstract -------- NEP-18's ``__array_function__`` has been a mixed success. Some projects (e.g., -dask, CuPy, xarray, sparse, Pint) have enthusiastically adopted it. Others -(e.g., PyTorch, JAX, SciPy) have been more reluctant. Here we propose a new +dask, CuPy, xarray, sparse, Pint, MXNet) have enthusiastically adopted it. +Others (e.g., JAX) have been more reluctant. Here we propose a new protocol, ``__array_module__``, that we expect could eventually subsume most use-cases for ``__array_function__``. The protocol requires explicit adoption by both users and library authors, which ensures backwards compatibility, and @@ -26,39 +28,40 @@ Why ``__array_function__`` hasn't been enough There are two broad ways in which NEP-18 has fallen short of its goals: -1. **Maintainability concerns**. `__array_function__` has significant +1. **Backwards compatibility concerns**. `__array_function__` has significant implications for libraries that use it: - - Projects like `PyTorch - <https://github.com/pytorch/pytorch/issues/22402>`_, `JAX - <https://github.com/google/jax/issues/1565>`_ and even `scipy.sparse - <https://github.com/scipy/scipy/issues/10362>`_ have been reluctant to - implement `__array_function__` in part because they are concerned about - **breaking existing code**: users expect NumPy functions like + - `JAX <https://github.com/google/jax/issues/1565>`_ has been reluctant + to implement ``__array_function__`` in part because it is concerned about + breaking existing code: users expect NumPy functions like ``np.concatenate`` to return NumPy arrays. This is a fundamental limitation of the ``__array_function__`` design, which we chose to allow overriding the existing ``numpy`` namespace. + Libraries like Dask and CuPy have looked at and accepted the backwards + incompatibility impact of ``__array_function__``; it would still have been + better for them if that impact didn't exist. + + Note that projects like `PyTorch + <https://github.com/pytorch/pytorch/issues/22402>`_ and `scipy.sparse + <https://github.com/scipy/scipy/issues/10362>`_ have also not + adopted ``__array_function__`` yet, because they don't have a + NumPy-compatible API or semantics. In the case of PyTorch, that is likely + to be added in the future. ``scipy.sparse`` is in the same situation as + ``numpy.matrix``: its semantics are not compatible with ``numpy.ndarray`` + and therefore adding ``__array_function__`` (except to return ``NotImplemented`` + perhaps) is not a healthy idea. - ``__array_function__`` currently requires an "all or nothing" approach to implementing NumPy's API. There is no good pathway for **incremental adoption**, which is particularly problematic for established projects for which adopting ``__array_function__`` would result in breaking changes. - - It is no longer possible to use **aliases to NumPy functions** within - modules that support overrides. For example, both CuPy and JAX set - ``result_type = np.result_type``. - - Implementing **fall-back mechanisms** for unimplemented NumPy functions - by using NumPy's implementation is hard to get right (but see the - `version from dask <https://github.com/dask/dask/pull/5043>`_), because - ``__array_function__`` does not present a consistent interface. - Converting all arguments of array type requires recursing into generic - arguments of the form ``*args, **kwargs``. 2. **Limitations on what can be overridden.** ``__array_function__`` has some important gaps, most notably array creation and coercion functions: - **Array creation** routines (e.g., ``np.arange`` and those in ``np.random``) need some other mechanism for indicating what type of - arrays to create. `NEP 36 <https://github.com/numpy/numpy/pull/14715>`_ + arrays to create. `NEP 35 <https://numpy.org/neps/nep-0035-array-creation-dispatch-with-array-function.html>`_ proposed adding optional ``like=`` arguments to functions without existing array arguments. However, we still lack any mechanism to override methods on objects, such as those needed by @@ -71,6 +74,19 @@ There are two broad ways in which NEP-18 has fallen short of its goals: a separate ``np.duckarray`` function, but this still does not resolve how to cast one duck array into a type matching another duck array. +Other maintainability concerns that were raised include: + +- It is no longer possible to use **aliases to NumPy functions** within + modules that support overrides. For example, both CuPy and JAX set + ``result_type = np.result_type`` and now have to wrap use of + ``np.result_type`` in their own ``result_type`` function instead. +- Implementing **fall-back mechanisms** for unimplemented NumPy functions + by using NumPy's implementation is hard to get right (but see the + `version from dask <https://github.com/dask/dask/pull/5043>`_), because + ``__array_function__`` does not present a consistent interface. + Converting all arguments of array type requires recursing into generic + arguments of the form ``*args, **kwargs``. + ``get_array_module`` and the ``__array_module__`` protocol ---------------------------------------------------------- @@ -439,7 +455,7 @@ input arguments: class ArrayModule: def __init__(self, prefer_gpu): self.prefer_gpu = prefer_gpu - + def __getattr__(self, name): import base_module base_func = getattr(base_module, name) @@ -493,23 +509,27 @@ Both ``__array_ufunc__`` and ``__array_function__`` have implicit control over dispatching: the dispatched functions are determined via the appropriate protocols in every function call. This generalizes well to handling many different types of objects, as evidenced by its use for implementing arithmetic -operators in Python, but it has two downsides: - -1. *Speed*: it imposes additional overhead in every function call, because each - function call needs to inspect each of its arguments for overrides. This is - why arithmetic on builtin Python numbers is slow. -2. *Readability*: it is not longer immediately evident to readers of code what - happens when a function is called, because the function's implementation - could be overridden by any of its arguments. - -In contrast, importing a new library (e.g., ``import dask.array as da``) with -an API matching NumPy is entirely explicit. There is no overhead from dispatch -or ambiguity about which implementation is being used. +operators in Python, but it has an important downside for **readability**: +it is not longer immediately evident to readers of code what happens when a +function is called, because the function's implementation could be overridden +by any of its arguments. + +The **speed** implications are: + +- When using a *duck-array type*, ``get_array_module`` means type checking only + needs to happen once inside each function that supports duck typing, whereas + with ``__array_function__`` it happens every time a NumPy function is called. + Obvious it's going to depend on the function, but if a typical duck-array + supporting function calls into other NumPy functions 3-5 times this is a factor + of 3-5x more overhead. +- When using *NumPy arrays*, ``get_array_module`` is one extra call per + function (``__array_function__`` overhead remains the same), which means a + small amount of extra overhead. Explicit and implicit choice of implementations are not mutually exclusive options. Indeed, most implementations of NumPy API overrides via -``__array_function__`` that we are familiar with (namely, dask, CuPy and -sparse, but not Pint) also include an explicit way to use their version of +``__array_function__`` that we are familiar with (namely, Dask, CuPy and +Sparse, but not Pint) also include an explicit way to use their version of NumPy's API by importing a module directly (``dask.array``, ``cupy`` or ``sparse``, respectively). diff --git a/doc/neps/nep-0038-SIMD-optimizations.rst b/doc/neps/nep-0038-SIMD-optimizations.rst index ab16868e4..396ba1371 100644 --- a/doc/neps/nep-0038-SIMD-optimizations.rst +++ b/doc/neps/nep-0038-SIMD-optimizations.rst @@ -1,3 +1,5 @@ +.. _NEP38: + ============================================================= NEP 38 — Using SIMD optimization instructions for performance ============================================================= diff --git a/doc/neps/nep-0040-legacy-datatype-impl.rst b/doc/neps/nep-0040-legacy-datatype-impl.rst index e4aa8f94a..c247e3d62 100644 --- a/doc/neps/nep-0040-legacy-datatype-impl.rst +++ b/doc/neps/nep-0040-legacy-datatype-impl.rst @@ -1,3 +1,5 @@ +.. _NEP40: + ================================================ NEP 40 — Legacy Datatype Implementation in NumPy ================================================ @@ -14,7 +16,8 @@ NEP 40 — Legacy Datatype Implementation in NumPy This NEP is part of a series of NEPs encompassing first information about the previous dtype implementation and issues with it in NEP 40 (this document). - NEP 41 then provides an overview and generic design choices for the refactor. + :ref:`NEP 41 <NEP41>` then provides an overview and generic design choices + for the refactor. Further NEPs 42 and 43 go into the technical details of the datatype and universal function related internal and external API changes. In some cases it may be necessary to consult the other NEPs for a full @@ -29,7 +32,7 @@ As a preparation to further NumPy enhancement proposals 41, 42, and 43. This NEP details the current status of NumPy datatypes as of NumPy 1.18. It describes some of the technical aspects and concepts that motivated the other proposals. -For more general information most readers should begin by reading NEP 41 +For more general information most readers should begin by reading :ref:`NEP 41 <NEP41>` and use this document only as a reference or for additional details. @@ -63,7 +66,7 @@ values are unaffected by it, and it is always possible to cast them to the native, canonical representation without any loss of information. The concept of flexibility can be generalized to parametric datatypes. -For example the private ``AdaptFlexibleDType`` function also accepts the +For example the private ``PyArray_AdaptFlexibleDType`` function also accepts the naive datetime dtype as input to find the correct time unit. The datetime dtype is thus parametric not in the size of its storage, but instead in what the stored value represents. @@ -107,11 +110,11 @@ This is useful for example in expressions such as:: In this expression, the python value (which originally has no datatype) is represented as an ``int8`` or ``int16`` (the smallest possible data type). -NumPy currently does this even for NumPy scalars and zero dimensional arrays, +NumPy currently does this even for NumPy scalars and zero-dimensional arrays, so that replacing ``5`` with ``np.int64(5)`` or ``np.array(5, dtype="int64")`` -will lead to the same results, and thus ignores the existing datatype. -The same logic also applies to floating point scalars, which are allowed to -lose precision. +in the above expression will lead to the same results, and thus ignores the +existing datatype. The same logic also applies to floating-point scalars, +which are allowed to lose precision. The behavior is not used when both inputs are scalars, so that ``5 + np.int8(5)`` returns the default integer size (32 or 64-bit) and not an ``np.int8``. @@ -178,8 +181,8 @@ These issues do not need to solved right away: scalars behave much like NumPy arrays, a feature that general Python objects do not have. * Seamless integration probably requires that ``np.array(scalar)`` finds the - correct DType automatically since some operations (such as indexing) are - return the scalar instead of a 0D array. + correct DType automatically since some operations (such as indexing) return + the scalar instead of a 0D array. This is problematic if multiple users independently decide to implement for example a DType for ``decimal.Decimal``. @@ -205,7 +208,7 @@ Many datatype-specific functions are defined within a C structure called :c:type:`PyArray_ArrFuncs`, which is part of each ``dtype`` instance and has a similarity to Python's ``PyNumberMethods``. For user-defined datatypes this structure is exposed to the user, making -ABI-compatible changes changes impossible. +ABI-compatible changes impossible. This structure holds important information such as how to copy or cast, and provides space for pointers to functions, such as comparing elements, converting to bool, or sorting. diff --git a/doc/neps/nep-0041-improved-dtype-support.rst b/doc/neps/nep-0041-improved-dtype-support.rst index 32200f27f..56ff5eac6 100644 --- a/doc/neps/nep-0041-improved-dtype-support.rst +++ b/doc/neps/nep-0041-improved-dtype-support.rst @@ -1,20 +1,23 @@ +.. _NEP41: + ================================================= NEP 41 — First step towards a new Datatype System ================================================= -:title: Improved Datatype Support +:title: First step towards a new Datatype System :Author: Sebastian Berg :Author: Stéfan van der Walt :Author: Matti Picus -:Status: Draft +:Status: Accepted :Type: Standard Track :Created: 2020-02-03 - +:Resolution: https://mail.python.org/pipermail/numpy-discussion/2020-April/080573.html and https://mail.python.org/pipermail/numpy-discussion/2020-March/080495.html .. note:: This NEP is part of a series of NEPs encompassing first information - about the previous dtype implementation and issues with it in NEP 40. + about the previous dtype implementation and issues with it in + :ref:`NEP 40 <NEP40>`. NEP 41 (this document) then provides an overview and generic design choices for the refactor. Further NEPs 42 and 43 go into the technical details of the datatype @@ -26,7 +29,7 @@ NEP 41 — First step towards a new Datatype System Abstract -------- -`Datatypes <data-type-objects-dtype>` in NumPy describe how to interpret each +:ref:`Datatypes <arrays.dtypes>` in NumPy describe how to interpret each element in arrays. NumPy provides ``int``, ``float``, and ``complex`` numerical types, as well as string, datetime, and structured datatype capabilities. The growing Python community, however, has need for more diverse datatypes. @@ -58,12 +61,13 @@ Motivation One of the main issues with the current API is the definition of typical functions such as addition and multiplication for parametric datatypes -(see also NEP 40) which require additional steps to determine the output type. +(see also :ref:`NEP 40 <NEP40>`) +which require additional steps to determine the output type. For example when adding two strings of length 4, the result is a string of length 8, which is different from the input. Similarly, a datatype which embeds a physical unit must calculate the new unit information: dividing a distance by a time results in a speed. -A related difficulty is that the :ref:`current casting rules <_ufuncs.casting>` +A related difficulty is that the :ref:`current casting rules <ufuncs.casting>` -- the conversion between different datatypes -- cannot describe casting for such parametric datatypes implemented outside of NumPy. @@ -141,7 +145,8 @@ in more details in the detailed description section: Storage information such as itemsize and byteorder can differ between different dtype instances (e.g. "S3" vs. "S8") and will remain part of the instance. This means that in the long run the current lowlevel access to dtype methods - will be removed (see ``PyArray_ArrFuncs`` in NEP 40). + will be removed (see ``PyArray_ArrFuncs`` in + :ref:`NEP 40 <NEP40>`). 2. The current NumPy scalars will *not* change, they will not be instances of datatypes. This will also be true for new datatypes, scalars will not be @@ -156,6 +161,14 @@ Further, the public API will be designed in a way that is extensible in the futu as much as possible. The public API should be an identical, but limited, version of the C-API used for the internal NumPy datatypes. +The datatype system may be targeted to work with NumPy arrays, +for example by providing strided-loops, but should avoid direct +interactions with the array-object (typically `np.ndarray` instances). +Instead, the design principle will be that the array-object is a consumer +of the datatype. +While only a guiding principle, this may allow splitting the datatype system +or even the NumPy datatypes into their own project which NumPy depends on. + The changes to the datatype system in Phase II must include a large refactor of the UFunc machinery, which will be further defined in NEP 43: @@ -214,7 +227,7 @@ Simple Numerical Types """""""""""""""""""""" Mainly used where memory is a consideration, lower-precision numeric types -such as :ref:```bfloat16`` <https://en.wikipedia.org/wiki/Bfloat16_floating-point_format>` +such as `bfloat16 <https://en.wikipedia.org/wiki/Bfloat16_floating-point_format>`_ are common in other computational frameworks. For these types the definitions of things such as ``np.common_type`` and ``np.can_cast`` are some of the most important interfaces. Once they @@ -379,7 +392,7 @@ numerical dtypes since there is no "universal" output type:: In fact ``np.result_type(meters, seconds)`` must error without context of the operation being done. This example highlights how the specific ufunc loop -(loop with known, specific DTypes as inputs), has to be able to to make +(loop with known, specific DTypes as inputs), has to be able to make certain decisions before the actual calculation can start. @@ -572,7 +585,8 @@ Organizing these methods and information in a more Pythonic way provides a solid foundation for refining and extending the API in the future. The current API cannot be extended due to how it is exposed publically. This means for example that the methods currently stored in ``PyArray_ArrFuncs`` -on each datatype (see NEP 40) will be defined differently in the future and +on each datatype (see :ref:`NEP 40 <NEP40>`) +will be defined differently in the future and deprecated in the long run. The most prominent visible side effect of this will be that @@ -590,12 +604,22 @@ Inheritance, however, appears problematic and a complexity best avoided Further, subclasses may be more interesting for interoperability for example with GPU backends (CuPy) storing additional methods related to the GPU rather than as a mechanism to define new datatypes. -A class hierarchy does provides value, this may be achieved by +A class hierarchy does provides value, and one can be achieved by allowing the creation of *abstract* datatypes. An example for an abstract datatype would be the datatype equivalent of ``np.floating``, representing any floating point number. These can serve the same purpose as Python's abstract base classes. +This NEP chooses to duplicate the scalar hierarchy fully or in part. +The main reason is to uncouple the implementation of the DType and scalar. +To add a DType to NumPy, in theory the scalar will not need to be +modified or know about NumPy. Also note that the categorical DType as +currently implemented in pandas does not have a scalar correspondence +making it less straight forward to rely on scalars to implement behaviour. +While DType and Scalar describe the same concept/type (e.g. an `int64`), +it seems practical to split out the information and functionality necessary +for numpy into the DType class. + Scalars should not be instances of the datatypes (2) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -673,8 +697,9 @@ C-API Changes to the UFunc Machinery (4) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Proposed changes to the UFunc machinery will be part of NEP 43. -However, the following changes will be necessary (see NEP 40 for a detailed -description of the current implementation and its issues): +However, the following changes will be necessary +(see :ref:`NEP 40 <NEP40>` +for a detailed description of the current implementation and its issues): * The current UFunc type resolution must be adapted to allow better control for user-defined dtypes as well as resolve current inconsistencies. @@ -693,7 +718,15 @@ functions after handling the unit related computations. Discussion ---------- -See NEP 40 for a list of previous meetings and discussions. +See :ref:`NEP 40 <NEP40>` +for a list of previous meetings and discussions. + +Additional discussion around this specific NEP has occured on both +the mailing list and the pull request: + +* `Mailing list discussion <https://mail.python.org/pipermail/numpy-discussion/2020-March/080481.html>`_ +* `NEP 41 pull request <https://github.com/numpy/numpy/pull/15506>`_ +* `Pull request thread on Dtype hierarchy and Scalars <https://github.com/numpy/numpy/pull/15506#discussion_r390016298>`_ References @@ -701,7 +734,7 @@ References .. [pandas_extension_arrays] https://pandas.pydata.org/pandas-docs/stable/development/extending.html#extension-types -.. _xarray_dtype_issue: https://github.com/pydata/xarray/issues/1262 +.. [xarray_dtype_issue] https://github.com/pydata/xarray/issues/1262 .. [pygeos] https://github.com/caspervdw/pygeos @@ -727,4 +760,3 @@ We would like to thank especially Stephan Hoyer, Nathaniel Smith, and Eric Wiese for repeated in-depth discussion about datatype design. We are very grateful for the community input in reviewing and revising this NEP and would like to thank especially Ross Barnowski and Ralf Gommers. - diff --git a/doc/neps/nep-0044-restructuring-numpy-docs.rst b/doc/neps/nep-0044-restructuring-numpy-docs.rst index 87da4edb0..229856547 100644 --- a/doc/neps/nep-0044-restructuring-numpy-docs.rst +++ b/doc/neps/nep-0044-restructuring-numpy-docs.rst @@ -1,3 +1,5 @@ +.. _NEP44: + =================================================== NEP 44 — Restructuring the NumPy Documentation =================================================== diff --git a/doc/neps/nep-0045-c_style_guide.rst b/doc/neps/nep-0045-c_style_guide.rst new file mode 100644 index 000000000..3c21e404f --- /dev/null +++ b/doc/neps/nep-0045-c_style_guide.rst @@ -0,0 +1,261 @@ +================================= +NEP 45 — C Style Guide +================================= + +:Author: Charles Harris <charlesr.harris@gmail.com> +:Status: Accepted +:Type: Process +:Created: 2012-02-26 +:Resolution: https://github.com/numpy/numpy/issues/11911 + +Abstract +-------- + +This document gives coding conventions for the C code comprising +the C implementation of NumPy. + +Motivation and Scope +-------------------- + +The NumPy C coding conventions are based on Python +`PEP 7 -- Style Guide for C Code <https://www.python.org/dev/peps/pep-0007>`_ +by Guido van Rossum with a few added strictures. + +Because the NumPy conventions are very close to those in PEP 7, that PEP is +used as a template with the NumPy additions and variations in the appropriate +spots. + +Usage and Impact +---------------- + +There are many C coding conventions and it must be emphasized that the primary +goal of the NumPy conventions isn't to choose the "best," about which there is +certain to be disagreement, but to achieve uniformity. + +Two good reasons to break a particular rule: + +1. When applying the rule would make the code less readable, even + for someone who is used to reading code that follows the rules. + +2. To be consistent with surrounding code that also breaks it + (maybe for historic reasons) -- although this is also an + opportunity to clean up someone else's mess. + + +Backward compatibility +---------------------- + +No impact. + + +Detailed description +-------------------- + +C dialect +========= + +* Use C99 (that is, the standard defined by ISO/IEC 9899:1999). + +* Don't use GCC extensions (for instance, don't write multi-line strings + without trailing backslashes). Preferably break long strings + up onto separate lines like so:: + + "blah blah" + "blah blah" + + This will work with MSVC, which otherwise chokes on very long + strings. + +* All function declarations and definitions must use full prototypes (that is, + specify the types of all arguments). + +* No compiler warnings with major compilers (gcc, VC++, a few others). + +.. Note:: + NumPy still produces compiler warnings that need to be addressed. + +Code layout +============ + +* Use 4-space indents and no tabs at all. + +* No line should be longer than 80 characters. If this and the + previous rule together don't give you enough room to code, your code is + too complicated -- consider using subroutines. + +* No line should end in whitespace. If you think you need + significant trailing whitespace, think again; somebody's editor might + delete it as a matter of routine. + +* Function definition style: function name in column 1, outermost + curly braces in column 1, blank line after local variable declarations:: + + static int + extra_ivars(PyTypeObject *type, PyTypeObject *base) + { + int t_size = PyType_BASICSIZE(type); + int b_size = PyType_BASICSIZE(base); + + assert(t_size >= b_size); /* type smaller than base! */ + ... + return 1; + } + + If the transition to C++ goes through it is possible that this form will + be relaxed so that short class methods meant to be inlined can have the + return type on the same line as the function name. However, that is yet to + be determined. + +* Code structure: one space between keywords like ``if``, ``for`` and + the following left parenthesis; no spaces inside the parenthesis; braces + around all ``if`` branches, and no statements on the same line as the + ``if``. They should be formatted as shown:: + + if (mro != NULL) { + one_line_statement; + } + else { + ... + } + + + for (i = 0; i < n; i++) { + one_line_statement; + } + + + while (isstuff) { + dostuff; + } + + + do { + stuff; + } while (isstuff); + + + switch (kind) { + /* Boolean kind */ + case 'b': + return 0; + /* Unsigned int kind */ + case 'u': + ... + /* Anything else */ + default: + return 3; + } + + +* The return statement should *not* get redundant parentheses:: + + return Py_None; /* correct */ + return(Py_None); /* incorrect */ + +* Function and macro call style: ``foo(a, b, c)``, no space before + the open paren, no spaces inside the parens, no spaces before + commas, one space after each comma. + +* Always put spaces around the assignment, Boolean, and comparison + operators. In expressions using a lot of operators, add spaces + around the outermost (lowest priority) operators. + +* Breaking long lines: If you can, break after commas in the + outermost argument list. Always indent continuation lines + appropriately: :: + + PyErr_SetString(PyExc_TypeError, + "Oh dear, you messed up."); + + Here appropriately means at least a double indent (8 spaces). It isn't + necessary to line everything up with the opening parenthesis of the function + call. + +* When you break a long expression at a binary operator, the + operator goes at the end of the previous line, for example: :: + + if (type > tp_dictoffset != 0 && + base > tp_dictoffset == 0 && + type > tp_dictoffset == b_size && + (size_t)t_size == b_size + sizeof(PyObject *)) { + return 0; + } + + Note that the terms in the multi-line Boolean expression are indented so + as to make the beginning of the code block clearly visible. + +* Put blank lines around functions, structure definitions, and + major sections inside functions. + +* Comments go before the code they describe. Multi-line comments should + be like so: :: + + /* + * This would be a long + * explanatory comment. + */ + + Trailing comments should be used sparingly. Instead of :: + + if (yes) { // Success! + + do :: + + if (yes) { + // Success! + +* All functions and global variables should be declared static + when they aren't needed outside the current compilation unit. + +* Declare external functions and variables in a header file. + + +Naming conventions +================== + +* There has been no consistent prefix for NumPy public functions, but + they all begin with a prefix of some sort, followed by an underscore, and + are in camel case: ``PyArray_DescrAlignConverter``, ``NpyIter_GetIterNext``. + In the future the names should be of the form ``Npy*_PublicFunction``, + where the star is something appropriate. + +* Public Macros should have a ``NPY_`` prefix and then use upper case, + for example, ``NPY_DOUBLE``. + +* Private functions should be lower case with underscores, for example: + ``array_real_get``. Single leading underscores should not be used, but + some current function names violate that rule due to historical accident. + +.. Note:: + Functions whose names begin with a single underscore should be renamed at + some point. + + +Function documentation +====================== + +NumPy doesn't have a C function documentation standard at this time, but +needs one. Most NumPy functions are not documented in the code, and that +should change. One possibility is Doxygen with a plugin so that the same +NumPy style used for Python functions can also be used for documenting +C functions, see the files in ``doc/cdoc/``. + + +Related Work +------------ + +Based on Van Rossum and Warsaw, `PEP 7 -- Style Guide for C Code <https://www.python.org/dev/peps/pep-0007>`_ + + +Discussion +---------- + +https://github.com/numpy/numpy/issues/11911 +recommended that this proposal, which originated as ``doc/C_STYLE_GUIDE.rst.txt``, +be turned into an NEP. + + +Copyright +--------- + +This document has been placed in the public domain. diff --git a/doc/neps/roadmap.rst b/doc/neps/roadmap.rst index c5abc5f25..3780499a0 100644 --- a/doc/neps/roadmap.rst +++ b/doc/neps/roadmap.rst @@ -17,10 +17,10 @@ may include (among other things) interoperability protocols, better duck typing support and ndarray subclass handling. - The ``__array_function__`` protocol is currently experimental and needs to be - matured. See `NEP 18`_ for details. + matured. See :ref`NEP18` for details. - New protocols for overriding other functionality in NumPy may be needed. - Array duck typing, or handling "duck arrays", needs improvements. See - `NEP 22`_ for details. + :ref:`NEP22` for details. Extensibility ------------- @@ -71,13 +71,13 @@ Random number generation policy & rewrite ----------------------------------------- A new random number generation framework with higher performance generators is -close to completion, see `NEP 19`_ and `PR 13163`_. +close to completion, see :ref:`NEP19` and `PR 13163`_. Indexing -------- We intend to add new indexing modes for "vectorized indexing" and "outer indexing", -see `NEP 21`_. +see :ref:`NEP21`. Continuous Integration ---------------------- @@ -105,10 +105,6 @@ Other functionality - Deprecate ``np.matrix`` (very slowly) -.. _`NEP 19`: https://www.numpy.org/neps/nep-0019-rng-policy.html -.. _`NEP 22`: http://www.numpy.org/neps/nep-0022-ndarray-duck-typing-overview.html -.. _`NEP 18`: https://www.numpy.org/neps/nep-0018-array-function-protocol.html .. _implementation: https://gist.github.com/shoyer/1f0a308a06cd96df20879a1ddb8f0006 .. _`reference implementation`: https://github.com/bashtage/randomgen -.. _`NEP 21`: https://www.numpy.org/neps/nep-0021-advanced-indexing.html .. _`PR 13163`: https://github.com/numpy/numpy/pull/13163 diff --git a/doc/neps/scope.rst b/doc/neps/scope.rst index a675b8c96..93887c4b1 100644 --- a/doc/neps/scope.rst +++ b/doc/neps/scope.rst @@ -14,7 +14,8 @@ Here, we describe aspects of N-d array computation that are within scope for Num - NumPy is a *de facto* standard for array APIs in Python - Indexing and fast iteration over elements (ufunc) - - Interoperability protocols with other data container implementations (like `__array_ufunc__`). + - Interoperability protocols with other data container implementations (like + :ref:`__array_ufunc__ and __array_function__ <basics.dispatch>`. - **Python API and a C API** to the ndarray's methods and attributes. @@ -35,7 +36,8 @@ Here, we describe aspects of N-d array computation that are within scope for Num - NumPy provides some **infrastructure for other packages in the scientific Python ecosystem**: - - numpy.distutils (build support for C++, Fortran, BLAS/LAPACK, and other relevant libraries for scientific computing + - numpy.distutils (build support for C++, Fortran, BLAS/LAPACK, and other + relevant libraries for scientific computing) - f2py (generating bindings for Fortran code) - testing utilities diff --git a/doc/neps/tools/build_index.py b/doc/neps/tools/build_index.py index b131fcd04..51227a6f1 100644 --- a/doc/neps/tools/build_index.py +++ b/doc/neps/tools/build_index.py @@ -22,6 +22,7 @@ def nep_metadata(): meta_re = r':([a-zA-Z\-]*): (.*)' + has_provisional = False neps = {} print('Loading metadata for:') for source in sources: @@ -34,15 +35,23 @@ def nep_metadata(): tags = [match.groups() for match in tags if match is not None] tags = {tag[0]: tag[1] for tag in tags} - # We could do a clever regexp, but for now just assume the title is - # the second line of the document - tags['Title'] = lines[1].strip() + # The title should be the first line after a line containing only + # * or = signs. + for i, line in enumerate(lines[:-1]): + chars = set(line.rstrip()) + if len(chars) == 1 and ("=" in chars or "*" in chars): + break + else: + raise RuntimeError("Unable to find NEP title.") + + tags['Title'] = lines[i+1].strip() tags['Filename'] = source if not tags['Title'].startswith(f'NEP {nr} — '): raise RuntimeError( f'Title for NEP {nr} does not start with "NEP {nr} — " ' - '(note that — here is a special, enlongated dash)') + '(note that — here is a special, enlongated dash). Got: ' + f' {tags["Title"]!r}') if tags['Status'] in ('Accepted', 'Rejected', 'Withdrawn'): if not 'Resolution' in tags: @@ -50,6 +59,8 @@ def nep_metadata(): f'NEP {nr} is Accepted/Rejected/Withdrawn but ' 'has no Resolution tag' ) + if tags['Status'] == 'Provisional': + has_provisional = True neps[nr] = tags @@ -87,7 +98,7 @@ def nep_metadata(): f'been set to Superseded' ) - return {'neps': neps} + return {'neps': neps, 'has_provisional': has_provisional} infile = 'index.rst.tmpl' diff --git a/doc/release/13421.improvement.rst b/doc/release/13421.improvement.rst deleted file mode 100644 index 6d4573aa3..000000000 --- a/doc/release/13421.improvement.rst +++ /dev/null @@ -1,8 +0,0 @@ -Improve detection of CPU features -================================= - -Replace ``npy_cpu_supports`` which was a gcc-specific mechanism to test support -of avx with more general functions ``npy_cpu_init`` and ``npy_cpu_have``, and -expose the results via a ``NPY_CPU_HAVE`` c-macro as well as a python-level -``__cpu_features__`` dictionary. - diff --git a/doc/release/upcoming_changes/14933.compatibility.rst b/doc/release/upcoming_changes/14933.compatibility.rst deleted file mode 100644 index 1b5f1b113..000000000 --- a/doc/release/upcoming_changes/14933.compatibility.rst +++ /dev/null @@ -1,10 +0,0 @@ -Scalar promotion in ``PyArray_ConvertToCommonType`` ---------------------------------------------------- - -The promotion of mixed scalars and arrays in ``PyArray_ConvertToCommonType`` -has been changed to adhere to those used by ``np.result_type``. -This means that input such as ``(1000, np.array([1], dtype=np.uint8)))`` -will now return ``uint16`` dtypes. In most cases the behaviour is unchanged. -Note that the use of this C-API function is generally discouraged. -This also fixes ``np.choose`` to behave the same way as the rest of NumPy -in this respect. diff --git a/doc/release/upcoming_changes/14942.compatibility.rst b/doc/release/upcoming_changes/14942.compatibility.rst deleted file mode 100644 index 461758c95..000000000 --- a/doc/release/upcoming_changes/14942.compatibility.rst +++ /dev/null @@ -1,6 +0,0 @@ -Fasttake and fastputmask slots are deprecated and NULL'ed ---------------------------------------------------------- -The fasttake and fastputmask slots are now never used and -must always be set to NULL. This will result in no change in behaviour. -However, if a user dtype should set one of these a DeprecationWarning -will be given. diff --git a/doc/release/upcoming_changes/14981.compatibility.rst b/doc/release/upcoming_changes/14981.compatibility.rst deleted file mode 100644 index 90cf866f2..000000000 --- a/doc/release/upcoming_changes/14981.compatibility.rst +++ /dev/null @@ -1,10 +0,0 @@ -`np.ediff1d` casting behaviour with ``to_end`` and ``to_begin`` ---------------------------------------------------------------- - -`np.ediff1d` now uses the ``"same_kind"`` casting rule for -its additional ``to_end`` and ``to_begin`` arguments. This -ensures type safety except when the input array has a smaller -integer type than ``to_begin`` or ``to_end``. -In rare cases, the behaviour will be more strict than it was -previously in 1.16 and 1.17. This is necessary to solve issues -with floating point NaN. diff --git a/doc/release/upcoming_changes/14995.compatibility.rst b/doc/release/upcoming_changes/14995.compatibility.rst deleted file mode 100644 index b3b8b5933..000000000 --- a/doc/release/upcoming_changes/14995.compatibility.rst +++ /dev/null @@ -1,10 +0,0 @@ -Converting of empty array-like objects to NumPy arrays ------------------------------------------------------- -Objects with ``len(obj) == 0`` which implement an "array-like" interface, -meaning an object implementing ``obj.__array__()``, -``obj.__array_interface__``, ``obj.__array_struct__``, or the python -buffer interface and which are also sequences (i.e. Pandas objects) -will now always retain there shape correctly when converted to an array. -If such an object has a shape of ``(0, 1)`` previously, it could -be converted into an array of of shape ``(0,)`` (losing all dimensions -after the first 0). diff --git a/doc/release/upcoming_changes/15118.change.rst b/doc/release/upcoming_changes/15118.change.rst deleted file mode 100644 index f14beebbe..000000000 --- a/doc/release/upcoming_changes/15118.change.rst +++ /dev/null @@ -1,7 +0,0 @@ -Remove handling of extra argument to ``__array__`` --------------------------------------------------- -A code path and test have been in the code since NumPy 0.4 for a two-argument -variant of ``__array__(dtype=None, context=None)``. It was activated when -calling ``ufunc(op)`` or ``ufunc.reduce(op)`` if ``op.__array__`` existed. -However that variant is not documented, and it is not clear what the intention -was for its use. It has been removed. diff --git a/doc/release/upcoming_changes/15119.deprecation.rst b/doc/release/upcoming_changes/15119.deprecation.rst deleted file mode 100644 index d18e440fe..000000000 --- a/doc/release/upcoming_changes/15119.deprecation.rst +++ /dev/null @@ -1,8 +0,0 @@ - -Deprecate automatic ``dtype=object`` for ragged input ------------------------------------------------------ -Calling ``np.array([[1, [1, 2, 3]])`` will issue a ``DeprecationWarning`` as -per `NEP 34`_. Users should explicitly use ``dtype=object`` to avoid the -warning. - -.. _`NEP 34`: https://numpy.org/neps/nep-0034.html diff --git a/doc/release/upcoming_changes/15217.deprecation.rst b/doc/release/upcoming_changes/15217.deprecation.rst deleted file mode 100644 index d49de20b5..000000000 --- a/doc/release/upcoming_changes/15217.deprecation.rst +++ /dev/null @@ -1,13 +0,0 @@ -Passing ``shape=0`` to factory functions in ``numpy.rec`` is deprecated ------------------------------------------------------------------------ - -``0`` is treated as a special case and is aliased to ``None`` in the functions: - -* `numpy.core.records.fromarrays` -* `numpy.core.records.fromrecords` -* `numpy.core.records.fromstring` -* `numpy.core.records.fromfile` - -In future, ``0`` will not be special cased, and will be treated as an array -length like any other integer. - diff --git a/doc/release/upcoming_changes/15218.improvement.rst b/doc/release/upcoming_changes/15218.improvement.rst deleted file mode 100644 index ccbbbd66f..000000000 --- a/doc/release/upcoming_changes/15218.improvement.rst +++ /dev/null @@ -1,6 +0,0 @@ -Use 64-bit integer size on 64-bit platforms in fallback lapack_lite -------------------------------------------------------------------- - -Use 64-bit integer size on 64-bit platforms in the fallback LAPACK library, -which is used when the system has no LAPACK installed, allowing it to deal with -linear algebra for large arrays. diff --git a/doc/release/upcoming_changes/15229.compatibility.rst b/doc/release/upcoming_changes/15229.compatibility.rst deleted file mode 100644 index 404f7774f..000000000 --- a/doc/release/upcoming_changes/15229.compatibility.rst +++ /dev/null @@ -1,7 +0,0 @@ -Removed ``multiarray.int_asbuffer`` ------------------------------------ - -As part of the continued removal of Python 2 compatibility, -``multiarray.int_asbuffer`` was removed. On Python 3, it threw a -``NotImplementedError`` and was unused internally. It is expected that there -are no downstream use cases for this method with Python 3. diff --git a/doc/release/upcoming_changes/15233.highlight.rst b/doc/release/upcoming_changes/15233.highlight.rst deleted file mode 100644 index df96ee871..000000000 --- a/doc/release/upcoming_changes/15233.highlight.rst +++ /dev/null @@ -1,4 +0,0 @@ -* Code compatibility with Python versions < 3.5 (including Python 2) was - dropped from both the python and C code. The shims in numpy.compat will - remain to support third-party packages, but they may be deprecated in a - future release. diff --git a/doc/release/upcoming_changes/15251.c_api.rst b/doc/release/upcoming_changes/15251.c_api.rst deleted file mode 100644 index f391c904b..000000000 --- a/doc/release/upcoming_changes/15251.c_api.rst +++ /dev/null @@ -1,10 +0,0 @@ -Better support for ``const`` dimensions in API functions --------------------------------------------------------- -The following functions now accept a constant array of ``npy_intp``: - -* `PyArray_BroadcastToShape` -* `PyArray_IntTupleFromIntp` -* `PyArray_OverflowMultiplyList` - -Previously the caller would have to cast away the const-ness to call these -functions. diff --git a/doc/release/upcoming_changes/15255.compatibility.rst b/doc/release/upcoming_changes/15255.compatibility.rst deleted file mode 100644 index e360eeeb3..000000000 --- a/doc/release/upcoming_changes/15255.compatibility.rst +++ /dev/null @@ -1,12 +0,0 @@ -``numpy.distutils.compat`` has been removed -------------------------------------------- -This module contained only the function ``get_exception()``, which was used as:: - - try: - ... - except Exception: - e = get_exception() - -Its purpose was to handle the change in syntax introduced in Python 2.6, from -``except Exception, e:`` to ``except Exception as e:``, meaning it was only -necessary for codebases supporting Python 2.5 and older. diff --git a/doc/release/upcoming_changes/15355.c_api.rst b/doc/release/upcoming_changes/15355.c_api.rst deleted file mode 100644 index ffc1972cf..000000000 --- a/doc/release/upcoming_changes/15355.c_api.rst +++ /dev/null @@ -1,7 +0,0 @@ -Const qualify UFunc inner loops
--------------------------------
-``UFuncGenericFunction`` now expects pointers to const ``dimension`` and
-``strides`` as arguments. This means inner loops may no longer modify
-either ``dimension`` or ``strides``. This change leads to an
-``incompatible-pointer-types`` warning forcing users to either ignore
-the compiler warnings or to const qualify their own loop signatures.
diff --git a/doc/release/upcoming_changes/15385.new_feature.rst b/doc/release/upcoming_changes/15385.new_feature.rst deleted file mode 100644 index 24e9b793c..000000000 --- a/doc/release/upcoming_changes/15385.new_feature.rst +++ /dev/null @@ -1,5 +0,0 @@ -``np.str_`` scalars now support the buffer protocol ---------------------------------------------------- -``np.str_`` arrays are always stored as UCS4, so the corresponding scalars -now expose this through the buffer interface, meaning -``memoryview(np.str_('test'))`` now works. diff --git a/doc/release/upcoming_changes/15427.deprecation.rst b/doc/release/upcoming_changes/15427.deprecation.rst deleted file mode 100644 index 0100406b3..000000000 --- a/doc/release/upcoming_changes/15427.deprecation.rst +++ /dev/null @@ -1,12 +0,0 @@ -Deprecation of probably unused C-API functions ----------------------------------------------- -The following C-API functions are probably unused and have been -deprecated: - -* ``PyArray_GetArrayParamsFromObject`` -* ``PyUFunc_GenericFunction`` -* ``PyUFunc_SetUsesArraysAsData`` - -In most cases ``PyArray_GetArrayParamsFromObject`` should be replaced -by converting to an array, while ``PyUFunc_GenericFunction`` can be -replaced with ``PyObject_Call`` (see documentation for details). diff --git a/doc/release/upcoming_changes/15463.change.rst b/doc/release/upcoming_changes/15463.change.rst deleted file mode 100644 index 25befbcc7..000000000 --- a/doc/release/upcoming_changes/15463.change.rst +++ /dev/null @@ -1,12 +0,0 @@ -``numpy.random._bit_generator`` moved to ``numpy.random.bit_generator`` ------------------------------------------------------------------------ - -In order to expose `numpy.random.BitGenerator` and `numpy.random.SeedSequence` -to cython, the ``_bitgenerator`` module is now public as -`numpy.random.bit_generator` - -Cython access to the random distributions is provided via a `pxd` file ----------------------------------------------------------------------- - -``c_distributions.pxd`` provides access to the c functions behind many of the -random distributions from Cython, making it convenient to use and extend them. diff --git a/doc/release/upcoming_changes/15534.deprecation.rst b/doc/release/upcoming_changes/15534.deprecation.rst deleted file mode 100644 index 243e224ba..000000000 --- a/doc/release/upcoming_changes/15534.deprecation.rst +++ /dev/null @@ -1,11 +0,0 @@ -Converting certain types to dtypes is Deprecated ------------------------------------------------- -The super classes of scalar types, such as ``np.integer``, ``np.generic``, -or ``np.inexact`` will now give a deprecation warning when converted -to a dtype (or used in a dtype keyword argument). -The reason for this is that `np.integer` is converted to ``np.int_``, -while it would be expected to represent *any* integer (e.g. also -``int8``, ``int16``, etc. -For example, ``dtype=np.floating`` is currently identical to -``dtype=np.float64``, even though also ``np.float32`` is a subclass of -``np.floating``. diff --git a/doc/release/upcoming_changes/15685.new_feature.rst b/doc/release/upcoming_changes/15685.new_feature.rst deleted file mode 100644 index c4ed04e93..000000000 --- a/doc/release/upcoming_changes/15685.new_feature.rst +++ /dev/null @@ -1,9 +0,0 @@ -``subok`` option for `numpy.copy` ---------------------------------- -A new kwarg, ``subok``, was added to `numpy.copy` to allow users to toggle the -behavior of `numpy.copy` with respect to array subclasses. The default value -is ``False`` which is consistent with the behavior of `numpy.copy` for -previous numpy versions. To create a copy that preserves an array subclass with -`numpy.copy`, call ``np.copy(arr, subok=True)``. This addition better documents -that the default behavior of `numpy.copy` differs from the -`numpy.ndarray.copy` method which respects array subclasses by default. diff --git a/doc/release/upcoming_changes/15759.improvement.rst b/doc/release/upcoming_changes/15759.improvement.rst new file mode 100644 index 000000000..0a1b255f7 --- /dev/null +++ b/doc/release/upcoming_changes/15759.improvement.rst @@ -0,0 +1,4 @@ +Remove the Accelerate library as a candidate LAPACK library +----------------------------------------------------------- +Apple no longer supports Accelerate. Remove it. + diff --git a/doc/release/upcoming_changes/15886.deprecation.rst b/doc/release/upcoming_changes/15886.deprecation.rst new file mode 100644 index 000000000..050817e66 --- /dev/null +++ b/doc/release/upcoming_changes/15886.deprecation.rst @@ -0,0 +1,7 @@ +Passing ``shape=None`` to functions with a non-optional shape argument is deprecated +------------------------------------------------------------------------------------ +Previously, this was an alias for passing ``shape=()``. +This deprecation is emitted by `PyArray_IntpConverter` in the C API. If your +API is intended to support passing `None`, then you should check for `None` +prior to invoking the converter, so as to be able to distinguish `None` and +``()``. diff --git a/doc/release/upcoming_changes/15900.deprecation.rst b/doc/release/upcoming_changes/15900.deprecation.rst new file mode 100644 index 000000000..22be711d0 --- /dev/null +++ b/doc/release/upcoming_changes/15900.deprecation.rst @@ -0,0 +1,16 @@ +Indexing errors will be reported even when index result is empty +---------------------------------------------------------------- +In the future, NumPy will raise an IndexError when an +integer array index contains out of bound values even if a non-indexed +dimension is of length 0. This will now emit a DeprecationWarning. +This can happen when the array is previously empty, or an empty +slice is involved:: + + arr1 = np.zeros((5, 0)) + arr1[[20]] + arr2 = np.zeros((5, 5)) + arr2[[20], :0] + +Previously the non-empty index ``[20]`` was not checked for correctness. +It will now be checked causing a deprecation warning which will be turned +into an error. This also applies to assignments. diff --git a/doc/release/upcoming_changes/15997.improvement.rst b/doc/release/upcoming_changes/15997.improvement.rst new file mode 100644 index 000000000..9b5feacb8 --- /dev/null +++ b/doc/release/upcoming_changes/15997.improvement.rst @@ -0,0 +1,12 @@ +Object arrays containing multi-line objects have a more readable ``repr`` +------------------------------------------------------------------------- +If elements of an object array have a ``repr`` containing new lines, then the +wrapped lines will be aligned by column. Notably, this improves the ``repr`` of +nested arrays:: + + >>> np.array([np.eye(2), np.eye(3)], dtype=object) + array([array([[1., 0.], + [0., 1.]]), + array([[1., 0., 0.], + [0., 1., 0.], + [0., 0., 1.]])], dtype=object) diff --git a/doc/release/upcoming_changes/16156.deprecation.rst b/doc/release/upcoming_changes/16156.deprecation.rst new file mode 100644 index 000000000..153cc4428 --- /dev/null +++ b/doc/release/upcoming_changes/16156.deprecation.rst @@ -0,0 +1,5 @@ +Deprecation of `numpy.dual` +--------------------------- +The module `numpy.dual` is deprecated. Instead of importing functions +from `numpy.dual`, the functions should be imported directly from NumPy +or SciPy. diff --git a/doc/release/upcoming_changes/8255.new_feature.rst b/doc/release/upcoming_changes/8255.new_feature.rst deleted file mode 100644 index c0bc21b3e..000000000 --- a/doc/release/upcoming_changes/8255.new_feature.rst +++ /dev/null @@ -1,5 +0,0 @@ -`numpy.frompyfunc` now accepts an identity argument ---------------------------------------------------- -This allows the :attr:`numpy.ufunc.identity` attribute to be set on the -resulting ufunc, meaning it can be used for empty and multi-dimensional -calls to :meth:`numpy.ufunc.reduce`. diff --git a/doc/source/_templates/indexcontent.html b/doc/source/_templates/indexcontent.html index 294d39233..d77c5a85e 100644 --- a/doc/source/_templates/indexcontent.html +++ b/doc/source/_templates/indexcontent.html @@ -1,35 +1,43 @@ {% extends "defindex.html" %} {% block tables %} - <p><strong>Parts of the documentation:</strong></p> - <table class="contentstable" align="center"><tr> +<p><strong>For users:</strong></p> +<table class="contentstable" align="center"><tr> <td width="50%"> - <p class="biglink"><a class="biglink" href="{{ pathto("user/index") }}">NumPy User Guide</a><br/> - <span class="linkdescr">start here</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("reference/index") }}">NumPy Reference</a><br/> - <span class="linkdescr">reference documentation</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("benchmarking") }}">Benchmarking</a><br/> - <span class="linkdescr">benchmarking NumPy</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("user/setting-up") }}">Setting Up</a><br/> + <span class="linkdescr">Learn about what NumPy is and how to install it</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("user/quickstart") }}">Quickstart Tutorial</a><br/> + <span class="linkdescr">Aimed at domain experts or people migrating to NumPy</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("user/absolute_beginners") }}">Absolute Beginners Tutorial</a><br/> + <span class="linkdescr">Start here for an overview of NumPy features and syntax</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("user/tutorials_index") }}">Tutorials</a><br/> + <span class="linkdescr">Learn about concepts and submodules</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("user/howtos_index") }}">How Tos</a><br/> + <span class="linkdescr">How to do common tasks with NumPy</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("reference/index") }}">NumPy API Reference</a><br/> + <span class="linkdescr">Automatically generated reference documentation</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("user/explanations_index") }}">Explanations</a><br/> + <span class="linkdescr">In depth explanation of concepts, best practices and techniques</span></p> <p class="biglink"><a class="biglink" href="{{ pathto("f2py/index") }}">F2Py Guide</a><br/> - <span class="linkdescr">f2py documentation</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("dev/index") }}">NumPy Developer Guide</a><br/> - <span class="linkdescr">contributing to NumPy</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("docs/index") }}">Building and Extending the Documentation</a><br/> - <span class="linkdescr">about this documentation</span></p> - </td></tr> - </table> + <span class="linkdescr">Documentation for the f2py module (Fortran extensions for Python)</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("glossary") }}">Glossary</a><br/> + <span class="linkdescr">List of the most important terms</span></p> + </td></tr> +</table> - <p><strong>Indices and tables:</strong></p> - <table class="contentstable" align="center"><tr> +<p><strong>For developers/contributors:</strong></p> +<table class="contentstable" align="center"><tr> <td width="50%"> - <p class="biglink"><a class="biglink" href="{{ pathto("genindex") }}">General Index</a><br/> - <span class="linkdescr">all functions, classes, terms</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("glossary") }}">Glossary</a><br/> - <span class="linkdescr">the most important terms explained</span></p> - </td><td width="50%"> - <p class="biglink"><a class="biglink" href="{{ pathto("search") }}">Search page</a><br/> - <span class="linkdescr">search this documentation</span></p> - <p class="biglink"><a class="biglink" href="{{ pathto("contents") }}">Complete Table of Contents</a><br/> - <span class="linkdescr">lists all sections and subsections</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("dev/index") }}">NumPy Contributor Guide</a><br/> + <span class="linkdescr">Contributing to NumPy</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("dev/underthehood") }}">Under-the-hood docs</a><br/> + <span class="linkdescr">Specialized, in-depth documentation</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("docs/index") }}">Building and Extending the Documentation</a><br/> + <span class="linkdescr">How to contribute to this documentation (user and API)</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("docs/howto_document") }}">The numpydoc docstring guide</a><br/> + <span class="linkdescr">How to write docstrings in the numpydoc format</span></p> + <p class="biglink"><a class="biglink" href="{{ pathto("benchmarking") }}">Benchmarking</a><br/> + <span class="linkdescr">benchmarking NumPy</span></p> + <p class="biglink"><a class="biglink" href="https://www.numpy.org/neps/index.html">NumPy Enhancement Proposals</a><br/> </td></tr> </table> @@ -37,11 +45,9 @@ <table class="contentstable" align="center"><tr> <td width="50%"> <p class="biglink"><a class="biglink" href="{{ pathto("bugs") }}">Reporting bugs</a></p> - <p class="biglink"><a class="biglink" href="{{ pathto("about") }}">About NumPy</a></p> - <p class="biglink"><a class="biglink" href="https://www.numpy.org/neps/index.html"> - NumPy Enhancement Proposals</a><br/> - </td><td width="50%"> <p class="biglink"><a class="biglink" href="{{ pathto("release") }}">Release Notes</a></p> + </td><td width="50%"> + <p class="biglink"><a class="biglink" href="{{ pathto("about") }}">About NumPy</a></p> <p class="biglink"><a class="biglink" href="{{ pathto("license") }}">License of NumPy</a></p> </td></tr> </table> @@ -56,7 +62,7 @@ </p> <p> The preferred way to update the documentation is by submitting a pull - request on Github (see the <a href="{{ pathto("docs/index") }}">Documentation Index</a>. + request on Github (see the <a href="{{ pathto("docs/index") }}">Documentation Index</a>). Please help us to further improve the NumPy documentation! </p> {% endblock %} diff --git a/doc/source/conf.py b/doc/source/conf.py index a58dc4897..d6a0f8bf3 100644 --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -155,6 +155,9 @@ plot_html_show_source_link = False # The font size ('10pt', '11pt' or '12pt'). #latex_font_size = '10pt' +# XeLaTeX for better support of unicode characters +latex_engine = 'xelatex' + # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, document class [howto/manual]). _stdauthor = 'Written by the NumPy community' diff --git a/doc/source/contents.rst b/doc/source/contents.rst index 019dcc71d..baea7784c 100644 --- a/doc/source/contents.rst +++ b/doc/source/contents.rst @@ -1,16 +1,27 @@ -##################### -NumPy manual contents -##################### +.. _numpy_docs_mainpage: + +################### +NumPy Documentation +################### .. toctree:: - user/index + user/setting-up + user/quickstart + user/absolute_beginners + user/tutorials_index + user/howtos_index reference/index + user/explanations_index f2py/index + glossary dev/index + dev/underthehood docs/index + docs/howto_document + benchmarking + bugs release about - bugs license - glossary + diff --git a/doc/source/dev/conduct/code_of_conduct.rst b/doc/source/dev/conduct/code_of_conduct.rst index aca39d8a7..f2f0a536d 100644 --- a/doc/source/dev/conduct/code_of_conduct.rst +++ b/doc/source/dev/conduct/code_of_conduct.rst @@ -113,8 +113,8 @@ You can report issues to the NumPy Code of Conduct committee, at numpy-conduct@googlegroups.com. Currently, the committee consists of: - Stefan van der Walt -- Nathaniel J. Smith -- Ralf Gommers +- Melissa Weber Mendonça +- Anirudh Subramanian If your report involves any members of the committee, or if they feel they have a conflict of interest in handling it, then they will recuse themselves from diff --git a/doc/source/dev/development_environment.rst b/doc/source/dev/development_environment.rst index 1f404d6e0..ff78cecc5 100644 --- a/doc/source/dev/development_environment.rst +++ b/doc/source/dev/development_environment.rst @@ -211,10 +211,20 @@ Debugging --------- Another frequently asked question is "How do I debug C code inside NumPy?". -The easiest way to do this is to first write a Python script that invokes the C -code whose execution you want to debug. For instance ``mytest.py``:: +First, ensure that you have gdb installed on your system with the Python +extensions (often the default on Linux). You can see which version of +Python is running inside gdb to verify your setup:: - from numpy import linspace + (gdb) python + >import sys + >print(sys.version_info) + >end + sys.version_info(major=3, minor=7, micro=0, releaselevel='final', serial=0) + +Next you need to write a Python script that invokes the C code whose execution +you want to debug. For instance ``mytest.py``:: + + import numpy as np x = np.arange(5) np.empty_like(x) @@ -228,10 +238,14 @@ And then in the debugger:: (gdb) run The execution will now stop at the corresponding C function and you can step -through it as usual. With the Python extensions for gdb installed (often the -default on Linux), a number of useful Python-specific commands are available. +through it as usual. A number of useful Python-specific commands are available. For example to see where in the Python code you are, use ``py-list``. For more -details, see `DebuggingWithGdb`_. +details, see `DebuggingWithGdb`_. Here are some commonly used commands: + + - ``list``: List specified function or line. + - ``next``: Step program, proceeding through subroutine calls. + - ``step``: Continue program being debugged, after signal or breakpoint. + - ``print``: Print value of expression EXP. Instead of plain ``gdb`` you can of course use your favourite alternative debugger; run it on the python binary with arguments diff --git a/doc/source/dev/development_workflow.rst b/doc/source/dev/development_workflow.rst index 9f2ecede6..d5a49a9f9 100644 --- a/doc/source/dev/development_workflow.rst +++ b/doc/source/dev/development_workflow.rst @@ -26,7 +26,7 @@ In short: - *Contributors*: push your feature branch to your own Github repo, and :ref:`create a pull request <asking-for-merging>`. - - *Core developers* If you want to push changes without + - *Core developers*: If you want to push changes without further review, see the notes :ref:`below <pushing-to-main>`. This way of working helps to keep work well organized and the history @@ -377,10 +377,10 @@ Deleting a branch on github_ # delete branch locally git branch -D my-unwanted-branch # delete branch on github - git push origin :my-unwanted-branch + git push origin --delete my-unwanted-branch -(Note the colon ``:`` before ``test-branch``. See also: -https://github.com/guides/remove-a-remote-branch +See also: +https://stackoverflow.com/questions/2003505/how-do-i-delete-a-git-branch-locally-and-remotely Several people sharing a single repository @@ -477,7 +477,7 @@ backport. Pushing changes to the main repo ================================ -*This is only relevant if you have commit rights to the main NumPy repo.* +*Requires commit rights to the main NumPy repo.* When you have a set of "ready" changes in a feature branch ready for NumPy's ``master`` or ``maintenance`` branches, you can push diff --git a/doc/source/dev/gitwash/git_links.inc b/doc/source/dev/gitwash/git_links.inc index f69a3cf62..82e6c75e5 100644 --- a/doc/source/dev/gitwash/git_links.inc +++ b/doc/source/dev/gitwash/git_links.inc @@ -2,7 +2,7 @@ and name substitutions. It may be included in many files, therefore it should only contain link targets and name substitutions. Try grepping for "^\.\. _" to find plausible - candidates for this list. + candidates for this list. .. NOTE: reST targets are __not_case_sensitive__, so only one target definition is needed for @@ -18,7 +18,7 @@ .. _pro git book: https://git-scm.com/book/ .. _git svn crash course: https://git-scm.com/course/svn.html .. _learn.github: https://learn.github.com/ -.. _network graph visualizer: https://github.com/blog/39-say-hello-to-the-network-graph-visualizer +.. _network graph visualizer: https://github.blog/2008-04-10-say-hello-to-the-network-graph-visualizer/ .. _git user manual: https://www.kernel.org/pub/software/scm/git/docs/user-manual.html .. _git tutorial: https://www.kernel.org/pub/software/scm/git/docs/gittutorial.html .. _git community book: https://book.git-scm.com/ @@ -41,18 +41,18 @@ .. _git config: https://www.kernel.org/pub/software/scm/git/docs/git-config.html .. _why the -a flag?: http://www.gitready.com/beginner/2009/01/18/the-staging-area.html .. _git staging area: http://www.gitready.com/beginner/2009/01/18/the-staging-area.html -.. _tangled working copy problem: https://tomayko.com/writings/the-thing-about-git +.. _tangled working copy problem: https://tomayko.com/writings/the-thing-about-git .. _git management: http://kerneltrap.org/Linux/Git_Management .. _linux git workflow: https://www.mail-archive.com/dri-devel@lists.sourceforge.net/msg39091.html -.. _ipython git workflow: http://mail.python.org/pipermail/ipython-dev/2010-October/006746.html +.. _ipython git workflow: https://mail.python.org/pipermail/ipython-dev/2010-October/005632.html .. _git parable: http://tom.preston-werner.com/2009/05/19/the-git-parable.html .. _git foundation: http://matthew-brett.github.com/pydagogue/foundation.html .. _numpy/master: https://github.com/numpy/numpy .. _git cherry-pick: https://www.kernel.org/pub/software/scm/git/docs/git-cherry-pick.html .. _git blame: https://www.kernel.org/pub/software/scm/git/docs/git-blame.html -.. _this blog post: https://github.com/blog/612-introducing-github-compare-view -.. _this article on merging conflicts: https://git-scm.com/book/en/Git-Branching-Basic-Branching-and-Merging#Basic-Merge-Conflicts -.. _learn git: https://www.atlassian.com/git/tutorials/ +.. _this blog post: https://github.com/blog/612-introducing-github-compare-view +.. _this article on merging conflicts: https://git-scm.com/book/en/Git-Branching-Basic-Branching-and-Merging#Basic-Merge-Conflicts +.. _learn git: https://try.github.io/ .. _filing pull requests: https://help.github.com/articles/using-pull-requests/#initiating-the-pull-request .. _pull request review: https://help.github.com/articles/using-pull-requests/#reviewing-the-pull-request diff --git a/doc/source/dev/governance/people.rst b/doc/source/dev/governance/people.rst index 10af7f221..db2704d09 100644 --- a/doc/source/dev/governance/people.rst +++ b/doc/source/dev/governance/people.rst @@ -56,7 +56,7 @@ NumFOCUS Subcommittee Institutional Partners ---------------------- -* UC Berkeley (Stefan van der Walt, Matti Picus, Tyler Reddy, Sebastian Berg) +* UC Berkeley (Stefan van der Walt, Sebastian Berg, Warren Weckesser, Ross Barnowski) -* Quansight (Ralf Gommers, Hameer Abbasi) +* Quansight (Ralf Gommers, Hameer Abbasi, Melissa Weber Mendonça, Mars Lee, Matti Picus) diff --git a/doc/source/dev/howto-docs.rst b/doc/source/dev/howto-docs.rst new file mode 100644 index 000000000..0e2e03e0b --- /dev/null +++ b/doc/source/dev/howto-docs.rst @@ -0,0 +1,149 @@ +.. _howto-docs: + +############################################ +How to contribute to the NumPy documentation +############################################ + +The *Documentation* for a software project is the set of reference, +instructional, educational, informative material generated by the project +developers and contributors, as well as discussions, presentations, videos and +other user-generated content. It may include learning-oriented content (such as +tutorials and how-tos), use-cases or in-depth explanations and reference for +developers. + +If you're reading this page, you probably want to help. This guide is meant to +help you decide which kind of content you'll write, as well as give you some +tips and instructions for submitting it to the official NumPy documentation +(that is, the documentation that ships with NumPy and lives on the +:ref:`official project pages <numpy_docs_mainpage>`). Keep in mind that if you +don't want to do this, writing a tutorial on your own blog, creating a YouTube +video or answering questions on social media or Stack Overflow are also great +contributions! + +NumPy has a Documentation Team. We have open meetings on Zoom every three weeks +and invite everyone to join. Don't hesitate to reach out if you have questions +or just need someone to guide you through your first steps - we're always happy +to help. Meetings are usually announced on the `numpy-discussion mailing list +<https://mail.python.org/mailman/listinfo/numpy-discussion>`__. Meeting minutes +are taken `on hackmd.io <https://hackmd.io/oB_boakvRqKR-_2jRV-Qjg>`__ and stored +in the `NumPy Archive repository <https://github.com/numpy/archive>`__. + +You can find larger planned and in-progress documentation improvement ideas `at +our GitHub project <https://github.com/orgs/numpy/projects/2>`__. + +Current vision for the documentation: NEP 44 +-------------------------------------------- + +Recently, the NumPy community approved a *NumPy Enhancement Proposal (NEP)* +about documentation, `NEP 44 - Restructuring the NumPy Documentation +<https://www.numpy.org/neps/nep-0044-restructuring-numpy-docs>`__. + +**Where is the documentation?** + +The main page for the :ref:`NumPy Documentation <numpy_docs_mainpage>` lists +several categories. The documents mentioned there live in different places. + +- **Tutorials, How-Tos, Explanations:** These documents are stored in the NumPy + source code tree, which means that in order to add them to the official + documentation, you have to download the NumPy source code, + :ref:`build it <howto-build-docs>` and submit your changes via a + :ref:`GitHub pull request <devindex>`. + +- **API Reference:** These are mainly the result of rendering the NumPy code + `docstrings <https://www.python.org/dev/peps/pep-0257/>`__ into formatted + documents. They are automatically generated when the NumPy documentation is + :ref:`built from source<howto-build-docs>`. + +**Datasets** + +If you are writing a tutorial or how-to, we encourage you to use real images and +data (provided they are appropriately licensed and available). This makes the +material more engaging for readers, and choosing the right data can add +pedagogical value to your content. + +*Note: currently we cannot easily use data in other packages (except, e.g., from +SciPy or Matplotlib). We plan to create a dedicated datasets package, but that's +not ready yet - please discuss with us if you have data sources in mind.* + +Creating new content +-------------------- + +The documentation is written in restructuredText, which is the format required +by Sphinx, the tool most Python projects use to automatically build and link the +documentation within the project. You can read the +`Quick reStructuredText Guide +<https://docutils.sourceforge.io/docs/user/rst/quickref.html>`__ or the +`reStructuredText Primer +<http://www.sphinx-doc.org/en/stable/usage/restructuredtext/basics.html>`__ for +more information. + +If you have already decided which type of document you want to write, you can +check out the following specific guides: + +- Guide to writing Tutorials (TODO) +- :ref:`Guide to writing reference (API) documentation: the numpydoc docstring + guide <howto-document>` + +Major additions to the documentation (e.g. new tutorials) should be proposed to +the `mailing list +<https://mail.python.org/mailman/listinfo/numpy-discussion>`__. + +Other ways to contribute +------------------------ + +Correcting technical inaccuracies in the documentation are high priority. For +example, if a docstring is missing a parameter or the description of a +fuction/parameter/method etc. is incorrect. Other "structural" defects like +broken links are also high priority. + +Proposals for changes that improve the clarity of the documentation are welcome. +However, "clarity" is a bit subjective, so such proposals are best made by +raising issues that describe what could be improved in the current +documentation. Proposals that include specific suggestions for the improvement +are encouraged as the proposed changes helps frame the discussion. + +Based on the above characterization, "high-priority" changes (i.e. fixing +technical inaccuracies, broken links, etc.) can be proposed via pull requests +directly as they are straightforward to review. Other changes should be raised +as issues first so that the discussion can happen before you make major +modifications, which in principle saves you from wasting your time on +undesired changes. + +If you see a good tutorial, how-to or explanation that is not included in the +official documentation, you can suggest it to be added by `opening an issue on +GitHub <https://github.com/numpy/numpy/issues>`__. Similarly, opening issues to +suggest a tutorial, how-to or explanation that you can't find anywhere is a +great way to help the documentation team direct efforts towards what users are +looking for. `See this issue <https://github.com/numpy/numpy/issues/15760>`__ +for an example of how to do this. + +Finally, if you detect a typo or an error in the documentation, or would like to +suggest a different approach, you can also open an issue or submit a pull +request with your suggestion. Keep in mind that changes fixing +grammatical/spelling errors are welcome but not necessarily the highest +priority. "Grammatical correctness" often gets confused with "style" which can +result in unfruitful discussions that don't necessarily improve anything. +Changes that modify wording or rearrange phrasing without changing the technical +content are discouraged. If you think that a different wording improves clarity, +you should open an issue as noted above, but again, changes along these lines +very often tend to be highly subjective and not necessarily do much to improve +the quality of the documentation. + +**Final tips** + +- Don't worry if English is not your first language. Do your best - we'll revise + your content and make sure we fix any issues with the code or text. +- If you are unsure whether your tutorial is useful to the community, consider + submitting an issue on GitHub suggesting it, or asking on the mailing + list or Stack Overflow. +- If you are unfamiliar with git/GitHub or the process of submitting a pull + request (PR), check our :ref:`Contributor Guide <devindex>`. + +**Other interesting material** + +- `writethedocs.org <https://www.writethedocs.org/>`__ has a lot of interesting + resources for technical writing. +- Google offers two free `Technical Writing Courses + <https://developers.google.com/tech-writing>`__ +- `Software Carpentry <https://software-carpentry.org/software>`__ has a lot of + nice recommendations for creating educational material. diff --git a/doc/source/dev/index.rst b/doc/source/dev/index.rst index 1aacc2c49..5270bfb77 100644 --- a/doc/source/dev/index.rst +++ b/doc/source/dev/index.rst @@ -1,3 +1,5 @@ +.. _devindex: + ##################### Contributing to NumPy ##################### @@ -10,24 +12,23 @@ we list them in alphabetical order): - Community coordination - DevOps - Developing educational content & narrative documentation -- Writing technical documentation - Fundraising -- Project management - Marketing +- Project management - Translating content - Website design and development +- Writing technical documentation The rest of this document discusses working on the NumPy code base and documentation. We're in the process of updating our descriptions of other activities and roles. If you are interested in these other activities, please contact us! You can do this via -the `numpy-discussion mailing list <https://scipy.org/scipylib/mailing-lists.html>`__, -or on GitHub (open an issue or comment on a relevant issue). These are our preferred -communication channels (open source is open by nature!), however if you prefer -to discuss in private first, please reach out to our community coordinators -at `numpy-team@googlegroups.com` or `numpy-team.slack.com` (send an email to -`numpy-team@googlegroups.com` for an invite the first time). - +the `numpy-discussion mailing list <https://mail.python.org/mailman/listinfo/numpy-discussion>`__, +or on `GitHub <https://github.com/numpy/numpy>`__ (open an issue or comment on a +relevant issue). These are our preferred communication channels (open source is open +by nature!), however if you prefer to discuss in private first, please reach out to +our community coordinators at `numpy-team@googlegroups.com` or `numpy-team.slack.com` +(send an email to `numpy-team@googlegroups.com` for an invite the first time). Development process - summary ============================= @@ -85,7 +86,7 @@ Here's the short summary, complete TOC links are below: * Enter your GitHub username and password (repeat contributors or advanced users can remove this step by connecting to GitHub with - :ref:`SSH<set-up-and-configure-a-github-account>` . + :ref:`SSH<set-up-and-configure-a-github-account>`). * Go to GitHub. The new branch will show up with a green Pull Request button. Make sure the title and message are clear, concise, and self- @@ -198,6 +199,13 @@ Pull requests (PRs) that modify code should either have new tests, or modify exi tests to fail before the PR and pass afterwards. You should :ref:`run the tests <development-environment>` before pushing a PR. +Running NumPy's test suite locally requires some additional packages, such as +``pytest`` and ``hypothesis``. The additional testing dependencies are listed +in ``test_requirements.txt`` in the top-level directory, and can conveniently +be installed with:: + + pip install -r test_requirements.txt + Tests for a module should ideally cover all code in that module, i.e., statement coverage should be at 100%. @@ -231,6 +239,11 @@ Requirements `Sphinx <http://www.sphinx-doc.org/en/stable/>`__ is needed to build the documentation. Matplotlib, SciPy, and IPython are also required. +These additional dependencies for building the documentation are listed in +``doc_requirements.txt`` and can be conveniently installed with:: + + pip install -r doc_requirements.txt + The numpy documentation also depends on the `numpydoc <https://numpydoc.readthedocs.io/en/latest/>`__ sphinx extension as well as an external sphinx theme. @@ -273,6 +286,7 @@ The rest of the story style_guide releasing governance/index + howto-docs NumPy-specific workflow is in :ref:`numpy-development-workflow <development-workflow>`. diff --git a/doc/source/dev/underthehood.rst b/doc/source/dev/underthehood.rst new file mode 100644 index 000000000..4dae48689 --- /dev/null +++ b/doc/source/dev/underthehood.rst @@ -0,0 +1,7 @@ +.. _underthehood: + +=========================================== +Under-the-hood Documentation for developers +=========================================== + +To be completed. diff --git a/doc/source/docs/howto_build_docs.rst b/doc/source/docs/howto_build_docs.rst index f46070d78..29912a5af 100644 --- a/doc/source/docs/howto_build_docs.rst +++ b/doc/source/docs/howto_build_docs.rst @@ -5,12 +5,12 @@ Building the NumPy API and reference docs ========================================= We currently use Sphinx_ for generating the API and reference -documentation for NumPy. You will need Sphinx >= 2.0.0. +documentation for NumPy. You will need Sphinx >= 2.2.0. If you only want to get the documentation, note that pre-built versions can be found at - https://docs.scipy.org/ + https://numpy.org/doc/ in several different formats. @@ -23,23 +23,30 @@ Instructions If you obtained NumPy via git, get also the git submodules that contain additional parts required for building the documentation:: - git submodule init - git submodule update + git submodule update --init In addition, building the documentation requires the Sphinx extension `plot_directive`, which is shipped with Matplotlib_. This Sphinx extension can -be installed by installing Matplotlib. You will also need python3.6. +be installed by installing Matplotlib. You will also need Python>=3.6. Since large parts of the main documentation are obtained from numpy via ``import numpy`` and examining the docstrings, you will need to first build NumPy, and install it so that the correct version is imported. +After NumPy is installed, install SciPy since some of the plots in the random +module require `scipy.special` to display properly. + Note that you can eg. install NumPy to a temporary location and set the PYTHONPATH environment variable appropriately. +Alternatively, if using Python virtual environments (via e.g. ``conda``, +``virtualenv`` or the ``venv`` module), installing numpy into a +new virtual environment is recommended. +All of the necessary dependencies for building the NumPy docs can be installed +with:: -After NumPy is installed, install SciPy since some of the plots in the random -module require `scipy.special` to display properly. Now you are ready to -generate the docs, so write:: + pip install -r doc_requirements.txt + +Now you are ready to generate the docs, so write:: make html @@ -69,9 +76,9 @@ which will rebuild NumPy, install it to a temporary location, and build the documentation in all formats. This will most likely again only work on Unix platforms. -The documentation for NumPy distributed at https://docs.scipy.org in html and -pdf format is also built with ``make dist``. See `HOWTO RELEASE`_ for details on -how to update https://docs.scipy.org. +The documentation for NumPy distributed at https://numpy.org/doc in html and +pdf format is also built with ``make dist``. See `HOWTO RELEASE`_ for details +on how to update https://numpy.org/doc. .. _Matplotlib: https://matplotlib.org/ .. _HOWTO RELEASE: https://github.com/numpy/numpy/blob/master/doc/HOWTO_RELEASE.rst.txt diff --git a/doc/source/docs/howto_document.rst b/doc/source/docs/howto_document.rst index 2a97a100d..cf86b7e99 100644 --- a/doc/source/docs/howto_document.rst +++ b/doc/source/docs/howto_document.rst @@ -4,6 +4,13 @@ A Guide to NumPy/SciPy Documentation ==================================== +User documentation +******************* +NumPy text documents should follow the `Google developer documentation style guide <https://developers.google.com/style>`_. + +Docstrings +********** + When using `Sphinx <http://www.sphinx-doc.org/>`__ in combination with the numpy conventions, you should use the ``numpydoc`` extension so that your docstrings will be handled correctly. For example, Sphinx will extract the diff --git a/doc/source/f2py/getting-started.rst b/doc/source/f2py/f2py.getting-started.rst index 27ddbb005..27ddbb005 100644 --- a/doc/source/f2py/getting-started.rst +++ b/doc/source/f2py/f2py.getting-started.rst diff --git a/doc/source/f2py/index.rst b/doc/source/f2py/index.rst index d6773a76f..492139651 100644 --- a/doc/source/f2py/index.rst +++ b/doc/source/f2py/index.rst @@ -19,7 +19,7 @@ from Python. .. toctree:: :maxdepth: 2 - getting-started + f2py.getting-started signature-file python-usage usage diff --git a/doc/source/reference/arrays.classes.rst b/doc/source/reference/arrays.classes.rst index 6b6f366df..6c8793342 100644 --- a/doc/source/reference/arrays.classes.rst +++ b/doc/source/reference/arrays.classes.rst @@ -326,9 +326,8 @@ NumPy provides several hooks that classes can customize: If a class (ndarray subclass or not) having the :func:`__array__` method is used as the output object of an :ref:`ufunc - <ufuncs-output-type>`, results will be written to the object - returned by :func:`__array__`. Similar conversion is done on - input arrays. + <ufuncs-output-type>`, results will *not* be written to the object + returned by :func:`__array__`. This practice will return ``TypeError``. Matrix objects diff --git a/doc/source/reference/arrays.indexing.rst b/doc/source/reference/arrays.indexing.rst index a425b6e8b..56b99f272 100644 --- a/doc/source/reference/arrays.indexing.rst +++ b/doc/source/reference/arrays.indexing.rst @@ -117,7 +117,7 @@ concepts to remember include: array([5, 6, 7, 8, 9]) - If the number of objects in the selection tuple is less than - *N* , then ``:`` is assumed for any subsequent dimensions. + *N*, then ``:`` is assumed for any subsequent dimensions. .. admonition:: Example diff --git a/doc/source/reference/arrays.interface.rst b/doc/source/reference/arrays.interface.rst index f36a083aa..4e95535c0 100644 --- a/doc/source/reference/arrays.interface.rst +++ b/doc/source/reference/arrays.interface.rst @@ -71,7 +71,7 @@ This approach to the interface consists of the object having an **typestr** (required) - A string providing the basic type of the homogenous array The + A string providing the basic type of the homogeneous array The basic string format consists of 3 parts: a character describing the byteorder of the data (``<``: little-endian, ``>``: big-endian, ``|``: not-relevant), a character code giving the diff --git a/doc/source/reference/arrays.ndarray.rst b/doc/source/reference/arrays.ndarray.rst index 59a925e47..689240c7d 100644 --- a/doc/source/reference/arrays.ndarray.rst +++ b/doc/source/reference/arrays.ndarray.rst @@ -133,6 +133,11 @@ Both the C and Fortran orders are :term:`contiguous`, *i.e.,* single-segment, memory layouts, in which every part of the memory block can be accessed by some combination of the indices. +.. note:: + + `Contiguous arrays` and `single-segment arrays` are synonymous + and are used interchangeably throughout the documentation. + While a C-style and Fortran-style contiguous array, which has the corresponding flags set, can be addressed with the above strides, the actual strides may be different. This can happen in two cases: @@ -158,10 +163,12 @@ base offset itself is a multiple of `self.itemsize`. Understanding .. note:: - Points (1) and (2) are not yet applied by default. Beginning with - NumPy 1.8.0, they are applied consistently only if the environment - variable ``NPY_RELAXED_STRIDES_CHECKING=1`` was defined when NumPy - was built. Eventually this will become the default. + Points (1) and (2) can currently be disabled by the compile time + environmental variable ``NPY_RELAXED_STRIDES_CHECKING=0``, + which was the default before NumPy 1.10. + No users should have to do this. ``NPY_RELAXED_STRIDES_DEBUG=1`` + can be used to help find errors when incorrectly relying on the strides + in C-extension code (see below warning). You can check whether this option was enabled when your NumPy was built by looking at the value of ``np.ones((10,1), diff --git a/doc/source/reference/arrays.nditer.cython.rst b/doc/source/reference/arrays.nditer.cython.rst new file mode 100644 index 000000000..2cc7763ed --- /dev/null +++ b/doc/source/reference/arrays.nditer.cython.rst @@ -0,0 +1,147 @@ +Putting the Inner Loop in Cython +================================ + +Those who want really good performance out of their low level operations +should strongly consider directly using the iteration API provided +in C, but for those who are not comfortable with C or C++, Cython +is a good middle ground with reasonable performance tradeoffs. For +the :class:`nditer` object, this means letting the iterator take care +of broadcasting, dtype conversion, and buffering, while giving the inner +loop to Cython. + +For our example, we'll create a sum of squares function. To start, +let's implement this function in straightforward Python. We want to +support an 'axis' parameter similar to the numpy :func:`sum` function, +so we will need to construct a list for the `op_axes` parameter. +Here's how this looks. + +.. admonition:: Example + + >>> def axis_to_axeslist(axis, ndim): + ... if axis is None: + ... return [-1] * ndim + ... else: + ... if type(axis) is not tuple: + ... axis = (axis,) + ... axeslist = [1] * ndim + ... for i in axis: + ... axeslist[i] = -1 + ... ax = 0 + ... for i in range(ndim): + ... if axeslist[i] != -1: + ... axeslist[i] = ax + ... ax += 1 + ... return axeslist + ... + >>> def sum_squares_py(arr, axis=None, out=None): + ... axeslist = axis_to_axeslist(axis, arr.ndim) + ... it = np.nditer([arr, out], flags=['reduce_ok', + ... 'buffered', 'delay_bufalloc'], + ... op_flags=[['readonly'], ['readwrite', 'allocate']], + ... op_axes=[None, axeslist], + ... op_dtypes=['float64', 'float64']) + ... with it: + ... it.operands[1][...] = 0 + ... it.reset() + ... for x, y in it: + ... y[...] += x*x + ... return it.operands[1] + ... + >>> a = np.arange(6).reshape(2,3) + >>> sum_squares_py(a) + array(55.0) + >>> sum_squares_py(a, axis=-1) + array([ 5., 50.]) + +To Cython-ize this function, we replace the inner loop (y[...] += x*x) with +Cython code that's specialized for the float64 dtype. With the +'external_loop' flag enabled, the arrays provided to the inner loop will +always be one-dimensional, so very little checking needs to be done. + +Here's the listing of sum_squares.pyx:: + + import numpy as np + cimport numpy as np + cimport cython + + def axis_to_axeslist(axis, ndim): + if axis is None: + return [-1] * ndim + else: + if type(axis) is not tuple: + axis = (axis,) + axeslist = [1] * ndim + for i in axis: + axeslist[i] = -1 + ax = 0 + for i in range(ndim): + if axeslist[i] != -1: + axeslist[i] = ax + ax += 1 + return axeslist + + @cython.boundscheck(False) + def sum_squares_cy(arr, axis=None, out=None): + cdef np.ndarray[double] x + cdef np.ndarray[double] y + cdef int size + cdef double value + + axeslist = axis_to_axeslist(axis, arr.ndim) + it = np.nditer([arr, out], flags=['reduce_ok', 'external_loop', + 'buffered', 'delay_bufalloc'], + op_flags=[['readonly'], ['readwrite', 'allocate']], + op_axes=[None, axeslist], + op_dtypes=['float64', 'float64']) + with it: + it.operands[1][...] = 0 + it.reset() + for xarr, yarr in it: + x = xarr + y = yarr + size = x.shape[0] + for i in range(size): + value = x[i] + y[i] = y[i] + value * value + return it.operands[1] + +On this machine, building the .pyx file into a module looked like the +following, but you may have to find some Cython tutorials to tell you +the specifics for your system configuration.:: + + $ cython sum_squares.pyx + $ gcc -shared -pthread -fPIC -fwrapv -O2 -Wall -I/usr/include/python2.7 -fno-strict-aliasing -o sum_squares.so sum_squares.c + +Running this from the Python interpreter produces the same answers +as our native Python/NumPy code did. + +.. admonition:: Example + + >>> from sum_squares import sum_squares_cy + >>> a = np.arange(6).reshape(2,3) + >>> sum_squares_cy(a) + array(55.0) + >>> sum_squares_cy(a, axis=-1) + array([ 5., 50.]) + +Doing a little timing in IPython shows that the reduced overhead and +memory allocation of the Cython inner loop is providing a very nice +speedup over both the straightforward Python code and an expression +using NumPy's built-in sum function.:: + + >>> a = np.random.rand(1000,1000) + + >>> timeit sum_squares_py(a, axis=-1) + 10 loops, best of 3: 37.1 ms per loop + + >>> timeit np.sum(a*a, axis=-1) + 10 loops, best of 3: 20.9 ms per loop + + >>> timeit sum_squares_cy(a, axis=-1) + 100 loops, best of 3: 11.8 ms per loop + + >>> np.all(sum_squares_cy(a, axis=-1) == np.sum(a*a, axis=-1)) + True + + >>> np.all(sum_squares_py(a, axis=-1) == np.sum(a*a, axis=-1)) + True diff --git a/doc/source/reference/arrays.nditer.rst b/doc/source/reference/arrays.nditer.rst index 7dab09a71..2db12a408 100644 --- a/doc/source/reference/arrays.nditer.rst +++ b/doc/source/reference/arrays.nditer.rst @@ -1,5 +1,9 @@ .. currentmodule:: numpy +.. for doctests + The last section on Cython is 'included' at the end of this file. The tests + for that section are disabled. + .. _arrays.nditer: ********************* @@ -218,21 +222,21 @@ produce identical results to the ones in the previous section. >>> it = np.nditer(a, flags=['f_index']) >>> while not it.finished: ... print("%d <%d>" % (it[0], it.index), end=' ') - ... it.iternext() + ... is_not_finished = it.iternext() ... 0 <0> 1 <2> 2 <4> 3 <1> 4 <3> 5 <5> >>> it = np.nditer(a, flags=['multi_index']) >>> while not it.finished: ... print("%d <%s>" % (it[0], it.multi_index), end=' ') - ... it.iternext() + ... is_not_finished = it.iternext() ... 0 <(0, 0)> 1 <(0, 1)> 2 <(0, 2)> 3 <(1, 0)> 4 <(1, 1)> 5 <(1, 2)> >>> with np.nditer(a, flags=['multi_index'], op_flags=['writeonly']) as it: ... while not it.finished: ... it[0] = it.multi_index[1] - it.multi_index[0] - ... it.iternext() + ... is_not_finished = it.iternext() ... >>> a array([[ 0, 1, 2], @@ -316,12 +320,13 @@ specified as an iterator flag. ... op_dtypes=['complex128']): ... print(np.sqrt(x), end=' ') ... - 1.73205080757j 1.41421356237j 1j 0j (1+0j) (1.41421356237+0j) + 1.7320508075688772j 1.4142135623730951j 1j 0j (1+0j) (1.4142135623730951+0j) >>> for x in np.nditer(a, flags=['buffered'], op_dtypes=['complex128']): ... print(np.sqrt(x), end=' ') ... - 1.73205080757j 1.41421356237j 1j 0j (1+0j) (1.41421356237+0j) + 1.7320508075688772j 1.4142135623730951j 1j 0j (1+0j) (1.4142135623730951+0j) + The iterator uses NumPy's casting rules to determine whether a specific conversion is permitted. By default, it enforces 'safe' casting. This means, @@ -405,8 +410,8 @@ which includes the input shapes to help diagnose the problem. ... print("%d:%d" % (x,y), end=' ') ... Traceback (most recent call last): - File "<stdin>", line 1, in <module> - ValueError: operands could not be broadcast together with shapes (2) (2,3) + ... + ValueError: operands could not be broadcast together with shapes (2,) (2,3) Iterator-Allocated Output Arrays -------------------------------- @@ -482,9 +487,9 @@ reasons. >>> square(np.arange(6).reshape(2,3), out=b) Traceback (most recent call last): - File "<stdin>", line 1, in <module> - File "<stdin>", line 4, in square - ValueError: non-broadcastable output operand with shape (3) doesn't match the broadcast shape (2,3) + ... + ValueError: non-broadcastable output operand with shape (3,) doesn't + match the broadcast shape (2,3) Outer Product Iteration ----------------------- @@ -550,7 +555,7 @@ For a simple example, consider taking the sum of all elements in an array. >>> a = np.arange(24).reshape(2,3,4) >>> b = np.array(0) - >>> with np.nditer([a, b], flags=['reduce_ok', 'external_loop'], + >>> with np.nditer([a, b], flags=['reduce_ok'], ... op_flags=[['readonly'], ['readwrite']]) as it: ... for x,y in it: ... y[...] += x @@ -568,7 +573,7 @@ sums along the last axis of `a`. .. admonition:: Example >>> a = np.arange(24).reshape(2,3,4) - >>> it = np.nditer([a, None], flags=['reduce_ok', 'external_loop'], + >>> it = np.nditer([a, None], flags=['reduce_ok'], ... op_flags=[['readonly'], ['readwrite', 'allocate']], ... op_axes=[None, [0,1,-1]]) >>> with it: @@ -602,7 +607,7 @@ buffering. .. admonition:: Example >>> a = np.arange(24).reshape(2,3,4) - >>> it = np.nditer([a, None], flags=['reduce_ok', 'external_loop', + >>> it = np.nditer([a, None], flags=['reduce_ok', ... 'buffered', 'delay_bufalloc'], ... op_flags=[['readonly'], ['readwrite', 'allocate']], ... op_axes=[None, [0,1,-1]]) @@ -617,150 +622,8 @@ buffering. array([[ 6, 22, 38], [54, 70, 86]]) -Putting the Inner Loop in Cython -================================ - -Those who want really good performance out of their low level operations -should strongly consider directly using the iteration API provided -in C, but for those who are not comfortable with C or C++, Cython -is a good middle ground with reasonable performance tradeoffs. For -the :class:`nditer` object, this means letting the iterator take care -of broadcasting, dtype conversion, and buffering, while giving the inner -loop to Cython. - -For our example, we'll create a sum of squares function. To start, -let's implement this function in straightforward Python. We want to -support an 'axis' parameter similar to the numpy :func:`sum` function, -so we will need to construct a list for the `op_axes` parameter. -Here's how this looks. - -.. admonition:: Example - - >>> def axis_to_axeslist(axis, ndim): - ... if axis is None: - ... return [-1] * ndim - ... else: - ... if type(axis) is not tuple: - ... axis = (axis,) - ... axeslist = [1] * ndim - ... for i in axis: - ... axeslist[i] = -1 - ... ax = 0 - ... for i in range(ndim): - ... if axeslist[i] != -1: - ... axeslist[i] = ax - ... ax += 1 - ... return axeslist - ... - >>> def sum_squares_py(arr, axis=None, out=None): - ... axeslist = axis_to_axeslist(axis, arr.ndim) - ... it = np.nditer([arr, out], flags=['reduce_ok', 'external_loop', - ... 'buffered', 'delay_bufalloc'], - ... op_flags=[['readonly'], ['readwrite', 'allocate']], - ... op_axes=[None, axeslist], - ... op_dtypes=['float64', 'float64']) - ... with it: - ... it.operands[1][...] = 0 - ... it.reset() - ... for x, y in it: - ... y[...] += x*x - ... return it.operands[1] - ... - >>> a = np.arange(6).reshape(2,3) - >>> sum_squares_py(a) - array(55.0) - >>> sum_squares_py(a, axis=-1) - array([ 5., 50.]) - -To Cython-ize this function, we replace the inner loop (y[...] += x*x) with -Cython code that's specialized for the float64 dtype. With the -'external_loop' flag enabled, the arrays provided to the inner loop will -always be one-dimensional, so very little checking needs to be done. - -Here's the listing of sum_squares.pyx:: - - import numpy as np - cimport numpy as np - cimport cython - - def axis_to_axeslist(axis, ndim): - if axis is None: - return [-1] * ndim - else: - if type(axis) is not tuple: - axis = (axis,) - axeslist = [1] * ndim - for i in axis: - axeslist[i] = -1 - ax = 0 - for i in range(ndim): - if axeslist[i] != -1: - axeslist[i] = ax - ax += 1 - return axeslist - - @cython.boundscheck(False) - def sum_squares_cy(arr, axis=None, out=None): - cdef np.ndarray[double] x - cdef np.ndarray[double] y - cdef int size - cdef double value - - axeslist = axis_to_axeslist(axis, arr.ndim) - it = np.nditer([arr, out], flags=['reduce_ok', 'external_loop', - 'buffered', 'delay_bufalloc'], - op_flags=[['readonly'], ['readwrite', 'allocate']], - op_axes=[None, axeslist], - op_dtypes=['float64', 'float64']) - with it: - it.operands[1][...] = 0 - it.reset() - for xarr, yarr in it: - x = xarr - y = yarr - size = x.shape[0] - for i in range(size): - value = x[i] - y[i] = y[i] + value * value - return it.operands[1] - -On this machine, building the .pyx file into a module looked like the -following, but you may have to find some Cython tutorials to tell you -the specifics for your system configuration.:: - - $ cython sum_squares.pyx - $ gcc -shared -pthread -fPIC -fwrapv -O2 -Wall -I/usr/include/python2.7 -fno-strict-aliasing -o sum_squares.so sum_squares.c - -Running this from the Python interpreter produces the same answers -as our native Python/NumPy code did. - -.. admonition:: Example - - >>> from sum_squares import sum_squares_cy - >>> a = np.arange(6).reshape(2,3) - >>> sum_squares_cy(a) - array(55.0) - >>> sum_squares_cy(a, axis=-1) - array([ 5., 50.]) - -Doing a little timing in IPython shows that the reduced overhead and -memory allocation of the Cython inner loop is providing a very nice -speedup over both the straightforward Python code and an expression -using NumPy's built-in sum function.:: - - >>> a = np.random.rand(1000,1000) - - >>> timeit sum_squares_py(a, axis=-1) - 10 loops, best of 3: 37.1 ms per loop - - >>> timeit np.sum(a*a, axis=-1) - 10 loops, best of 3: 20.9 ms per loop - - >>> timeit sum_squares_cy(a, axis=-1) - 100 loops, best of 3: 11.8 ms per loop - - >>> np.all(sum_squares_cy(a, axis=-1) == np.sum(a*a, axis=-1)) - True +.. for doctests + Include Cython section separately. Those tests are skipped entirely via an + entry in RST_SKIPLIST - >>> np.all(sum_squares_py(a, axis=-1) == np.sum(a*a, axis=-1)) - True +.. include:: arrays.nditer.cython.rst diff --git a/doc/source/reference/arrays.rst b/doc/source/reference/arrays.rst index faa91a389..497dd9cd6 100644 --- a/doc/source/reference/arrays.rst +++ b/doc/source/reference/arrays.rst @@ -11,7 +11,7 @@ NumPy provides an N-dimensional array type, the :ref:`ndarray type. The items can be :ref:`indexed <arrays.indexing>` using for example N integers. -All ndarrays are :term:`homogenous`: every item takes up the same size +All ndarrays are :term:`homogeneous`: every item takes up the same size block of memory, and all blocks are interpreted in exactly the same way. How each item in the array is to be interpreted is specified by a separate :ref:`data-type object <arrays.dtypes>`, one of which is associated diff --git a/doc/source/reference/c-api/array.rst b/doc/source/reference/c-api/array.rst index 46af1b45b..10c1704c2 100644 --- a/doc/source/reference/c-api/array.rst +++ b/doc/source/reference/c-api/array.rst @@ -251,7 +251,6 @@ From scratch .. versionadded:: 1.6 This function steals a reference to *descr* if it is not NULL. - This array creation routine allows for the convenient creation of a new array matching an existing array's shapes and memory layout, possibly changing the layout and/or data type. @@ -634,8 +633,17 @@ From other objects requirements set to :c:data:`NPY_ARRAY_DEFAULT` and the type_num member of the type argument set to *typenum*. -.. c:function:: PyObject *PyArray_FromObject( \ - PyObject *op, int typenum, int min_depth, int max_depth) +.. c:function:: PyObject* PyArray_ContiguousFromObject( \ + PyObject* op, int typenum, int min_depth, int max_depth) + + This function returns a well-behaved C-style contiguous array from any nested + sequence or array-interface exporting object. The minimum number of dimensions + the array can have is given by `min_depth` while the maximum is `max_depth`. + This is equivalent to call :c:func:`PyArray_FromAny` with requirements + :c:data:`NPY_ARRAY_DEFAULT` and :c:data:`NPY_ARRAY_ENSUREARRAY`. + +.. c:function:: PyObject* PyArray_FromObject( \ + PyObject* op, int typenum, int min_depth, int max_depth) Return an aligned and in native-byteorder array from any nested sequence or array-interface exporting object, op, of a type given by @@ -834,7 +842,7 @@ General check of Python Type .. c:function:: PyArray_IsPythonScalar(op) Evaluates true if *op* is a builtin Python scalar object (int, - float, complex, str, unicode, long, bool). + float, complex, bytes, str, long, bool). .. c:function:: PyArray_IsAnyScalar(op) @@ -1886,7 +1894,7 @@ Item selection and manipulation Equivalent to :meth:`ndarray.sort<numpy.ndarray.sort>` (*self*, *axis*, *kind*). Return an array with the items of *self* sorted along *axis*. The array - is sorted using the algorithm denoted by *kind* , which is an integer/enum pointing + is sorted using the algorithm denoted by *kind*, which is an integer/enum pointing to the type of sorting algorithms used. .. c:function:: PyObject* PyArray_ArgSort(PyArrayObject* self, int axis) @@ -3026,7 +3034,7 @@ Other conversions .. code-block:: c - #define error_converting(x) (((x) == -1) && PyErr_Occurred() + #define error_converting(x) (((x) == -1) && PyErr_Occurred()) .. c:function:: npy_intp PyArray_PyIntAsIntp(PyObject* op) diff --git a/doc/source/reference/c-api/index.rst b/doc/source/reference/c-api/index.rst index 56fe8e473..bb1ed154e 100644 --- a/doc/source/reference/c-api/index.rst +++ b/doc/source/reference/c-api/index.rst @@ -21,7 +21,7 @@ experience at first. Be assured that the task becomes easier with practice, and you may be surprised at how simple the C-code can be to understand. Even if you don't think you can write C-code from scratch, it is much easier to understand and modify already-written source code -then create it *de novo*. +than create it *de novo*. Python extensions are especially straightforward to understand because they all have a very similar structure. Admittedly, NumPy is not a diff --git a/doc/source/reference/global_state.rst b/doc/source/reference/global_state.rst new file mode 100644 index 000000000..2a163390e --- /dev/null +++ b/doc/source/reference/global_state.rst @@ -0,0 +1,85 @@ +.. _global_state: + +************ +Global State +************ + +NumPy has a few import-time, compile-time, or runtime options +which change the global behaviour. +Most of these are related to performance or for debugging +purposes and will not be interesting to the vast majority +of users. + + +Performance-Related Options +=========================== + +Number of Threads used for Linear Algebra +----------------------------------------- + +NumPy itself is normally intentionally limited to a single thread +during function calls, however it does support multiple Python +threads running at the same time. +Note that for performant linear algebra NumPy uses a BLAS backend +such as OpenBLAS or MKL, which may use multiple threads that may +be controlled by environment variables such as ``OMP_NUM_THREADS`` +depending on what is used. +One way to control the number of threads is the package +`threadpoolctl <https://pypi.org/project/threadpoolctl/>`_ + + +Madvise Hugepage on Linux +------------------------- + +When working with very large arrays on modern Linux kernels, +you can experience a significant speedup when +`transparent hugepage <https://www.kernel.org/doc/html/latest/admin-guide/mm/transhuge.html>`_ +is used. +The current system policy for transparent hugepages can be seen by:: + + cat /sys/kernel/mm/transparent_hugepage/enabled + +When set to ``madvise`` NumPy will typically use hugepages for a performance +boost. This behaviour can be modified by setting the environment variable:: + + NUMPY_MADVISE_HUGEPAGE=0 + +or setting it to ``1`` to always enable it. When not set, the default +is to use madvise on Kernels 4.6 and newer. These kernels presumably +experience a large speedup with hugepage support. +This flag is checked at import time. + + +Interoperability-Related Options +================================ + +The array function protocol which allows array-like objects to +hook into the NumPy API is currently enabled by default. +This option exists since NumPy 1.16 and is enabled by default since +NumPy 1.17. It can be disabled using:: + + NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=0 + +See also :py:meth:`numpy.class.__array_function__` for more information. +This flag is checked at import time. + + +Debugging-Related Options +========================= + +Relaxed Strides Checking +------------------------ + +The *compile-time* environment variables:: + + NPY_RELAXED_STRIDES_DEBUG=0 + NPY_RELAXED_STRIDES_CHECKING=1 + +control how NumPy reports contiguity for arrays. +The default that it is enabled and the debug mode is disabled. +This setting should always be enabled. Setting the +debug option can be interesting for testing code written +in C which iterates through arrays that may or may not be +contiguous in memory. +Most users will have no reason to change these, for details +please see the `memory layout <memory-layout>`_ documentation. diff --git a/doc/source/reference/index.rst b/doc/source/reference/index.rst index 6742d605a..2e1dcafa2 100644 --- a/doc/source/reference/index.rst +++ b/doc/source/reference/index.rst @@ -12,7 +12,7 @@ NumPy Reference This reference manual details functions, modules, and objects included in NumPy, describing what they are and what they do. -For learning how to use NumPy, see also :ref:`user`. +For learning how to use NumPy, see the :ref:`complete documentation <numpy_docs_mainpage>`. .. toctree:: @@ -22,6 +22,7 @@ For learning how to use NumPy, see also :ref:`user`. constants ufuncs routines + global_state distutils distutils_guide c-api/index diff --git a/doc/source/reference/maskedarray.baseclass.rst b/doc/source/reference/maskedarray.baseclass.rst index 9864f21ea..5c1bdda23 100644 --- a/doc/source/reference/maskedarray.baseclass.rst +++ b/doc/source/reference/maskedarray.baseclass.rst @@ -33,7 +33,7 @@ defines several constants. Value indicating that a masked array has no invalid entry. :attr:`nomask` is used internally to speed up computations when the mask - is not needed. + is not needed. It is represented internally as ``np.False_``. .. data:: masked_print_options @@ -125,7 +125,6 @@ Conversion MaskedArray.__float__ MaskedArray.__int__ - MaskedArray.__long__ MaskedArray.view MaskedArray.astype diff --git a/doc/source/reference/random/index.rst b/doc/source/reference/random/index.rst index bda9c4d96..d559f2327 100644 --- a/doc/source/reference/random/index.rst +++ b/doc/source/reference/random/index.rst @@ -168,6 +168,9 @@ What's New or Different Python's `random.random`. * All BitGenerators in numpy use `SeedSequence` to convert seeds into initialized states. +* The addition of an ``axis`` keyword argument to methods such as + `Generator.choice`, `Generator.permutation`, and `Generator.shuffle` + improves support for sampling from and shuffling multi-dimensional arrays. See :ref:`new-or-different` for a complete list of improvements and differences from the traditional ``Randomstate``. diff --git a/doc/source/reference/random/new-or-different.rst b/doc/source/reference/random/new-or-different.rst index b3bddb443..03e7775a0 100644 --- a/doc/source/reference/random/new-or-different.rst +++ b/doc/source/reference/random/new-or-different.rst @@ -52,36 +52,37 @@ And in more detail: methods which are 2-10 times faster than NumPy's default implementation in `~.Generator.standard_normal`, `~.Generator.standard_exponential` or `~.Generator.standard_gamma`. -* `~.Generator.integers` is now the canonical way to generate integer - random numbers from a discrete uniform distribution. The ``rand`` and - ``randn`` methods are only available through the legacy `~.RandomState`. - This replaces both ``randint`` and the deprecated ``random_integers``. -* The Box-Muller method used to produce NumPy's normals is no longer available. -* All bit generators can produce doubles, uint64s and - uint32s via CTypes (`~PCG64.ctypes`) and CFFI (`~PCG64.cffi`). - This allows these bit generators to be used in numba. -* The bit generators can be used in downstream projects via - Cython. - + .. ipython:: python from numpy.random import Generator, PCG64 import numpy.random rg = Generator(PCG64()) - %timeit rg.standard_normal(100000) - %timeit numpy.random.standard_normal(100000) + %timeit -n 1 rg.standard_normal(100000) + %timeit -n 1 numpy.random.standard_normal(100000) .. ipython:: python - %timeit rg.standard_exponential(100000) - %timeit numpy.random.standard_exponential(100000) + %timeit -n 1 rg.standard_exponential(100000) + %timeit -n 1 numpy.random.standard_exponential(100000) .. ipython:: python - %timeit rg.standard_gamma(3.0, 100000) - %timeit numpy.random.standard_gamma(3.0, 100000) + %timeit -n 1 rg.standard_gamma(3.0, 100000) + %timeit -n 1 numpy.random.standard_gamma(3.0, 100000) + +* `~.Generator.integers` is now the canonical way to generate integer + random numbers from a discrete uniform distribution. The ``rand`` and + ``randn`` methods are only available through the legacy `~.RandomState`. + This replaces both ``randint`` and the deprecated ``random_integers``. +* The Box-Muller method used to produce NumPy's normals is no longer available. +* All bit generators can produce doubles, uint64s and + uint32s via CTypes (`~PCG64.ctypes`) and CFFI (`~PCG64.cffi`). + This allows these bit generators to be used in numba. +* The bit generators can be used in downstream projects via + Cython. * Optional ``dtype`` argument that accepts ``np.float32`` or ``np.float64`` to produce either single or double prevision uniform random variables for select distributions @@ -114,3 +115,15 @@ And in more detail: rg.random(out=existing[:2]) print(existing) +* Optional ``axis`` argument for methods like `~.Generator.choice`, + `~.Generator.permutation` and `~.Generator.shuffle` that controls which + axis an operation is performed over for multi-dimensional arrays. + +.. ipython:: python + + rg = Generator(PCG64(123456789)) + a = np.arange(12).reshape((3, 4)) + a + rg.choice(a, axis=1, size=5) + rg.shuffle(a, axis=1) # Shuffle in-place + a diff --git a/doc/source/reference/random/performance.rst b/doc/source/reference/random/performance.rst index d70dd064a..74dad4cc3 100644 --- a/doc/source/reference/random/performance.rst +++ b/doc/source/reference/random/performance.rst @@ -81,7 +81,7 @@ performance was computed using a geometric mean. .. note:: - All timings were taken using Linux on a i5-3570 processor. + All timings were taken using Linux on an i5-3570 processor. Performance on different Operating Systems ****************************************** @@ -150,4 +150,4 @@ Exponentials 100 33.7 26.3 109.8 Linux timings used Ubuntu 18.04 and GCC 7.4. Windows timings were made on Windows 10 using Microsoft C/C++ Optimizing Compiler Version 19 (Visual - Studio 2015). All timings were produced on a i5-3570 processor. + Studio 2015). All timings were produced on an i5-3570 processor. diff --git a/doc/source/reference/routines.array-manipulation.rst b/doc/source/reference/routines.array-manipulation.rst index bf43232ef..8d13a1800 100644 --- a/doc/source/reference/routines.array-manipulation.rst +++ b/doc/source/reference/routines.array-manipulation.rst @@ -69,11 +69,11 @@ Joining arrays concatenate stack - column_stack - dstack - hstack - vstack block + vstack + hstack + dstack + column_stack Splitting arrays ================ diff --git a/doc/source/reference/routines.dual.rst b/doc/source/reference/routines.dual.rst index 4ed7098d6..01814e9a7 100644 --- a/doc/source/reference/routines.dual.rst +++ b/doc/source/reference/routines.dual.rst @@ -1,4 +1,4 @@ -Optionally Scipy-accelerated routines (:mod:`numpy.dual`) +Optionally SciPy-accelerated routines (:mod:`numpy.dual`) ********************************************************* .. automodule:: numpy.dual diff --git a/doc/source/reference/routines.ma.rst b/doc/source/reference/routines.ma.rst index 5b2098c7a..97859ac67 100644 --- a/doc/source/reference/routines.ma.rst +++ b/doc/source/reference/routines.ma.rst @@ -145,13 +145,13 @@ Joining arrays .. autosummary:: :toctree: generated/ + ma.concatenate ma.stack + ma.vstack + ma.hstack + ma.dstack ma.column_stack - ma.concatenate ma.append - ma.dstack - ma.hstack - ma.vstack _____ @@ -396,6 +396,7 @@ Miscellanea ma.allequal ma.allclose ma.apply_along_axis + ma.apply_over_axes ma.arange ma.choose ma.ediff1d diff --git a/doc/source/reference/ufuncs.rst b/doc/source/reference/ufuncs.rst index 20c89e0b3..6d58d1a6d 100644 --- a/doc/source/reference/ufuncs.rst +++ b/doc/source/reference/ufuncs.rst @@ -380,7 +380,7 @@ advanced usage and will not typically be used. result as a dimension with size one, so that the result will broadcast correctly against the inputs. This option can only be used for generalized ufuncs that operate on inputs that all have the same number of core - dimensions and with outputs that have no core dimensions , i.e., with + dimensions and with outputs that have no core dimensions, i.e., with signatures like ``(i),(i)->()`` or ``(m,m)->()``. If used, the location of the dimensions in the output can be controlled with ``axes`` and ``axis``. @@ -441,13 +441,13 @@ advanced usage and will not typically be used. *extobj* - a list of length 1, 2, or 3 specifying the ufunc buffer-size, the - error mode integer, and the error call-back function. Normally, these + a list of length 3 specifying the ufunc buffer-size, the error + mode integer, and the error call-back function. Normally, these values are looked up in a thread-specific dictionary. Passing them here circumvents that look up and uses the low-level specification - provided for the error mode. This may be useful, for example, as an - optimization for calculations requiring many ufunc calls on small arrays - in a loop. + provided for the error mode. This may be useful, for example, as + an optimization for calculations requiring many ufunc calls on + small arrays in a loop. diff --git a/doc/source/release.rst b/doc/source/release.rst index ea4c6320c..5a890178c 100644 --- a/doc/source/release.rst +++ b/doc/source/release.rst @@ -5,7 +5,10 @@ Release Notes .. toctree:: :maxdepth: 3 + 1.20.0 <release/1.20.0-notes> 1.19.0 <release/1.19.0-notes> + 1.18.4 <release/1.18.4-notes> + 1.18.3 <release/1.18.3-notes> 1.18.2 <release/1.18.2-notes> 1.18.1 <release/1.18.1-notes> 1.18.0 <release/1.18.0-notes> diff --git a/doc/source/release/1.18.3-notes.rst b/doc/source/release/1.18.3-notes.rst new file mode 100644 index 000000000..1ebad52b8 --- /dev/null +++ b/doc/source/release/1.18.3-notes.rst @@ -0,0 +1,45 @@ +.. currentmodule:: numpy + +========================== +NumPy 1.18.3 Release Notes +========================== + +This release contains various bug/regression fixes. + +The Python versions supported in this release are 3.5-3.8. Downstream +developers should use Cython >= 0.29.15 for Python 3.8 support and OpenBLAS >= +3.7 to avoid errors on the Skylake architecture. + + +Highlights +========== + +* Fix for the `method='eigh'` and `method='cholesky'` methods in + `numpy.random.multivariate_normal`. Those were producing samples from the + wrong distribution. + + +Contributors +============ + +A total of 6 people contributed to this release. People with a "+" by their +names contributed a patch for the first time. + +* Charles Harris +* Max Balandat + +* @Mibu287 + +* Pan Jan + +* Sebastian Berg +* @panpiort8 + + + +Pull requests merged +==================== + +A total of 5 pull requests were merged for this release. + +* `#15916 <https://github.com/numpy/numpy/pull/15916>`__: BUG: Fix eigh and cholesky methods of numpy.random.multivariate_normal +* `#15929 <https://github.com/numpy/numpy/pull/15929>`__: BUG,MAINT: Remove incorrect special case in string to number... +* `#15930 <https://github.com/numpy/numpy/pull/15930>`__: BUG: Guarantee array is in valid state after memory error occurs... +* `#15954 <https://github.com/numpy/numpy/pull/15954>`__: BUG: Check that `pvals` is 1D in `_generator.multinomial`. +* `#16017 <https://github.com/numpy/numpy/pull/16017>`__: BUG: Alpha parameter must be 1D in `generator.dirichlet` diff --git a/doc/source/release/1.18.4-notes.rst b/doc/source/release/1.18.4-notes.rst new file mode 100644 index 000000000..25ef1d127 --- /dev/null +++ b/doc/source/release/1.18.4-notes.rst @@ -0,0 +1,38 @@ +.. currentmodule:: numpy + +========================== +NumPy 1.18.4 Release Notes +========================== + +This is the last planned release in the 1.18.x series. It reverts the +``bool("0")`` behavior introduced in 1.18.3 and fixes a bug in +``Generator.integers``. There is also a link to a new troubleshooting section +in the documentation included in the error message emitted when numpy import +fails. + +The Python versions supported in this release are 3.5-3.8. Downstream +developers should use Cython >= 0.29.15 for Python 3.8 support and +OpenBLAS >= 3.7 to avoid errors on the Skylake architecture. + +Contributors +============ + +A total of 4 people contributed to this release. People with a "+" by their +names contributed a patch for the first time. + +* Charles Harris +* Matti Picus +* Sebastian Berg +* Warren Weckesser + +Pull requests merged +==================== + +A total of 6 pull requests were merged for this release. + +* `#16055 <https://github.com/numpy/numpy/pull/16055>`__: BLD: add i686 for 1.18 builds +* `#16090 <https://github.com/numpy/numpy/pull/16090>`__: BUG: random: ``Generator.integers(2**32)`` always returned 0. +* `#16091 <https://github.com/numpy/numpy/pull/16091>`__: BLD: fix path to libgfortran on macOS +* `#16109 <https://github.com/numpy/numpy/pull/16109>`__: REV: Reverts side-effect changes to casting +* `#16114 <https://github.com/numpy/numpy/pull/16114>`__: BLD: put openblas library in local directory on windows +* `#16132 <https://github.com/numpy/numpy/pull/16132>`__: DOC: Change import error "howto" to link to new troubleshooting... diff --git a/doc/source/release/1.19.0-notes.rst b/doc/source/release/1.19.0-notes.rst index 6e7fd69d4..ea0ac6193 100644 --- a/doc/source/release/1.19.0-notes.rst +++ b/doc/source/release/1.19.0-notes.rst @@ -1,3 +1,472 @@ +========================== +NumPy 1.19.0 Release Notes +========================== + + +Highlights +========== + +* Code compatibility with Python versions < 3.5 (including Python 2) was + dropped from both the python and C code. The shims in numpy.compat will + remain to support third-party packages, but they may be deprecated in a + future release. + + (`gh-15233 <https://github.com/numpy/numpy/pull/15233>`__) + + +Deprecations +============ + + +Deprecate automatic ``dtype=object`` for ragged input +----------------------------------------------------- +Calling ``np.array([[1, [1, 2, 3]])`` will issue a ``DeprecationWarning`` as +per `NEP 34`_. Users should explicitly use ``dtype=object`` to avoid the +warning. + +.. _`NEP 34`: https://numpy.org/neps/nep-0034.html + +(`gh-15119 <https://github.com/numpy/numpy/pull/15119>`__) + +Passing ``shape=0`` to factory functions in ``numpy.rec`` is deprecated +----------------------------------------------------------------------- + +``0`` is treated as a special case and is aliased to ``None`` in the functions: + +* `numpy.core.records.fromarrays` +* `numpy.core.records.fromrecords` +* `numpy.core.records.fromstring` +* `numpy.core.records.fromfile` + +In future, ``0`` will not be special cased, and will be treated as an array +length like any other integer. + +(`gh-15217 <https://github.com/numpy/numpy/pull/15217>`__) + +Deprecation of probably unused C-API functions +---------------------------------------------- +The following C-API functions are probably unused and have been +deprecated: + +* ``PyArray_GetArrayParamsFromObject`` +* ``PyUFunc_GenericFunction`` +* ``PyUFunc_SetUsesArraysAsData`` + +In most cases ``PyArray_GetArrayParamsFromObject`` should be replaced +by converting to an array, while ``PyUFunc_GenericFunction`` can be +replaced with ``PyObject_Call`` (see documentation for details). + +(`gh-15427 <https://github.com/numpy/numpy/pull/15427>`__) + +Converting certain types to dtypes is Deprecated +------------------------------------------------ +The super classes of scalar types, such as ``np.integer``, ``np.generic``, +or ``np.inexact`` will now give a deprecation warning when converted +to a dtype (or used in a dtype keyword argument). +The reason for this is that `np.integer` is converted to ``np.int_``, +while it would be expected to represent *any* integer (e.g. also +``int8``, ``int16``, etc. +For example, ``dtype=np.floating`` is currently identical to +``dtype=np.float64``, even though also ``np.float32`` is a subclass of +``np.floating``. + +(`gh-15534 <https://github.com/numpy/numpy/pull/15534>`__) + +Deprecation of `round` for ``np.complexfloating`` scalars +----------------------------------------------------------- + +Output of the ``__round__`` dunder method and consequently the Python built-in +`round` has been deprecated on complex scalars. This does not affect +`np.round`. + +(`gh-15840 <https://github.com/numpy/numpy/pull/15840>`__) + +``numpy.ndarray.tostring()`` is deprecated in favor of ``tobytes()`` +-------------------------------------------------------------------- +`~numpy.ndarray.tobytes` has existed since the 1.9 release, but until this +release `~numpy.ndarray.tostring` emitted no warning. The change to emit a +warning brings NumPy in line with the builtin `array.array` methods of the +same name. + +(`gh-15867 <https://github.com/numpy/numpy/pull/15867>`__) + + +Expired deprecations +==================== + +`numpy.insert` and `numpy.delete` can no longer be passed an axis on 0d arrays +------------------------------------------------------------------------------ +This concludes a deprecation from 1.9, where when an ``axis`` argument was +passed to a call to `~numpy.insert` and `~numpy.delete` on a 0d array, the +``axis`` and ``obj`` argument and indices would be completely ignored. +In these cases, ``insert(arr, "nonsense", 42, axis=0)`` would actually overwrite the +entire array, while ``delete(arr, "nonsense", axis=0)`` would be ``arr.copy()`` + +Now passing ``axis`` on a 0d array raises `~numpy.AxisError`. + +(`gh-15802 <https://github.com/numpy/numpy/pull/15802>`__) + +`numpy.delete` no longer ignores out-of-bounds indices +------------------------------------------------------ +This concludes deprecations from 1.8 and 1.9, where ``np.delete`` would ignore +both negative and out-of-bounds items in a sequence of indices. This was at +odds with its behavior when passed a single index. + +Now out-of-bounds items throw ``IndexError``, and negative items index from the +end. + +(`gh-15804 <https://github.com/numpy/numpy/pull/15804>`__) + +`numpy.insert` and `numpy.delete` no longer accept non-integral indices +----------------------------------------------------------------------- +This concludes a deprecation from 1.9, where sequences of non-integers indices +were allowed and cast to integers. Now passing sequences of non-integral +indices raises ``IndexError``, just like it does when passing a single +non-integral scalar. + +(`gh-15805 <https://github.com/numpy/numpy/pull/15805>`__) + +`numpy.delete` no longer casts boolean indices to integers +---------------------------------------------------------- +This concludes a deprecation from 1.8, where ``np.delete`` would cast boolean +arrays and scalars passed as an index argument into integer indices. The +behavior now is to treat boolean arrays as a mask, and to raise an error +on boolean scalars. + +(`gh-15815 <https://github.com/numpy/numpy/pull/15815>`__) + + +Compatibility notes +=================== + +Changed random variate stream from `numpy.random.Generator.dirichlet` +--------------------------------------------------------------------- +A bug in the generation of random variates for the Dirichlet distribution +with small `alpha` values was fixed by using a different algorithm when +``max(alpha) < 0.1``. Because of the change, the stream of variates +generated by `dirichlet` in this case will be different from previous +releases. + +(`gh-14924 <https://github.com/numpy/numpy/pull/14924>`__) + +Scalar promotion in ``PyArray_ConvertToCommonType`` +--------------------------------------------------- + +The promotion of mixed scalars and arrays in ``PyArray_ConvertToCommonType`` +has been changed to adhere to those used by ``np.result_type``. +This means that input such as ``(1000, np.array([1], dtype=np.uint8)))`` +will now return ``uint16`` dtypes. In most cases the behaviour is unchanged. +Note that the use of this C-API function is generally discouraged. +This also fixes ``np.choose`` to behave the same way as the rest of NumPy +in this respect. + +(`gh-14933 <https://github.com/numpy/numpy/pull/14933>`__) + +Fasttake and fastputmask slots are deprecated and NULL'ed +--------------------------------------------------------- +The fasttake and fastputmask slots are now never used and +must always be set to NULL. This will result in no change in behaviour. +However, if a user dtype should set one of these a DeprecationWarning +will be given. + +(`gh-14942 <https://github.com/numpy/numpy/pull/14942>`__) + +`np.ediff1d` casting behaviour with ``to_end`` and ``to_begin`` +--------------------------------------------------------------- + +`np.ediff1d` now uses the ``"same_kind"`` casting rule for +its additional ``to_end`` and ``to_begin`` arguments. This +ensures type safety except when the input array has a smaller +integer type than ``to_begin`` or ``to_end``. +In rare cases, the behaviour will be more strict than it was +previously in 1.16 and 1.17. This is necessary to solve issues +with floating point NaN. + +(`gh-14981 <https://github.com/numpy/numpy/pull/14981>`__) + +Converting of empty array-like objects to NumPy arrays +------------------------------------------------------ +Objects with ``len(obj) == 0`` which implement an "array-like" interface, +meaning an object implementing ``obj.__array__()``, +``obj.__array_interface__``, ``obj.__array_struct__``, or the python +buffer interface and which are also sequences (i.e. Pandas objects) +will now always retain there shape correctly when converted to an array. +If such an object has a shape of ``(0, 1)`` previously, it could +be converted into an array of shape ``(0,)`` (losing all dimensions +after the first 0). + +(`gh-14995 <https://github.com/numpy/numpy/pull/14995>`__) + +Removed ``multiarray.int_asbuffer`` +----------------------------------- + +As part of the continued removal of Python 2 compatibility, +``multiarray.int_asbuffer`` was removed. On Python 3, it threw a +``NotImplementedError`` and was unused internally. It is expected that there +are no downstream use cases for this method with Python 3. + +(`gh-15229 <https://github.com/numpy/numpy/pull/15229>`__) + +``numpy.distutils.compat`` has been removed +------------------------------------------- +This module contained only the function ``get_exception()``, which was used as:: + + try: + ... + except Exception: + e = get_exception() + +Its purpose was to handle the change in syntax introduced in Python 2.6, from +``except Exception, e:`` to ``except Exception as e:``, meaning it was only +necessary for codebases supporting Python 2.5 and older. + +(`gh-15255 <https://github.com/numpy/numpy/pull/15255>`__) + +``issubdtype`` no longer interprets ``float`` as ``np.floating`` +---------------------------------------------------------------- + +`numpy.issubdtype` had a FutureWarning since NumPy 1.14 which +has expired now. This means that certain input where the second +argument was neither a datatype nor a NumPy scalar type +(such as a string or a python type like ``int`` or ``float``) +will now be consistent with passing in ``np.dtype(arg2).type``. +This makes the result consistent with expectations and leads to +a false result in some cases which previously returned true. + +(`gh-15773 <https://github.com/numpy/numpy/pull/15773>`__) + +Change output of ``round`` on scalars to be consistent with Python +------------------------------------------------------------------ + +Output of the ``__round__`` dunder method and consequently the Python +built-in `round` has been changed to be a Python `int` to be consistent +with calling it on Python ``float`` objects when called with no arguments. +Previously, it would return a scalar of the `np.dtype` that was passed in. + +(`gh-15840 <https://github.com/numpy/numpy/pull/15840>`__) + +The ``numpy.ndarray`` constructor no longer interprets ``strides=()`` as ``strides=None`` +----------------------------------------------------------------------------------------- +The former has changed to have the expected meaning of setting +`numpy.ndarray.strides` to ``()``, while the latter continues to result in +strides being chosen automatically. + +(`gh-15882 <https://github.com/numpy/numpy/pull/15882>`__) + +C-Level string to datetime casts changed +---------------------------------------- +The C-level casts from strings were simplified. This changed +also fixes string to datetime and timedelta casts to behave +correctly (i.e. like Python casts using ``string_arr.astype("M8")`` +while previously the cast would behave like +``string_arr.astype(np.int_).astype("M8")``. +This only affects code using low-level C-API to do manual casts +(not full array casts) of single scalar values or using e.g. +``PyArray_GetCastFunc``, and should thus not affect the vast majority +of users. + +(`gh-16068 <https://github.com/numpy/numpy/pull/16068>`__) + + +C API changes +============= + +Better support for ``const`` dimensions in API functions +-------------------------------------------------------- +The following functions now accept a constant array of ``npy_intp``: + +* `PyArray_BroadcastToShape` +* `PyArray_IntTupleFromIntp` +* `PyArray_OverflowMultiplyList` + +Previously the caller would have to cast away the const-ness to call these +functions. + +(`gh-15251 <https://github.com/numpy/numpy/pull/15251>`__) + +Const qualify UFunc inner loops
+-------------------------------
+``UFuncGenericFunction`` now expects pointers to const ``dimension`` and
+``strides`` as arguments. This means inner loops may no longer modify
+either ``dimension`` or ``strides``. This change leads to an
+``incompatible-pointer-types`` warning forcing users to either ignore
+the compiler warnings or to const qualify their own loop signatures. + +(`gh-15355 <https://github.com/numpy/numpy/pull/15355>`__) + + +New Features +============ + +`numpy.frompyfunc` now accepts an identity argument +--------------------------------------------------- +This allows the :attr:`numpy.ufunc.identity` attribute to be set on the +resulting ufunc, meaning it can be used for empty and multi-dimensional +calls to :meth:`numpy.ufunc.reduce`. + +(`gh-8255 <https://github.com/numpy/numpy/pull/8255>`__) + +``np.str_`` scalars now support the buffer protocol +--------------------------------------------------- +``np.str_`` arrays are always stored as UCS4, so the corresponding scalars +now expose this through the buffer interface, meaning +``memoryview(np.str_('test'))`` now works. + +(`gh-15385 <https://github.com/numpy/numpy/pull/15385>`__) + +``subok`` option for `numpy.copy` +--------------------------------- +A new kwarg, ``subok``, was added to `numpy.copy` to allow users to toggle the +behavior of `numpy.copy` with respect to array subclasses. The default value +is ``False`` which is consistent with the behavior of `numpy.copy` for +previous numpy versions. To create a copy that preserves an array subclass with +`numpy.copy`, call ``np.copy(arr, subok=True)``. This addition better documents +that the default behavior of `numpy.copy` differs from the +`numpy.ndarray.copy` method which respects array subclasses by default. + +(`gh-15685 <https://github.com/numpy/numpy/pull/15685>`__) + +`numpy.linalg.multi_dot` now accepts an ``out`` argument +-------------------------------------------------------- + +``out`` can be used to avoid creating unnecessary copies of the final product +computed by `numpy.linalg.multidot`. + +(`gh-15715 <https://github.com/numpy/numpy/pull/15715>`__) + +``keepdims`` parameter for `numpy.count_nonzero` +------------------------------------------------ +The parameter ``keepdims`` was added to `numpy.count_nonzero`. The +parameter has the same meaning as it does in reduction functions such +as `numpy.sum` or `numpy.mean`. + +(`gh-15870 <https://github.com/numpy/numpy/pull/15870>`__) + +``equal_nan`` parameter for `numpy.array_equal` +------------------------------------------------ +The keyword argument ``equal_nan`` was added to `numpy.array_equal`. +``equal_nan`` is a boolean value that toggles whether or not ``nan`` values +are considered equal in comparison (default is ``False``). This matches API +used in related functions such as `numpy.isclose` and `numpy.allclose`. + +(`gh-16128 <https://github.com/numpy/numpy/pull/16128>`__) + + +Improvements +============ + +Improve detection of CPU features +================================= + +Replace ``npy_cpu_supports`` which was a gcc-specific mechanism to test support +of avx with more general functions ``npy_cpu_init`` and ``npy_cpu_have``, and +expose the results via a ``NPY_CPU_HAVE`` c-macro as well as a python-level +``__cpu_features__`` dictionary. + +(`gh-13421 <https://github.com/numpy/numpy/pull/13421>`__) + +Use 64-bit integer size on 64-bit platforms in fallback lapack_lite +------------------------------------------------------------------- + +Use 64-bit integer size on 64-bit platforms in the fallback LAPACK library, +which is used when the system has no LAPACK installed, allowing it to deal with +linear algebra for large arrays. + +(`gh-15218 <https://github.com/numpy/numpy/pull/15218>`__) + +Use AVX512 intrinsic to implement ``np.exp`` when input is ``np.float64`` +-------------------------------------------------------------------------- +Use AVX512 intrinsic to implement ``np.exp`` when input is ``np.float64``, +which can improve the performance of ``np.exp`` with ``np.float64`` input 5-7x +faster than before. The _multiarray_umath.so module has grown about 63 KB on +linux64. + +(`gh-15648 <https://github.com/numpy/numpy/pull/15648>`__) + +Ability to disable madvise hugepages +------------------------------------ + +On Linux NumPy has previously added support for madavise +hugepages which can improve performance for very large arrays. +Unfortunately, on older Kernel versions this led to peformance +regressions, thus by default the support has been disabled on +kernels before version 4.6. To override the default, you can +use the environment variable:: + + NUMPY_MADVISE_HUGEPAGE=0 + +or set it to 1 to force enabling support. Note that this only makes +a difference if the operating system is set up to use madvise +transparent hugepage. + +(`gh-15769 <https://github.com/numpy/numpy/pull/15769>`__) + +`numpy.einsum` accepts NumPy ``int64`` type in subscript list +------------------------------------------------------------- +There is no longer a type error thrown when `numpy.einsum` is passed +a NumPy ``int64`` array as its subscript list. + +(`gh-16080 <https://github.com/numpy/numpy/pull/16080>`__) + +``np.logaddexp2.identity`` changed to ``-inf`` +---------------------------------------------- +The ufunc `~numpy.logaddexp2` now has an identity of ``-inf``, allowing it to +be called on empty sequences. This matches the identity of `~numpy.logaddexp`. + +(`gh-16102 <https://github.com/numpy/numpy/pull/16102>`__) + + +Changes +======= + +Remove handling of extra argument to ``__array__`` +-------------------------------------------------- +A code path and test have been in the code since NumPy 0.4 for a two-argument +variant of ``__array__(dtype=None, context=None)``. It was activated when +calling ``ufunc(op)`` or ``ufunc.reduce(op)`` if ``op.__array__`` existed. +However that variant is not documented, and it is not clear what the intention +was for its use. It has been removed. + +(`gh-15118 <https://github.com/numpy/numpy/pull/15118>`__) + +``numpy.random._bit_generator`` moved to ``numpy.random.bit_generator`` +----------------------------------------------------------------------- + +In order to expose `numpy.random.BitGenerator` and `numpy.random.SeedSequence` +to cython, the ``_bitgenerator`` module is now public as +`numpy.random.bit_generator` + +Cython access to the random distributions is provided via a `pxd` file +---------------------------------------------------------------------- + +``c_distributions.pxd`` provides access to the c functions behind many of the +random distributions from Cython, making it convenient to use and extend them. + +(`gh-15463 <https://github.com/numpy/numpy/pull/15463>`__) + +`Fixed `eigh` and `cholesky` methods in `numpy.random.multivariate_normal`` +--------------------------------------------------------------------------- + +Previously, when passing `method='eigh'` or `method='cholesky'`, +`numpy.random.multivariate_normal` produced samples from the wrong +distribution. This is now fixed. + +(`gh-15872 <https://github.com/numpy/numpy/pull/15872>`__) + +Fixed the jumping implementation in ``MT19937.jumped`` +------------------------------------------------------ + +This fix changes the stream produced from jumped MT19937 generators. It does +not affect the stream produced using ``RandomState`` or ``MT19937`` that +are directly seeded. + +The translation of the jumping code for the MT19937 contained a reversed loop +ordering. ``MT19937.jumped`` matches the Makoto Matsumoto's original +implementation of the Horner and Sliding Window jump methods. + +(`gh-16153 <https://github.com/numpy/numpy/pull/16153>`__) + + .. currentmodule:: numpy ========================== diff --git a/doc/source/release/1.20.0-notes.rst b/doc/source/release/1.20.0-notes.rst new file mode 100644 index 000000000..d91bea762 --- /dev/null +++ b/doc/source/release/1.20.0-notes.rst @@ -0,0 +1,6 @@ +.. currentmodule:: numpy + +========================== +NumPy 1.20.0 Release Notes +========================== + diff --git a/doc/source/user/absolute_beginners.rst b/doc/source/user/absolute_beginners.rst index 021f49bcb..bd44b70da 100644 --- a/doc/source/user/absolute_beginners.rst +++ b/doc/source/user/absolute_beginners.rst @@ -100,8 +100,8 @@ What’s the difference between a Python list and a NumPy array? NumPy gives you an enormous range of fast and efficient ways of creating arrays and manipulating numerical data inside them. While a Python list can contain different data types within a single list, all of the elements in a NumPy array -should be homogenous. The mathematical operations that are meant to be performed -on arrays would be extremely inefficient if the arrays weren't homogenous. +should be homogeneous. The mathematical operations that are meant to be performed +on arrays would be extremely inefficient if the arrays weren't homogeneous. **Why use NumPy?** @@ -1576,6 +1576,10 @@ If you created this array "a" :: ... [ 0.76989341, 0.81299683, -0.95068423, 0.11769564], ... [ 0.20484034, 0.34784527, 1.96979195, 0.51992837]]) +.. for doctests + The continuous integration truncates dataframe display without this setting. + >>> pd.set_option('max_columns', 10) + You could create a Pandas dataframe :: >>> df = pd.DataFrame(a) diff --git a/doc/source/user/building.rst b/doc/source/user/building.rst index c88a6cd1e..47a0a03c9 100644 --- a/doc/source/user/building.rst +++ b/doc/source/user/building.rst @@ -23,11 +23,11 @@ Building NumPy requires the following software installed: Various NumPy modules use FORTRAN 77 libraries, so you'll also need a FORTRAN 77 compiler installed. - Note that NumPy is developed mainly using GNU compilers. Compilers from - other vendors such as Intel, Absoft, Sun, NAG, Compaq, Vast, Portland, - Lahey, HP, IBM, Microsoft are only supported in the form of community - feedback, and may not work out of the box. GCC 4.x (and later) compilers - are recommended. + Note that NumPy is developed mainly using GNU compilers and tested on + MSVC and Clang compilers. Compilers from other vendors such as Intel, + Absoft, Sun, NAG, Compaq, Vast, Portland, Lahey, HP, IBM are only supported + in the form of community feedback, and may not work out of the box. + GCC 4.x (and later) compilers are recommended. 3) Linear Algebra libraries @@ -123,8 +123,7 @@ The default order for the libraries are: 2. BLIS 3. OpenBLAS 4. ATLAS -5. Accelerate (MacOS) -6. BLAS (NetLIB) +5. BLAS (NetLIB) If you wish to build against OpenBLAS but you also have BLIS available one may predefine the order of searching via the environment variable @@ -146,8 +145,7 @@ The default order for the libraries are: 2. OpenBLAS 3. libFLAME 4. ATLAS -5. Accelerate (MacOS) -6. LAPACK (NetLIB) +5. LAPACK (NetLIB) If you wish to build against OpenBLAS but you also have MKL available one diff --git a/doc/source/user/c-info.how-to-extend.rst b/doc/source/user/c-info.how-to-extend.rst index 7aeed57cf..d75242092 100644 --- a/doc/source/user/c-info.how-to-extend.rst +++ b/doc/source/user/c-info.how-to-extend.rst @@ -163,7 +163,7 @@ ignored. The *args* argument contains all of the arguments passed in to the function as a tuple. You can do anything you want at this point, but usually the easiest way to manage the input arguments is to call :c:func:`PyArg_ParseTuple` (args, format_string, -addresses_to_C_variables...) or :c:func:`PyArg_UnpackTuple` (tuple, "name" , +addresses_to_C_variables...) or :c:func:`PyArg_UnpackTuple` (tuple, "name", min, max, ...). A good description of how to use the first function is contained in the Python C-API reference manual under section 5.5 (Parsing arguments and building values). You should pay particular diff --git a/doc/source/user/c-info.python-as-glue.rst b/doc/source/user/c-info.python-as-glue.rst index 7b9b096af..8643d0dd1 100644 --- a/doc/source/user/c-info.python-as-glue.rst +++ b/doc/source/user/c-info.python-as-glue.rst @@ -163,27 +163,29 @@ be imported from Python:: f2py -c -m add add.f This command leaves a file named add.{ext} in the current directory -(where {ext} is the appropriate extension for a python extension +(where {ext} is the appropriate extension for a Python extension module on your platform --- so, pyd, *etc.* ). This module may then be imported from Python. It will contain a method for each subroutine in add (zadd, cadd, dadd, sadd). The docstring of each method contains information about how the module method may be called:: >>> import add - >>> print add.zadd.__doc__ - zadd - Function signature: - zadd(a,b,c,n) - Required arguments: - a : input rank-1 array('D') with bounds (*) - b : input rank-1 array('D') with bounds (*) - c : input rank-1 array('D') with bounds (*) - n : input int + >>> print(add.zadd.__doc__) + zadd(a,b,c,n) + Wrapper for ``zadd``. + + Parameters + ---------- + a : input rank-1 array('D') with bounds (*) + b : input rank-1 array('D') with bounds (*) + c : input rank-1 array('D') with bounds (*) + n : input int Improving the basic interface ----------------------------- -The default interface is a very literal translation of the fortran +The default interface is a very literal translation of the Fortran code into Python. The Fortran array arguments must now be NumPy arrays and the integer argument should be an integer. The interface will attempt to convert all arguments to their required types (and shapes) @@ -192,7 +194,7 @@ about the semantics of the arguments (such that C is an output and n should really match the array sizes), it is possible to abuse this function in ways that can cause Python to crash. For example:: - >>> add.zadd([1,2,3], [1,2], [3,4], 1000) + >>> add.zadd([1, 2, 3], [1, 2], [3, 4], 1000) will cause a program crash on most systems. Under the covers, the lists are being converted to proper arrays but then the underlying add @@ -240,27 +242,32 @@ necessary to tell f2py that the value of n depends on the input a (so that it won't try to create the variable n until the variable a is created). -After modifying ``add.pyf``, the new python module file can be generated -by compiling both ``add.f95`` and ``add.pyf``:: +After modifying ``add.pyf``, the new Python module file can be generated +by compiling both ``add.f`` and ``add.pyf``:: - f2py -c add.pyf add.f95 + f2py -c add.pyf add.f The new interface has docstring:: >>> import add - >>> print add.zadd.__doc__ - zadd - Function signature: - c = zadd(a,b) - Required arguments: - a : input rank-1 array('D') with bounds (n) - b : input rank-1 array('D') with bounds (n) - Return objects: - c : rank-1 array('D') with bounds (n) + >>> print(add.zadd.__doc__) + c = zadd(a,b) + + Wrapper for ``zadd``. + + Parameters + ---------- + a : input rank-1 array('D') with bounds (n) + b : input rank-1 array('D') with bounds (n) + + Returns + ------- + c : rank-1 array('D') with bounds (n) Now, the function can be called in a much more robust way:: - >>> add.zadd([1,2,3],[4,5,6]) - array([ 5.+0.j, 7.+0.j, 9.+0.j]) + >>> add.zadd([1, 2, 3], [4, 5, 6]) + array([5.+0.j, 7.+0.j, 9.+0.j]) Notice the automatic conversion to the correct format that occurred. @@ -269,7 +276,7 @@ Inserting directives in Fortran source -------------------------------------- The nice interface can also be generated automatically by placing the -variable directives as special comments in the original fortran code. +variable directives as special comments in the original Fortran code. Thus, if I modify the source code to contain: .. code-block:: none @@ -655,7 +662,7 @@ To use ctypes you must 2. Load the shared library. -3. Convert the python objects to ctypes-understood arguments. +3. Convert the Python objects to ctypes-understood arguments. 4. Call the function from the library with the ctypes arguments. @@ -671,7 +678,7 @@ simply have a shared library available to you). Items to remember are: - A shared library must be compiled in a special way ( *e.g.* using the ``-shared`` flag with gcc). -- On some platforms (*e.g.* Windows) , a shared library requires a +- On some platforms (*e.g.* Windows), a shared library requires a .def file that specifies the functions to be exported. For example a mylib.def file might contain:: @@ -802,7 +809,7 @@ Calling the function The function is accessed as an attribute of or an item from the loaded shared-library. Thus, if ``./mylib.so`` has a function named -``cool_function1`` , I could access this function either as: +``cool_function1``, I could access this function either as: .. code-block:: python diff --git a/doc/source/user/explanations_index.rst b/doc/source/user/explanations_index.rst new file mode 100644 index 000000000..f515e36b1 --- /dev/null +++ b/doc/source/user/explanations_index.rst @@ -0,0 +1,14 @@ +.. _explanations: + +################ +Explanations +################ + +These documents are intended to explain in detail the concepts and techniques +used in NumPy. For the reference documentation of the functions and classes +contained in the package, see the :ref:`API reference <reference>`. + +.. toctree:: + :maxdepth: 1 + + basics.broadcasting diff --git a/doc/source/user/howtos_index.rst b/doc/source/user/howtos_index.rst new file mode 100644 index 000000000..c052286b9 --- /dev/null +++ b/doc/source/user/howtos_index.rst @@ -0,0 +1,14 @@ +.. _howtos: + +################ +NumPy How Tos +################ + +These documents are intended as recipes to common tasks using NumPy. For +detailed reference documentation of the functions and classes contained in +the package, see the :ref:`API reference <reference>`. + +.. toctree:: + :maxdepth: 1 + + ionumpy diff --git a/doc/source/user/index.rst b/doc/source/user/index.rst index d506a5a98..4e6a29d9f 100644 --- a/doc/source/user/index.rst +++ b/doc/source/user/index.rst @@ -1,3 +1,5 @@ +:orphan: + .. _user: ################ @@ -18,6 +20,7 @@ classes contained in the package, see the :ref:`reference`. basics misc numpy-for-matlab-users - tutorial-svd building c-info + tutorials_index + howtos_index diff --git a/doc/source/user/ionumpy.rst b/doc/source/user/ionumpy.rst new file mode 100644 index 000000000..a31720322 --- /dev/null +++ b/doc/source/user/ionumpy.rst @@ -0,0 +1,20 @@ +================================================ +How to read and write data using NumPy +================================================ + +.. currentmodule:: numpy + +.. testsetup:: + + import numpy as np + np.random.seed(1) + +**Objectives** + +- Writing NumPy arrays to files +- Reading NumPy arrays from files +- Dealing with encoding and dtype issues + +**Content** + +To be completed. diff --git a/doc/source/user/numpy-for-matlab-users.rst b/doc/source/user/numpy-for-matlab-users.rst index e53d1ca45..602192ecd 100644 --- a/doc/source/user/numpy-for-matlab-users.rst +++ b/doc/source/user/numpy-for-matlab-users.rst @@ -394,7 +394,8 @@ Linear Algebra Equivalents * - ``y=x(:)`` - ``y = x.flatten()`` - - turn array into vector (note that this forces a copy) + - turn array into vector (note that this forces a copy). To obtain the + same data ordering as in Matlab, use ``x.flatten('F')``. * - ``1:10`` - ``arange(1.,11.)`` or ``r_[1.:11.]`` or ``r_[1:10:10j]`` diff --git a/doc/source/user/quickstart.rst b/doc/source/user/quickstart.rst index 05a247f8a..29a4d80ca 100644 --- a/doc/source/user/quickstart.rst +++ b/doc/source/user/quickstart.rst @@ -8,7 +8,6 @@ Quickstart tutorial >>> import numpy as np >>> import sys - >>> rg = np.random.default_rng(1) Prerequisites ============= @@ -21,6 +20,24 @@ If you wish to work the examples in this tutorial, you must also have some software installed on your computer. Please see https://scipy.org/install.html for instructions. +**Learner profile** + +This tutorial is intended as a quick overview of +algebra and arrays in NumPy and want to understand how n-dimensional +(:math:`n>=2`) arrays are represented and can be manipulated. In particular, if +you don't know how to apply common functions to n-dimensional arrays (without +using for-loops), or if you want to understand axis and shape properties for +n-dimensional arrays, this tutorial might be of help. + +**Learning Objectives** + +After this tutorial, you should be able to: + +- Understand the difference between one-, two- and n-dimensional arrays in + NumPy; +- Understand how to apply some linear algebra operations to n-dimensional + arrays without using for-loops; +- Understand axis and shape properties for n-dimensional arrays. .. _quickstart.the-basics: @@ -129,8 +146,8 @@ rather than providing a single sequence as an argument. >>> a = np.array(1,2,3,4) # WRONG Traceback (most recent call last): - ... - ValueError: only 2 non-keyword arguments accepted + ... + TypeError: array() takes from 1 to 2 positional arguments but 4 were given >>> a = np.array([1,2,3,4]) # RIGHT ``array`` transforms sequences of sequences into two-dimensional arrays, @@ -330,6 +347,7 @@ existing array rather than create a new one. :: + >>> rg = np.random.default_rng(1) # create instance of default random number generator >>> a = np.ones((2,3), dtype=int) >>> b = rg.random((2,3)) >>> a *= 3 @@ -340,7 +358,7 @@ existing array rather than create a new one. >>> b array([[3.51182162, 3.9504637 , 3.14415961], [3.94864945, 3.31183145, 3.42332645]]) - >>> a += b # b is not automatically converted to integer type + >>> a += b # b is not automatically converted to integer type Traceback (most recent call last): ... numpy.core._exceptions.UFuncTypeError: Cannot cast ufunc 'add' output from dtype('float64') to dtype('int64') with casting rule 'same_kind' @@ -742,7 +760,7 @@ stacks 1D arrays as columns into a 2D array. It is equivalent to [2., 8.]]) >>> np.hstack((a,b)) # the result is different array([4., 2., 3., 8.]) - >>> a[:,newaxis] # this allows to have a 2D columns vector + >>> a[:,newaxis] # view `a` as a 2D column vector array([[4.], [2.]]) >>> np.column_stack((a[:,newaxis],b[:,newaxis])) diff --git a/doc/source/user/setting-up.rst b/doc/source/user/setting-up.rst index f70dacf82..7ca3a365c 100644 --- a/doc/source/user/setting-up.rst +++ b/doc/source/user/setting-up.rst @@ -7,3 +7,4 @@ Setting up whatisnumpy install + troubleshooting-importerror diff --git a/doc/source/user/troubleshooting-importerror.rst b/doc/source/user/troubleshooting-importerror.rst new file mode 100644 index 000000000..7d4846f77 --- /dev/null +++ b/doc/source/user/troubleshooting-importerror.rst @@ -0,0 +1,140 @@ +*************************** +Troubleshooting ImportError +*************************** + +.. note:: + + Since this information may be updated regularly, please ensure you are + viewing the most `up-to-date version <https://numpy.org/devdocs/user/troubleshooting-importerror.html>`_. + + +ImportError +=========== + +In certain cases a failed installation or setup issue can cause you to +see the following error message:: + + IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE! + + Importing the numpy c-extensions failed. This error can happen for + different reasons, often due to issues with your setup. + +The error also has additional information to help you troubleshoot: + +* Your Python version +* Your NumPy version + +Please check both of these carefully to see if they are what you expect. +You may need to check your ``PATH`` or ``PYTHONPATH`` environment variables +(see `Check Environment Variables`_ below). + +The following sections list commonly reported issues depending on your setup. +If you have an issue/solution that you think should appear please open a +NumPy issue so that it will be added. + +There are a few commonly reported issues depending on your system/setup. +If none of the following tips help you, please be sure to note the following: + +* how you installed Python +* how you installed NumPy +* your operating system +* whether or not you have multiple versions of Python installed +* if you built from source, your compiler versions and ideally a build log + +when investigating further and asking for support. + + +Using Python from ``conda`` (Anaconda) +-------------------------------------- + +Please make sure that you have activated your conda environment. +See also the `conda user-guide <https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#activating-an-environment>`_. +If you use an external editor/development environment it will have to be set +up correctly. See below for solutions for some common setups. + +Using PyCharm with Anaconda/conda Python +---------------------------------------- + +There are fairly common issues when using PyCharm together with Anaconda, +please see the `PyCharm support <https://www.jetbrains.com/help/pycharm/conda-support-creating-conda-virtual-environment.html>`_ + +Using VSCode with Anaconda/conda Python (or environments) +--------------------------------------------------------- + +A commonly reported issue is related to the environment activation within +VSCode. Please see the `VSCode support <https://code.visualstudio.com/docs/python/environments>`_ +for information on how to correctly set up VSCode with virtual environments +or conda. + +Using Eclipse/PyDev with Anaconda/conda Python (or environments) +---------------------------------------------------------------- + +Please see the +`Anaconda Documentation <https://docs.anaconda.com/anaconda/user-guide/tasks/integration/eclipse-pydev/>`_ +on how to properly configure Eclipse/PyDev to use Anaconda Python with specific +conda environments. + + +Raspberry Pi +------------ + +There are sometimes issues reported on Raspberry Pi setups when installing +using ``pip3 install`` (or ``pip`` install). These will typically mention:: + + libf77blas.so.3: cannot open shared object file: No such file or directory + + +The solution will be to either:: + + sudo apt-get install libatlas-base-dev + +to install the missing libraries expected by the self-compiled NumPy +(ATLAS is a possible provider of linear algebra). + +*Alternatively* use the NumPy provided by Raspbian. In which case run:: + + pip3 uninstall numpy # remove previously installed version + apt install python3-numpy + + +Debug build on Windows +---------------------- + +Rather than building your project in ``DEBUG`` mode on windows, try +building in ``RELEASE`` mode with debug symbols and no optimization. +Full ``DEBUG`` mode on windows changes the names of the DLLs python +expects to find, so if you wish to truly work in ``DEBUG`` mode you will +need to recompile the entire stack of python modules you work with +including NumPy + + +All Setups +---------- + +Occasionally there may be simple issues with old or bad installations +of NumPy. In this case you may just try to uninstall and reinstall NumPy. +Make sure that NumPy is not found after uninstalling. + + +Development Setup +----------------- + +If you are using a development setup, make sure to run ``git clean -xdf`` +to delete all files not under version control (be careful not to lose +any modifications you made, e.g. ``site.cfg``). +In many cases files from old builds may lead to incorrect builds. + + +Check Environment Variables +--------------------------- + +In general how to set and check your environment variables depends on +your system. If you can open a correct python shell, you can also run the +following in python:: + + import os + print("PYTHONPATH:", os.environ.get('PYTHONPATH')) + print("PATH:", os.environ.get('PATH')) + +This may mainly help you if you are not running the python and/or NumPy +version you are expecting to run. diff --git a/doc/source/user/tutorial-svd.rst b/doc/source/user/tutorial-svd.rst index e1918f394..086e0a6de 100644 --- a/doc/source/user/tutorial-svd.rst +++ b/doc/source/user/tutorial-svd.rst @@ -455,7 +455,7 @@ and :include-source: 0 should give you an image indistinguishable from the original one (although we -may introduce floating point errors for this reconstruction. In fact, +may introduce floating point errors for this reconstruction). In fact, you might see a warning message saying `"Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers)."` This is expected from the manipulation we just did on the original diff --git a/doc/source/user/tutorials_index.rst b/doc/source/user/tutorials_index.rst new file mode 100644 index 000000000..c5e859fad --- /dev/null +++ b/doc/source/user/tutorials_index.rst @@ -0,0 +1,19 @@ +.. _tutorials: + +################ +NumPy Tutorials +################ + +These documents are intended as an introductory overview of NumPy and its +features. For detailed reference documentation of the functions and +classes contained in the package, see the :ref:`API reference <reference>`. + +.. toctree:: + :maxdepth: 1 + + basics + misc + numpy-for-matlab-users + tutorial-svd + building + c-info diff --git a/doc/sphinxext b/doc/sphinxext -Subproject a482f66913c1079d7439770f0119b55376bb1b8 +Subproject b4c5fd17e2b85c2416a5e586933eee353b58bf7 |