diff options
Diffstat (limited to 'doc')
51 files changed, 1435 insertions, 444 deletions
diff --git a/doc/CAPI.rst.txt b/doc/CAPI.rst.txt index f38815e2a..ccee0fdb6 100644 --- a/doc/CAPI.rst.txt +++ b/doc/CAPI.rst.txt @@ -6,7 +6,7 @@ C-API for NumPy :Discussions to: `numpy-discussion@python.org`__ :Created: October 2005 -__ http://scipy.org/scipylib/mailing-lists.html +__ https://scipy.org/scipylib/mailing-lists.html The C API of NumPy is (mostly) backward compatible with Numeric. diff --git a/doc/HOWTO_RELEASE.rst.txt b/doc/HOWTO_RELEASE.rst.txt index 3ed15e99c..a6a8fe8ab 100644 --- a/doc/HOWTO_RELEASE.rst.txt +++ b/doc/HOWTO_RELEASE.rst.txt @@ -18,16 +18,16 @@ Source tree NumPy Docs ---------- * https://github.com/numpy/numpy/blob/master/doc/HOWTO_RELEASE.rst.txt -* http://projects.scipy.org/numpy/wiki/MicrosoftToolchainSupport +* http://projects.scipy.org/numpy/wiki/MicrosoftToolchainSupport (dead link) SciPy.org wiki -------------- -* http://www.scipy.org/Installing_SciPy and links on that page. -* http://new.scipy.org/building/windows.html +* https://www.scipy.org/Installing_SciPy and links on that page. +* http://new.scipy.org/building/windows.html (dead link) Doc wiki -------- -* http://docs.scipy.org/numpy/docs/numpy-docs/user/install.rst/ +* http://docs.scipy.org/numpy/docs/numpy-docs/user/install.rst/ (dead link) Release Scripts --------------- @@ -56,7 +56,7 @@ Windows We build 32- and 64-bit wheels for Python 2.7, 3.4, 3.5 on Windows. Windows XP, Vista, 7, 8 and 10 are supported. We build numpy using the MSVC compilers on Appveyor, but we are hoping to update to a `mingw-w64 toolchain -<http://mingwpy.github.io>`_. The Windows wheels use ATLAS for BLAS / LAPACK. +<https://mingwpy.github.io>`_. The Windows wheels use ATLAS for BLAS / LAPACK. Linux ----- @@ -101,7 +101,7 @@ Building source archives and wheels You will need write permission for numpy-wheels in order to trigger wheel builds. -* Python(s) from `python.org <http://python.org>`_ or linux distro. +* Python(s) from `python.org <https://python.org>`_ or linux distro. * cython * virtualenv (pip) * Paver (pip) @@ -131,7 +131,7 @@ Generating author/pr lists -------------------------- You will need an personal access token -`<https://help.github.com/articles/creating-an-access-token-for-command-line-use/>`_ +`<https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/>`_ so that scripts can access the github numpy repository * gitpython (pip) @@ -206,7 +206,7 @@ Do:: to check that the documentation is in a buildable state. See doc/HOWTO_BUILD_DOCS.rst.txt for more details and for how to update -http://docs.scipy.org. +https://docs.scipy.org. Check deprecations ------------------ @@ -322,7 +322,7 @@ message). Unfortunately the name of a tag can be changed without breaking the signature, the contents of the message cannot. See : https://github.com/scipy/scipy/issues/4919 for a discussion of signing -release tags, and http://keyring.debian.org/creating-key.html for instructions +release tags, and https://keyring.debian.org/creating-key.html for instructions on creating a GPG key if you do not have one. To make your key more readily identifiable as you, consider sending your key @@ -387,7 +387,7 @@ The tar-files and binary releases for distribution should be uploaded to SourceF together with the Release Notes and the Changelog. Uploading can be done through a web interface or, more efficiently, through scp/sftp/rsync as described in the SourceForge -`upload guide <https://sourceforge.net/apps/trac/sourceforge/wiki/Release%20files%20for%20download>`_. +`upload guide <https://sourceforge.net/apps/trac/sourceforge/wiki/Release%20files%20for%20download>`_ (dead link). For example:: scp <filename> <username>,numpy@frs.sourceforge.net:/home/frs/project/n/nu/numpy/NumPy/<releasedir>/ @@ -403,44 +403,18 @@ expecting a binary wheel. You can do this automatically using the ``wheel-uploader`` script from https://github.com/MacPython/terryfy. Here is the recommended incantation for -downloading all the Windows, Manylinux, OSX wheels and uploading to PyPI. +downloading all the Windows, Manylinux, OSX wheels and uploading to PyPI. :: -:: - - cd ~/wheelhouse # local directory to cache wheel downloads + NPY_WHLS=~/wheelhouse # local directory to cache wheel downloads CDN_URL=https://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a43.ssl.cf2.rackcdn.com - wheel-uploader -u $CDN_URL -w warehouse -v -s -t win numpy 1.11.1rc1 + wheel-uploader -u $CDN_URL -w $NPY_WHLS -v -s -t win numpy 1.11.1rc1 wheel-uploader -u $CDN_URL -w warehouse -v -s -t macosx numpy 1.11.1rc1 wheel-uploader -u $CDN_URL -w warehouse -v -s -t manylinux1 numpy 1.11.1rc1 The ``-v`` flag gives verbose feedback, ``-s`` causes the script to sign the -wheels with your GPG key before upload. ``-r warehouse`` causes the upload to -use the Warehouse PyPI server. This is a good idea because the Warehouse -server seems to be a lot more reliable in receiving automated wheel uploads. -For this flag to work, you will need a ``warehouse`` section in your -``~/.pypirc`` file, of form: - - [distutils] - index-servers = - pypi - warehouse - - [pypi] - username:your_user_name - password:your_password - - [warehouse] - repository: https://upload.pypi.io/legacy/ - username: your_user_name - password: your_password - - [server-login] - username:your_user_name - password:your_password - -Don't forget to upload the wheels before the source tarball, so there is no -period for which people switch from an expected binary install to a source -install from PyPI. +wheels with your GPG key before upload. Don't forget to upload the wheels +before the source tarball, so there is no period for which people switch from +an expected binary install to a source install from PyPI. There are two ways to update the source release on PyPI, the first one is:: @@ -472,7 +446,7 @@ repository. Update docs.scipy.org --------------------- -All documentation for a release can be updated on http://docs.scipy.org/ with: +All documentation for a release can be updated on https://docs.scipy.org/ with: make dist make upload USERNAME=<yourname> RELEASE=1.11.0 diff --git a/doc/Py3K.rst.txt b/doc/Py3K.rst.txt index 44111eeb5..f78b9e5db 100644 --- a/doc/Py3K.rst.txt +++ b/doc/Py3K.rst.txt @@ -22,8 +22,8 @@ Resources Information on porting to 3K: -- http://wiki.python.org/moin/cporting -- http://wiki.python.org/moin/PortingExtensionModulesToPy3k +- https://wiki.python.org/moin/cporting +- https://wiki.python.org/moin/PortingExtensionModulesToPy3k Prerequisites diff --git a/doc/RELEASE_WALKTHROUGH.rst.txt b/doc/RELEASE_WALKTHROUGH.rst.txt index ad14c16c1..260e0c1b7 100644 --- a/doc/RELEASE_WALKTHROUGH.rst.txt +++ b/doc/RELEASE_WALKTHROUGH.rst.txt @@ -1,23 +1,36 @@ -This file contains a walkthrough of the NumPy 1.14.4 release on Linux. +This file contains a walkthrough of the NumPy 1.14.5 release on Linux. The commands can be copied into the command line, but be sure to -replace 1.14.4 by the correct version. +replace 1.14.5 by the correct version. Release Walkthrough ==================== +Backport Pull Requests +---------------------- + +Changes that have been marked for this release must be backported to the +maintenance/1.14.x branch. + Update Release documentation ---------------------------- -The file ``doc/changelog/1.14.4-changelog.rst`` should be updated to reflect +The file ``doc/changelog/1.14.5-changelog.rst`` should be updated to reflect the final list of changes and contributors. This text can be generated by:: - $ python tools/changelog.py $GITHUB v1.14.3..maintenance/1.14.x > doc/changelog/1.14.4-changelog.rst + $ python tools/changelog.py $GITHUB v1.14.4..maintenance/1.14.x > doc/changelog/1.14.5-changelog.rst where ``GITHUB`` contains your github access token. This text may also be -appended to ``doc/release/1.14.4-notes.rst`` for release updates, though not +appended to ``doc/release/1.14.5-notes.rst`` for release updates, though not for new releases like ``1.14.0``, as the changelogs for latter tend to be excessively long. The ``doc/source/release.rst`` file should also be -updated with a link to the new release notes. +updated with a link to the new release notes. These changes should be committed +to the maintenance branch, and later will be forward ported to master. + +Finish the Release Note +----------------------- + +Fill out the release note ``doc/release/1.14.5-notes.rst`` calling out +significant changes. Prepare the release commit -------------------------- @@ -33,7 +46,7 @@ repository:: Edit pavement.py and setup.py as detailed in HOWTO_RELEASE:: $ gvim pavement.py setup.py - $ git commit -a -m"REL: NumPy 1.14.4 release." + $ git commit -a -m"REL: NumPy 1.14.5 release." Sanity check:: @@ -52,9 +65,10 @@ Build source releases Paver is used to build the source releases. It will create the ``release`` and ``release/installers`` directories and put the ``*.zip`` and ``*.tar.gz`` -source releases in the latter. +source releases in the latter. :: - $ paver sdist # sdist will do a git clean -xdf, so we omit that + $ cython --version # check that you have the correct cython version + $ paver sdist # sdist will do a git clean -xdf, so we omit that Build wheels ------------ @@ -102,9 +116,9 @@ upload later using ``twine``:: $ git pull origin master $ CDN_URL=https://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a43.ssl.cf2.rackcdn.com $ NPY_WHLS=../numpy/release/installers - $ ./wheel-uploader -u $CDN_URL -n -v -w $NPY_WHLS -t win numpy 1.14.4 - $ ./wheel-uploader -u $CDN_URL -n -v -w $NPY_WHLS -t manylinux1 numpy 1.14.4 - $ ./wheel-uploader -u $CDN_URL -n -v -w $NPY_WHLS -t macosx numpy 1.14.4 + $ ./wheel-uploader -u $CDN_URL -n -v -w $NPY_WHLS -t win numpy 1.14.5 + $ ./wheel-uploader -u $CDN_URL -n -v -w $NPY_WHLS -t manylinux1 numpy 1.14.5 + $ ./wheel-uploader -u $CDN_URL -n -v -w $NPY_WHLS -t macosx numpy 1.14.5 If you do this often, consider making CDN_URL and NPY_WHLS part of your default environment. Note that we need local copies of the files in order to generate @@ -117,7 +131,7 @@ Once the wheels have been built and downloaded without errors, go back to your numpy repository in the maintenance branch and tag the ``REL`` commit, signing it with your gpg key, and build the source distribution archives:: - $ git tag -s v1.14.4 + $ git tag -s v1.14.5 You should upload your public gpg key to github, so that the tag will appear "verified" there. @@ -125,7 +139,7 @@ You should upload your public gpg key to github, so that the tag will appear Check that the files in ``release/installers`` have the correct versions, then push the tag upstream:: - $ git push upstream v1.14.4 + $ git push upstream v1.14.5 We wait until this point to push the tag because it is very difficult to change the tag after it has been pushed. @@ -140,9 +154,6 @@ Add another ``REL`` commit to the numpy maintenance branch, which resets the $ git commit -a -m"REL: prepare 1.14.x for further development" $ git push upstream maintenance/1.14.x -This strategy is copied from the scipy release procedure and was used in numpy -for the first time in 1.14.3. It needed to be modified a little since numpy -has more strict requirements for the version number. Upload to PyPI -------------- @@ -152,7 +163,7 @@ after recent PyPI changes, version ``1.11.0`` was used here. :: $ cd ../numpy $ twine upload release/installers/*.whl - $ twine upload release/installers/numpy-1.14.4.zip # Upload last. + $ twine upload release/installers/numpy-1.14.5.zip # Upload last. If one of the commands breaks in the middle, which is not uncommon, you may need to selectively upload the remaining files because PyPI does not allow the @@ -168,15 +179,15 @@ Generate the ``release/README.*`` files:: $ paver write_release_and_log -Go to `<https://github.com/numpy/numpy/releases>`_, there should be a ``v1.14.4 +Go to `<https://github.com/numpy/numpy/releases>`_, there should be a ``v1.14.5 tag``, click on it and hit the edit button for that tag. There are two ways to add files, using an editable text window and as binary uploads. - Cut and paste the ``release/README.md`` file contents into the text window. -- Upload ``release/installers/numpy-1.14.4.tar.gz`` as a binary file. -- Upload ``release/installers/numpy-1.14.4.zip`` as a binary file. +- Upload ``release/installers/numpy-1.14.5.tar.gz`` as a binary file. +- Upload ``release/installers/numpy-1.14.5.zip`` as a binary file. - Upload ``release/README.rst`` as a binary file. -- Upload ``doc/changelog/1.14.4-changelog.rst`` as a binary file. +- Upload ``doc/changelog/1.14.5-changelog.rst`` as a binary file. - Check the pre-release button if this is a pre-releases. - Hit the ``{Publish,Update} release`` button at the bottom. @@ -191,7 +202,7 @@ upload the documentation. Otherwise:: $ pushd doc $ make dist - $ make upload USERNAME=<yourname> RELEASE=v1.14.4 + $ make upload USERNAME=<yourname> RELEASE=v1.14.5 $ popd If the release series is a new one, you will need to rebuild and upload the @@ -212,7 +223,7 @@ This assumes that you have forked `<https://github.com/scipy/scipy.org>`_:: $ cd ../scipy.org $ git checkout master $ git pull upstream master - $ git checkout -b numpy-1.14.4 + $ git checkout -b numpy-1.14.5 $ gvim www/index.rst # edit the News section $ git commit -a $ git push origin HEAD @@ -226,3 +237,10 @@ The release should be announced on the numpy-discussion, scipy-devel, scipy-user, and python-announce-list mailing lists. Look at previous announcements for the basic template. The contributor and PR lists are the same as generated for the release notes above. + +Post-Release Tasks +------------------ + +Forward port the documentation changes ``doc/release/1.14.5-notes.rst``, +``doc/changelog/1.14.5-changelog.rst`` and add the release note to +``doc/source/release.rst``. diff --git a/doc/TESTS.rst.txt b/doc/TESTS.rst.txt index 68b0eace4..170519110 100644 --- a/doc/TESTS.rst.txt +++ b/doc/TESTS.rst.txt @@ -9,17 +9,17 @@ Introduction '''''''''''' SciPy uses the `Nose testing system -<http://nose.readthedocs.io>`__, with some +<https://nose.readthedocs.io>`__, with some minor convenience features added. Nose is an extension of the unit testing framework offered by `unittest.py -<http://docs.python.org/lib/module-unittest.html>`__. Our goal is that +<https://docs.python.org/library/unittest.html>`__. Our goal is that every module and package in SciPy should have a thorough set of unit tests. These tests should exercise the full functionality of a given routine as well as its robustness to erroneous or unexpected input arguments. Long experience has shown that by far the best time to write the tests is before you write or change the code - this is `test-driven development -<http://en.wikipedia.org/wiki/Test-driven_development>`__. The +<https://en.wikipedia.org/wiki/Test-driven_development>`__. The arguments for this can sound rather abstract, but we can assure you that you will find that writing the tests first leads to more robust and better designed code. Well-designed tests with good coverage make @@ -306,7 +306,7 @@ minor variations, it can be helpful to create a base class containing all the common tests, and then create a subclass for each variation. Several examples of this technique exist in NumPy; below are excerpts from one in `numpy/linalg/tests/test_linalg.py -<http://github.com/numpy/numpy/blob/master/numpy/linalg/tests/test_linalg.py>`__:: +<https://github.com/numpy/numpy/blob/master/numpy/linalg/tests/test_linalg.py>`__:: class LinalgTestCase: def test_single(self): diff --git a/doc/cdoc/Doxyfile b/doc/cdoc/Doxyfile index d80e98558..886a3440e 100644 --- a/doc/cdoc/Doxyfile +++ b/doc/cdoc/Doxyfile @@ -18,7 +18,7 @@ # that follow. The default is UTF-8 which is also the encoding used for all # text before the first occurrence of this tag. Doxygen uses libiconv (or the # iconv built into libc) for the transcoding. See -# http://www.gnu.org/software/libiconv for the list of possible encodings. +# https://www.gnu.org/software/libiconv for the list of possible encodings. DOXYFILE_ENCODING = UTF-8 @@ -596,7 +596,7 @@ INPUT = ../../numpy/core/src \ # This tag can be used to specify the character encoding of the source files # that doxygen parses. Internally doxygen uses the UTF-8 encoding, which is # also the default input encoding. Doxygen uses libiconv (or the iconv built -# into libc) for the transcoding. See http://www.gnu.org/software/libiconv for +# into libc) for the transcoding. See https://www.gnu.org/software/libiconv for # the list of possible encodings. INPUT_ENCODING = UTF-8 @@ -739,7 +739,7 @@ REFERENCES_LINK_SOURCE = YES # If the USE_HTAGS tag is set to YES then the references to source code # will point to the HTML generated by the htags(1) tool instead of doxygen # built-in source browser. The htags tool is part of GNU's global source -# tagging system (see http://www.gnu.org/software/global/global.html). You +# tagging system (see https://www.gnu.org/software/global/global.html). You # will need version 4.8.6 or higher. USE_HTAGS = NO @@ -843,7 +843,8 @@ HTML_DYNAMIC_SECTIONS = NO # directory and running "make install" will install the docset in # ~/Library/Developer/Shared/Documentation/DocSets so that Xcode will find # it at startup. -# See http://developer.apple.com/tools/creatingdocsetswithdoxygen.html for more information. +# See http://developer.apple.com/tools/creatingdocsetswithdoxygen.html (dead link) +# for more information. GENERATE_DOCSET = NO @@ -920,30 +921,30 @@ QCH_FILE = # The QHP_NAMESPACE tag specifies the namespace to use when generating # Qt Help Project output. For more information please see -# http://doc.trolltech.com/qthelpproject.html#namespace +# https://doc.qt.io/qt-5/qthelpproject.html#namespace QHP_NAMESPACE = org.doxygen.Project # The QHP_VIRTUAL_FOLDER tag specifies the namespace to use when generating # Qt Help Project output. For more information please see -# http://doc.trolltech.com/qthelpproject.html#virtual-folders +# https://doc.qt.io/qt-5/qthelpproject.html#virtual-folders QHP_VIRTUAL_FOLDER = doc # If QHP_CUST_FILTER_NAME is set, it specifies the name of a custom filter to add. # For more information please see -# http://doc.trolltech.com/qthelpproject.html#custom-filters +# https://doc.qt.io/qt-5/qthelpproject.html#custom-filters QHP_CUST_FILTER_NAME = # The QHP_CUST_FILT_ATTRS tag specifies the list of the attributes of the custom filter to add.For more information please see -# <a href="http://doc.trolltech.com/qthelpproject.html#custom-filters">Qt Help Project / Custom Filters</a>. +# <a href="https://doc.qt.io/qt-5/qthelpproject.html#custom-filters">Qt Help Project / Custom Filters</a>. QHP_CUST_FILTER_ATTRS = # The QHP_SECT_FILTER_ATTRS tag specifies the list of the attributes this project's # filter section matches. -# <a href="http://doc.trolltech.com/qthelpproject.html#filter-attributes">Qt Help Project / Filter Attributes</a>. +# <a href="https://doc.qt.io/qt-5/qthelpproject.html#filter-attributes">Qt Help Project / Filter Attributes</a>. QHP_SECT_FILTER_ATTRS = diff --git a/doc/neps/conf.py b/doc/neps/conf.py index aa11d37b3..8cfb2b570 100644 --- a/doc/neps/conf.py +++ b/doc/neps/conf.py @@ -100,7 +100,7 @@ todo_include_todos = False ## to template names. ## ## This is required for the alabaster theme -## refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars +## refs: https://alabaster.readthedocs.io/en/latest/installation.html#sidebars #html_sidebars = { # '**': [ # 'relations.html', # needs 'show_related': True theme option to display @@ -127,8 +127,8 @@ if True: "edit_link": True, "sidebar": "right", "scipy_org_logo": True, - "rootlinks": [("http://scipy.org/", "Scipy.org"), - ("http://docs.scipy.org/", "Docs")] + "rootlinks": [("https://scipy.org/", "Scipy.org"), + ("https://docs.scipy.org/", "Docs")] } else: # Default build diff --git a/doc/neps/nep-0000.rst b/doc/neps/nep-0000.rst index 9c6646db2..0824a5635 100644 --- a/doc/neps/nep-0000.rst +++ b/doc/neps/nep-0000.rst @@ -203,7 +203,7 @@ References and Footnotes `GitHub <https://github.com/numpy/numpy/tree/master/doc/neps>`_. .. [2] The URL for viewing NEPs on the web is - http://numpy.github.io/neps/. + https://www.numpy.org/neps/. .. _repo: https://github.com/numpy/numpy @@ -220,7 +220,7 @@ References and Footnotes .. _reStructuredTextPrimer: http://www.sphinx-doc.org/en/stable/rest.html -.. _Sphinx: www.sphinx-doc.org/en/stable +.. _Sphinx: http://www.sphinx-doc.org/en/stable/ Copyright diff --git a/doc/neps/nep-0001-npy-format.rst b/doc/neps/nep-0001-npy-format.rst index 2057aed83..74512128d 100644 --- a/doc/neps/nep-0001-npy-format.rst +++ b/doc/neps/nep-0001-npy-format.rst @@ -290,15 +290,15 @@ included in the 1.9.0 release of numpy. Specifically, the file format.py in this directory implements the format as described here. - http://github.com/numpy/numpy/blob/master/numpy/lib/format.py + https://github.com/numpy/numpy/blob/master/numpy/lib/format.py References ---------- -[1] http://docs.python.org/lib/module-pickle.html +[1] https://docs.python.org/library/pickle.html -[2] http://hdf.ncsa.uiuc.edu/products/hdf5/index.html +[2] https://support.hdfgroup.org/HDF5/ Copyright diff --git a/doc/neps/nep-0004-datetime-proposal3.rst b/doc/neps/nep-0004-datetime-proposal3.rst index 46d8e314b..afeb00d73 100644 --- a/doc/neps/nep-0004-datetime-proposal3.rst +++ b/doc/neps/nep-0004-datetime-proposal3.rst @@ -562,9 +562,9 @@ examples of other derived units, and we find this a bit too overwhelming for this proposal purposes. -.. [1] http://docs.python.org/lib/module-datetime.html -.. [2] http://www.egenix.com/products/python/mxBase/mxDateTime -.. [3] http://en.wikipedia.org/wiki/Unix_time +.. [1] https://docs.python.org/library/datetime.html +.. [2] https://www.egenix.com/products/python/mxBase/mxDateTime +.. [3] https://en.wikipedia.org/wiki/Unix_time .. Local Variables: diff --git a/doc/neps/nep-0007-datetime-proposal.rst b/doc/neps/nep-0007-datetime-proposal.rst index 72d48d244..90894da49 100644 --- a/doc/neps/nep-0007-datetime-proposal.rst +++ b/doc/neps/nep-0007-datetime-proposal.rst @@ -662,9 +662,9 @@ operations mixing business days with other time units will not be allowed. -.. [1] http://docs.python.org/lib/module-datetime.html -.. [2] http://www.egenix.com/products/python/mxBase/mxDateTime -.. [3] http://en.wikipedia.org/wiki/Unix_time +.. [1] https://docs.python.org/library/datetime.html +.. [2] https://www.egenix.com/products/python/mxBase/mxDateTime +.. [3] https://en.wikipedia.org/wiki/Unix_time .. Local Variables: diff --git a/doc/neps/nep-0012-missing-data.rst b/doc/neps/nep-0012-missing-data.rst index 1553339f4..57c45b4b6 100644 --- a/doc/neps/nep-0012-missing-data.rst +++ b/doc/neps/nep-0012-missing-data.rst @@ -4,7 +4,7 @@ Missing Data Functionality in NumPy :Author: Mark Wiebe <mwwiebe@gmail.com> :Copyright: Copyright 2011 by Enthought, Inc -:License: CC By-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0/) +:License: CC By-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0/) :Date: 2011-06-23 :Status: Deferred @@ -224,7 +224,7 @@ but with semantics reflecting its status as a missing value. In particular, trying to treat it as a boolean will raise an exception, and comparisons with it will produce numpy.NA instead of True or False. These basics are adopted from the behavior of the NA value in the R project. To dig -deeper into the ideas, http://en.wikipedia.org/wiki/Ternary_logic#Kleene_logic +deeper into the ideas, https://en.wikipedia.org/wiki/Ternary_logic#Kleene_logic provides a starting point. For example,:: @@ -857,7 +857,7 @@ Shared Masks One feature of numpy.ma is called 'shared masks'. -http://docs.scipy.org/doc/numpy/reference/maskedarray.baseclass.html#numpy.ma.MaskedArray.sharedmask +https://docs.scipy.org/doc/numpy/reference/maskedarray.baseclass.html#numpy.ma.MaskedArray.sharedmask This feature cannot be supported by a masked implementation of missing values without directly violating the missing value abstraction. @@ -888,7 +888,7 @@ found from doing google searches of numpy C API array access. NumPy Documentation - How to extend NumPy ----------------------------------------- -http://docs.scipy.org/doc/numpy/user/c-info.how-to-extend.html#dealing-with-array-objects +https://docs.scipy.org/doc/numpy/user/c-info.how-to-extend.html#dealing-with-array-objects This page has a section "Dealing with array objects" which has some advice for how to access numpy arrays from C. When accepting arrays, the first step it suggests is @@ -898,7 +898,7 @@ advice will properly fail when given an NA-masked array it doesn't know how to h The way this is handled is that PyArray_FromAny requires a special flag, NPY_ARRAY_ALLOWNA, before it will allow NA-masked arrays to flow through. -http://docs.scipy.org/doc/numpy/reference/c-api.array.html#NPY_ARRAY_ALLOWNA +https://docs.scipy.org/doc/numpy/reference/c-api.array.html#NPY_ARRAY_ALLOWNA Code which does not follow this advice, and instead just calls PyArray_Check() to verify its an ndarray and checks some flags, will silently produce incorrect results. This style diff --git a/doc/neps/nep-0013-ufunc-overrides.rst b/doc/neps/nep-0013-ufunc-overrides.rst index c97b69023..61e2ceea9 100644 --- a/doc/neps/nep-0013-ufunc-overrides.rst +++ b/doc/neps/nep-0013-ufunc-overrides.rst @@ -53,7 +53,7 @@ changes in 3rd party code. .. [1] http://docs.python.org/doc/numpy/user/basics.subclassing.html .. [2] https://github.com/scipy/scipy/issues/2123 .. [3] https://github.com/scipy/scipy/issues/1569 -.. [4] http://technicaldiscovery.blogspot.com/2013/07/thoughts-after-scipy-2013-and-specific.html +.. [4] https://technicaldiscovery.blogspot.com/2013/07/thoughts-after-scipy-2013-and-specific.html Motivation @@ -134,7 +134,7 @@ which have multiplication semantics incompatible with numpy arrays. However, the aim is to enable writing other custom array types that have strictly ndarray compatible semantics. -.. [5] http://mail.python.org/pipermail/numpy-discussion/2011-June/056945.html +.. [5] https://mail.python.org/pipermail/numpy-discussion/2011-June/056945.html .. [6] https://github.com/numpy/numpy/issues/5844 @@ -635,7 +635,7 @@ simplify the dispatch logic for binary operations with NumPy arrays as much as possible, by making it possible to use Python's dispatch rules or NumPy's dispatch rules, but not some mixture of both at the same time. -.. [9] http://bugs.python.org/issue30140 +.. [9] https://bugs.python.org/issue30140 .. _neps.ufunc-overrides.list-of-operators: diff --git a/doc/neps/nep-0014-dropping-python2.7-proposal.rst b/doc/neps/nep-0014-dropping-python2.7-proposal.rst index 6cfd4707f..158b89e1c 100644 --- a/doc/neps/nep-0014-dropping-python2.7-proposal.rst +++ b/doc/neps/nep-0014-dropping-python2.7-proposal.rst @@ -50,6 +50,6 @@ to Python3 only, see the python3-statement_. For more information on porting your code to run on Python 3, see the python3-howto_. -.. _python3-statement: http://www.python3statement.org/ +.. _python3-statement: https://python3statement.org/ .. _python3-howto: https://docs.python.org/3/howto/pyporting.html diff --git a/doc/neps/nep-0018-array-function-protocol.rst b/doc/neps/nep-0018-array-function-protocol.rst index f42da2f88..58a2833e6 100644 --- a/doc/neps/nep-0018-array-function-protocol.rst +++ b/doc/neps/nep-0018-array-function-protocol.rst @@ -1,9 +1,12 @@ -============================================= -Dispatch Mechanism for NumPy's high level API -============================================= +=========================================================== +A dispatch mechanism for NumPy's high level array functions +=========================================================== :Author: Stephan Hoyer <shoyer@google.com> :Author: Matthew Rocklin <mrocklin@gmail.com> +:Author: Marten van Kerkwijk <mhvk@astro.utoronto.ca> +:Author: Hameer Abbasi <hameerabbasi@yahoo.com> +:Author: Eric Wieser <wieser.eric@gmail.com> :Status: Draft :Type: Standards Track :Created: 2018-05-29 @@ -11,7 +14,7 @@ Dispatch Mechanism for NumPy's high level API Abstact ------- -We propose the ``__array_function__`` protocol, to allow arguments of numpy +We propose the ``__array_function__`` protocol, to allow arguments of NumPy functions to define how that function operates on them. This will allow using NumPy as a high level API for efficient multi-dimensional array operations, even with array implementations that differ greatly from @@ -27,7 +30,7 @@ arrays (Dask array) as well as various NumPy-like implementations in the deep learning frameworks, like TensorFlow and PyTorch. Similarly there are many projects that build on top of the NumPy API -for labeled and indexed arrays (XArray), automatic differentation +for labeled and indexed arrays (XArray), automatic differentiation (Autograd, Tangent), masked arrays (numpy.ma), physical units (astropy.units, pint, unyt), etc. that add additional functionality on top of the NumPy API. Most of these project also implement a close variation of NumPy's level high @@ -78,7 +81,7 @@ is not covered by the ``__array_ufunc__`` protocol for universal functions the operation is specified by an arbitrary callable object rather than a ufunc instance and method. -A prototype implementation with microbenchmark results can be found in +A prototype implementation can be found in `this notebook <https://nbviewer.jupyter.org/gist/shoyer/1f0a308a06cd96df20879a1ddb8f0006>`_. The interface @@ -93,24 +96,23 @@ We propose the following signature for implementations of - ``func`` is an arbitrary callable exposed by NumPy's public API, which was called in the form ``func(*args, **kwargs)``. -- ``types`` is a list of argument types from the original NumPy - function call that implement ``__array_function__``, in the order in which - they will be called. +- ``types`` is a ``frozenset`` of unique argument types from the original NumPy + function call that implement ``__array_function__``. - The tuple ``args`` and dict ``kwargs`` are directly passed on from the original call. Unlike ``__array_ufunc__``, there are no high-level guarantees about the type of ``func``, or about which of ``args`` and ``kwargs`` may contain objects -implementing the array API. As a convenience for ``__array_function__`` -implementors, ``types`` contains a list of argument types with an -``'__array_function__'`` attribute. This allows downstream implementations to -quickly determine if they are likely able to support the operation. +implementing the array API. -Still be determined: what guarantees can we offer for ``types``? Should -we promise that types are unique, and appear in the order in which they -are checked? Should we pass in arguments directly instead, either the full -list of arguments in ``relevant_arguments`` (see below) or a single argument -for each unique type? +As a convenience for ``__array_function__`` implementors, ``types`` provides all +argument types with an ``'__array_function__'`` attribute. This +allows downstream implementations to quickly determine if they are likely able +to support the operation. A ``frozenset`` is used to ensure that +``__array_function__`` implementations cannot rely on the iteration order of +``types``, which would facilitate violating the well-defined "Type casting +hierarchy" described in +`NEP-13 <https://www.numpy.org/neps/nep-0013-ufunc-overrides.html>`_. Example for a project implementing the NumPy API ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -124,24 +126,55 @@ checks: If these conditions hold, ``__array_function__`` should return the result from calling its implementation for ``func(*args, **kwargs)``. Otherwise, it should return the sentinel value ``NotImplemented``, indicating -that the function is not implemented by these types. +that the function is not implemented by these types. This is preferable to +raising ``TypeError`` directly, because it gives *other* arguments the +opportunity to define the operations. + +There are no general requirements on the return value from +``__array_function__``, although most sensible implementations should probably +return array(s) with the same type as one of the function's arguments. +If/when Python gains +`typing support for protocols <https://www.python.org/dev/peps/pep-0544/>`_ +and NumPy adds static type annotations, the ``@overload`` implementation +for ``SupportsArrayFunction`` will indicate a return type of ``Any``. + +It may also be convenient to define a custom decorators (``implements`` below) +for registering ``__array_function__`` implementations. .. code:: python + HANDLED_FUNCTIONS = {} + class MyArray: def __array_function__(self, func, types, args, kwargs): if func not in HANDLED_FUNCTIONS: return NotImplemented + # Note: this allows subclasses that don't override + # __array_function__ to handle MyArray objects if not all(issubclass(t, MyArray) for t in types): return NotImplemented return HANDLED_FUNCTIONS[func](*args, **kwargs) - HANDLED_FUNCTIONS = { - np.concatenate: my_concatenate, - np.broadcast_to: my_broadcast_to, - np.sum: my_sum, - ... - } + def implements(numpy_function): + """Register an __array_function__ implementation for MyArray objects.""" + def decorator(func): + HANDLED_FUNCTIONS[numpy_function] = func + return func + return decorator + + @implements(np.concatenate) + def concatenate(arrays, axis=0, out=None): + ... # implementation of concatenate for MyArray objects + + @implements(np.broadcast_to) + def broadcast_to(array, shape): + ... # implementation of broadcast_to for MyArray objects + +Note that it is not required for ``__array_function__`` implementations to +include *all* of the corresponding NumPy function's optional arguments +(e.g., ``broadcast_to`` above omits the irrelevant ``subok`` argument). +Optional arguments are only passed in to ``__array_function__`` if they +were explicitly used in the NumPy function call. Necessary changes within the NumPy codebase itself ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -179,13 +212,46 @@ be nested within lists or dictionaries, such as in the case of ``np.concatenate([x, y, z])``. This can be problematic for two reasons: 1. Some functions are given long lists of values, and traversing them - might be prohibitively expensive -2. Some function may have arguments that we don't want to inspect, even - if they have the ``__array_function__`` method + might be prohibitively expensive. +2. Some functions may have arguments that we don't want to inspect, even + if they have the ``__array_function__`` method. + +To resolve these issues, NumPy functions should explicitly indicate which +of their arguments may be overloaded, and how these arguments should be +checked. As a rule, this should include all arguments documented as either +``array_like`` or ``ndarray``. + +We propose to do so by writing "dispatcher" functions for each overloaded +NumPy function: + +- These functions will be called with the exact same arguments that were passed + into the NumPy function (i.e., ``dispatcher(*args, **kwargs)``), and should + return an iterable of arguments to check for overrides. +- Dispatcher functions are required to share the exact same positional, + optional and keyword-only arguments as their corresponding NumPy functions. + Otherwise, valid invocations of a NumPy function could result in an error when + calling its dispatcher. +- Because default *values* for keyword arguments do not have + ``__array_function__`` attributes, by convention we set all default argument + values to ``None``. This reduces the likelihood of signatures falling out + of sync, and minimizes extraneous information in the dispatcher. + The only exception should be cases where the argument value in some way + effects dispatching, which should be rare. + +An example of the dispatcher for ``np.concatenate`` may be instructive: -To resolve these we ask the functions to provide an explicit list of -arguments that should be traversed. This is the ``relevant_arguments=`` -keyword in the examples below. +.. code:: python + + def _concatenate_dispatcher(arrays, axis=None, out=None): + for array in arrays: + yield array + if out is not None: + yield out + +The concatenate dispatcher is written as generator function, which allows it +to potentially include the value of the optional ``out`` argument without +needing to create a new sequence with the (potentially long) list of objects +to be concatenated. Trying ``__array_function__`` methods until the right one works ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' @@ -197,7 +263,7 @@ are valid then which has precedence? For the most part, the rules for dispatch with ``__array_function__`` match those for ``__array_ufunc__`` (see -`NEP-13 <http://www.numpy.org/neps/nep-0013-ufunc-overrides.html>`_). +`NEP-13 <https://www.numpy.org/neps/nep-0013-ufunc-overrides.html>`_). In particular: - NumPy will gather implementations of ``__array_function__`` from all @@ -250,93 +316,159 @@ an ``__array_function__`` method that is identical to Changes within NumPy functions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. warning:: - - This section is outdated. We intend to rewrite it to propose - an explicit `decorator based solution <https://github.com/numpy/numpy/pull/11303#issuecomment-396695348>`_ instead. - Given a function defining the above behavior, for now call it ``try_array_function_override``, we now need to call that function from within every relevant NumPy function. This is a pervasive change, but of fairly simple and innocuous code that should complete quickly and without effect if no arguments implement the ``__array_function__`` -protocol. Let us consider a few examples of NumPy functions and how they -might be affected by this change: +protocol. -.. code:: python +In most cases, these functions should written using the +``array_function_dispatch`` decorator, which also associates dispatcher +functions: - import itertools +.. code:: python + def array_function_dispatch(dispatcher): + """Wrap a function for dispatch with the __array_function__ protocol.""" + def decorator(func): + @functools.wraps(func) + def new_func(*args, **kwargs): + relevant_arguments = dispatcher(*args, **kwargs) + success, value = try_array_function_override( + new_func, relevant_arguments, args, kwargs) + if success: + return value + return func(*args, **kwargs) + return new_func + return decorator + + # example usage + def _broadcast_to_dispatcher(array, shape, subok=None, **ignored_kwargs): + return (array,) + + @array_function_dispatch(_broadcast_to_dispatcher) def broadcast_to(array, shape, subok=False): - success, value = try_array_function_override( - func=broadcast_to, - relevant_arguments=[array], - args=(array,), - kwargs=dict(shape=shape, subok=subok)) - if success: - return value - - ... # continue with the definition of broadcast_to - - def concatenate(arrays, axis=0, out=None) - success, value = try_array_function_override( - func=concatenate, - relevant_arguments=itertools.chain(arrays, [out]), - args=(arrays,), - kwargs=dict(axis=axis, out=out)) - if success: - return value - - ... # continue with the definition of concatenate - -The list of objects passed to ``relevant_arguments`` are those that should -be inspected for ``__array_function__`` implementations. - -Our microbenchmark results show that a pure Python implementation of -``try_array_function_override`` adds approximately 2-4 microseconds of -overhead to each function call without any overloaded arguments. -This is acceptable for functions implemented in Python but probably too -slow for functions written in C. Fortunately, we expect significantly less -overhead with a C implementation of ``try_array_function_override``. - -A more succinct alternative would be to write these overloads with a decorator -that builds overloaded functions automatically. Hypothetically, this might even -directly parse Python 3 type annotations, e.g., perhaps + ... # existing definition of np.broadcast_to + +Using a decorator is great! We don't need to change the definitions of +existing NumPy functions, and only need to write a few additional lines +for the dispatcher function. We could even reuse a single dispatcher for +families of functions with the same signature (e.g., ``sum`` and ``prod``). +For such functions, the largest change could be adding a few lines to the +docstring to note which arguments are checked for overloads. + +It's particularly worth calling out the decorator's use of +``functools.wraps``: + +- This ensures that the wrapped function has the same name and docstring as + the wrapped NumPy function. +- On Python 3, it also ensures that the decorator function copies the original + function signature, which is important for introspection based tools such as + auto-complete. If we care about preserving function signatures on Python 2, + for the `short while longer <http://www.numpy.org/neps/nep-0014-dropping-python2.7-proposal.html>`_ + that NumPy supports Python 2.7, we do could do so by adding a vendored + dependency on the (single-file, BSD licensed) + `decorator library <https://github.com/micheles/decorator>`_. +- Finally, it ensures that the wrapped function + `can be pickled <http://gael-varoquaux.info/programming/decoration-in-python-done-right-decorating-and-pickling.html>`_. + +In a few cases, it would not make sense to use the ``array_function_dispatch`` +decorator directly, but override implementation in terms of +``try_array_function_override`` should still be straightforward. + +- Functions written entirely in C (e.g., ``np.concatenate``) can't use + decorators, but they could still use a C equivalent of + ``try_array_function_override``. If performance is not a concern, they could + also be easily wrapped with a small Python wrapper. +- The ``__call__`` method of ``np.vectorize`` can't be decorated with + ``@array_function_dispatch``, because we should pass a ``vectorize`` object + itself as the ``func`` argument ``__array_function__``, not the unbound + ``vectorize.__call__`` method. +- ``np.einsum`` does complicated argument parsing to handle two different + function signatures. It would probably be best to avoid the overhead of + parsing it twice in the typical case of no overrides. + +Fortunately, in each of these cases so far, the functions already has a generic +signature of the form ``*args, **kwargs``, which means we don't need to worry +about potential inconsistency between how functions are called and what we pass +to ``__array_function__``. (In C, arguments for all Python functions are parsed +from a tuple ``*args`` and dict ``**kwargs``.) This shouldn't stop us from +writing overrides for functions with non-generic signatures that can't use the +decorator, but we should consider these cases carefully. + +Extensibility +~~~~~~~~~~~~~ + +An important virtue of this approach is that it allows for adding new +optional arguments to NumPy functions without breaking code that already +relies on ``__array_function__``. + +This is not a theoretical concern. The implementation of overrides *within* +functions like ``np.sum()`` rather than defining a new function capturing +``*args`` and ``**kwargs`` necessitated some awkward gymnastics to ensure that +the new ``keepdims`` argument is only passed in cases where it is used, e.g., .. code:: python - @overload_for_array_function - def broadcast_to(array: ArrayLike - shape: Tuple[int, ...], - subok: bool = False): - ... # continue with the definition of broadcast_to - -The decorator ``overload_for_array_function`` would be written in terms -of ``try_array_function_override``, but would also need some level of magic -for (1) access to the wrapper function (``np.broadcast_to``) for passing into -``__array_function__`` implementations and (2) dynamic code generation -resembling the `decorator library <https://github.com/micheles/decorator>`_ -to automatically write an overloaded function like the manually written -implemenations above with the exact same signature as the original. -Unfortunately, using the ``inspect`` module instead of code generation would -probably be too slow: our prototype implementation adds ~15 microseconds of -overhead. - -We like the idea of writing overloads with minimal syntax, but dynamic -code generation also has potential downsides, such as slower import times, less -transparent code and added difficulty for static analysis tools. It's not clear -that tradeoffs would be worth it, especially because functions with complex -signatures like ``np.einsum`` would assuredly still need to invoke -``try_array_function_override`` directly. - -So we don't propose adding such a decorator yet, but it's something worth -considering for the future. + def sum(array, ..., keepdims=np._NoValue): + kwargs = {} + if keepdims is not np._NoValue: + kwargs['keepdims'] = keepdims + return array.sum(..., **kwargs) + +This also makes it possible to add optional arguments to ``__array_function__`` +implementations incrementally and only in cases where it makes sense. For +example, a library implementing immutable arrays would not be required to +explicitly include an unsupported ``out`` argument. Doing this properly for all +optional arguments is somewhat onerous, e.g., + +.. code:: python + + def my_sum(array, ..., out=None): + if out is not None: + raise TypeError('out argument is not supported') + ... + +We thus avoid encouraging the tempting shortcut of adding catch-all +``**ignored_kwargs`` to the signatures of functions called by NumPy, which fails +silently for misspelled or ignored arguments. + +Performance +~~~~~~~~~~~ + +Performance is always a concern with NumPy, even though NumPy users have +already prioritized usability over pure speed with their choice of the Python +language itself. It's important that this new ``__array_function__`` protocol +not impose a significant cost in the typical case of NumPy functions acting +on NumPy arrays. + +Our `microbenchmark results <https://nbviewer.jupyter.org/gist/shoyer/1f0a308a06cd96df20879a1ddb8f0006>`_ +show that a pure Python implementation of the override machinery described +above adds roughly 2-3 microseconds of overhead to each NumPy function call +without any overloaded arguments. For context, typical NumPy functions on small +arrays have a runtime of 1-10 microseconds, mostly determined by what fraction +of the function's logic is written in C. For example, one microsecond is about +the difference in speed between the ``ndarray.sum()`` method (1.6 us) and +``numpy.sum()`` function (2.6 us). + +Fortunately, we expect significantly less overhead with a C implementation of +``try_array_function_override``, which is where the bulk of the runtime is. +This would leave the ``array_function_dispatch`` decorator and dispatcher +function on their own adding about 0.5 microseconds of overhead, for perhaps ~1 +microsecond of overhead in the typical case. + +In our view, this level of overhead is reasonable to accept for code written +in Python. We're pretty sure that the vast majority of NumPy users aren't +concerned about performance differences measured in microsecond(s) on NumPy +functions, because it's difficult to do *anything* in Python in less than a +microsecond. Use outside of NumPy ~~~~~~~~~~~~~~~~~~~~ Nothing about this protocol that is particular to NumPy itself. Should -we enourage use of the same ``__array_function__`` protocol third-party +we encourage use of the same ``__array_function__`` protocol third-party libraries for overloading non-NumPy functions, e.g., for making array-implementation generic functionality in SciPy? @@ -347,8 +479,9 @@ to be explicitly recognized. Libraries like Dask, CuPy, and Autograd already wrap a limited subset of SciPy functionality (e.g., ``scipy.linalg``) similarly to how they wrap NumPy. -If we want to do this, we should expose the helper function -``try_array_function_override()`` as a public API. +If we want to do this, we should expose at least the decorator +``array_function_dispatch()`` and possibly also the lower level +``try_array_function_override()`` as part of NumPy's public API. Non-goals --------- @@ -418,7 +551,7 @@ either inside or outside of NumPy. This has the advantage of alleviating any possible concerns about backwards compatibility and would provide the maximum freedom for quick -experimentation. In the long term, it would provide a clean abstration +experimentation. In the long term, it would provide a clean abstraction layer, separating NumPy's high level API from default implementations on ``numpy.ndarray`` objects. @@ -446,7 +579,7 @@ don't think this approach makes sense for NumPy in the near term. The main reason is that NumPy already has a well-proven dispatching mechanism with ``__array_ufunc__``, based on Python's own dispatching -system for arithemtic, and it would be confusing to add another +system for arithmetic, and it would be confusing to add another mechanism that works in a very different way. This would also be more invasive change to NumPy itself, which would need to gain a multiple dispatch implementation. @@ -460,7 +593,7 @@ would be straightforward to write a shim for a default Implementations in terms of a limited core API ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The internal implemenations of some NumPy functions is extremely simple. +The internal implementations of some NumPy functions is extremely simple. For example: - ``np.stack()`` is implemented in only a few lines of code by combining @@ -489,10 +622,16 @@ However, this also comes with several downsides: subject of a future NEP, but until then we should be reluctant to rely on these properties. -Given these concerns, we encourage relying on this approach only in -limited cases. +Given these concerns, we think it's valuable to support explicit overloading of +nearly every public function in NumPy's API. This does not preclude the future +possibility of rewriting NumPy functions in terms of simplified core +functionality with ``__array_function__`` and a protocol and/or base class for +ensuring that arrays expose methods and properties like ``numpy.ndarray``. +However, to work well this would require the possibility of implementing +*some* but not all functions with ``__array_function__``, e.g., as described +in the next section. -Coersion to a NumPy array as a catch-all fallback +Coercion to a NumPy array as a catch-all fallback ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With the current design, classes that implement ``__array_function__`` @@ -545,64 +684,125 @@ fall-back to their current behavior of coercing all array-like arguments. It is not yet clear to us yet if we need an optional like ``NotImplementedButCoercible``, so for now we propose to defer this issue. We can always implement ``np.NotImplementedButCoercible`` at some later time if -it proves critical to the numpy community in the future. Importantly, we don't +it proves critical to the NumPy community in the future. Importantly, we don't think this will stop critical libraries that desire to implement most of the high level NumPy API from adopting this proposal. -NOTE: If you are reading this NEP in its draft state and disagree, -please speak up on the mailing list! +A magic decorator that inspects type annotations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Drawbacks of this approach --------------------------- +In principle, Python 3 type annotations contain sufficient information to +automatically create most ``dispatcher`` functions. It would be convenient to +use these annotations to dispense with the need for manually writing +dispatchers, e.g., -Future difficulty extending NumPy's API -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. code:: python -One downside of passing on all arguments directly on to -``__array_function__`` is that it makes it hard to extend the signatures -of overloaded NumPy functions with new arguments, because adding even an -optional keyword argument would break existing overloads. + @array_function_dispatch + def broadcast_to(array: ArrayLike + shape: Tuple[int, ...], + subok: bool = False): + ... # existing definition of np.broadcast_to -This is not a new problem for NumPy. NumPy has occasionally changed the -signature for functions in the past, including functions like -``numpy.sum`` which support overloads. +This would require some form of automatic code generation, either at compile or +import time. -For adding new keyword arguments that do not change default behavior, we -would only include these as keyword arguments when they have changed -from default values. This is similar to `what NumPy already has -done <https://github.com/numpy/numpy/blob/v1.14.2/numpy/core/fromnumeric.py#L1865-L1867>`_, -e.g., for the optional ``keepdims`` argument in ``sum``: +We think this is an interesting possible extension to consider in the future. We +don't think it makes sense to do so now, because code generation involves +tradeoffs and NumPy's experience with type annotations is still +`quite limited <https://github.com/numpy/numpy-stubs>`_. Even if NumPy +was Python 3 only (which will happen +`sometime in 2019 <http://www.numpy.org/neps/nep-0014-dropping-python2.7-proposal.html>`_), +we aren't ready to annotate NumPy's codebase directly yet. + +Support for implementation-specific arguments +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +We could allow ``__array_function__`` implementations to add their own +optional keyword arguments by including ``**ignored_kwargs`` in dispatcher +functions, e.g., .. code:: python - def sum(array, ..., keepdims=np._NoValue): - kwargs = {} - if keepdims is not np._NoValue: - kwargs['keepdims'] = keepdims - return array.sum(..., **kwargs) + def _concatenate_dispatcher(arrays, axis=None, out=None, **ignored_kwargs): + ... # same implementation of _concatenate_dispatcher as above + +Implementation-specific arguments are somewhat common in libraries that +otherwise emulate NumPy's higher level API (e.g., ``dask.array.sum()`` adds +``split_every`` and ``tensorflow.reduce_sum()`` adds ``name``). Supporting +them in NumPy would be particularly useful for libraries that implement new +high-level array functions on top of NumPy functions, e.g., + +.. code:: python -In other cases, such as deprecated arguments, preserving the existing -behavior of overloaded functions may not be possible. Libraries that use -``__array_function__`` should be aware of this risk: we don't propose to -freeze NumPy's API in stone any more than it already is. + def mean_squared_error(x, y, **kwargs): + return np.mean((x - y) ** 2, **kwargs) -Difficulty adding implementation specific arguments -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Otherwise, we would need separate versions of ``mean_squared_error`` for each +array implementation in order to pass implementation-specific arguments to +``mean()``. -Some array implementations generally follow NumPy's API, but have -additional optional keyword arguments (e.g., ``dask.array.sum()`` has -``split_every`` and ``tensorflow.reduce_sum()`` has ``name``). A generic -dispatching library could potentially pass on all unrecognized keyword -argument directly to the implementation, but extending ``np.sum()`` to -pass on ``**kwargs`` would entail public facing changes in NumPy. -Customizing the detailed behavior of array libraries will require using -library specific functions, which could be limiting in the case of -libraries that consume the NumPy API such as xarray. +We wouldn't allow adding optional positional arguments, because these are +reserved for future use by NumPy itself, but conflicts between keyword arguments +should be relatively rare. + +However, this flexibility would come with a cost. In particular, it implicitly +adds ``**kwargs`` to the signature for all wrapped NumPy functions without +actually including it (because we use ``functools.wraps``). This means it is +unlikely to work well with static analysis tools, which could report invalid +arguments. Likewise, there is a price in readability: these optional arguments +won't be included in the docstrings for NumPy functions. + +It's not clear that this tradeoff is worth it, so we propose to leave this out +for now. Adding implementation-specific arguments will require using those +libraries directly. + +Other possible choices for the protocol +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The array function ``__array_function__`` includes only two arguments, ``func`` +and ``types``, that provide information about the context of the function call. + +``func`` is part of the protocol because there is no way to avoid it: +implementations need to be able to dispatch by matching a function to NumPy's +public API. + +``types`` is included because we can compute it almost for free as part of +collecting ``__array_function__`` implementations to call in +``try_array_function_override``. We also think it will be used by most +``__array_function__`` methods, which otherwise would need to extract this +information themselves. It would be equivalently easy to provide single +instances of each type, but providing only types seemed cleaner. + +Taking this even further, it was suggested that ``__array_function__`` should be +a ``classmethod``. We agree that it would be a little cleaner to remove the +redundant ``self`` argument, but feel that this minor clean-up would not be +worth breaking from the precedence of ``__array_ufunc__``. + +There are two other arguments that we think *might* be important to pass to +``__array_ufunc__`` implementations: + +- Access to the non-dispatched function (i.e., before wrapping with + ``array_function_dispatch``) in ``ndarray.__array_function__`` would allow + use to drop special case logic for that method from + ``try_array_function_override``. +- Access to the ``dispatcher`` function passed into + ``array_function_dispatch()`` would allow ``__array_function__`` + implementations to determine the list of "array-like" arguments in a generic + way by calling ``dispatcher(*args, **kwargs)``. This *could* be useful for + ``__array_function__`` implementations that dispatch based on the value of an + array attribute (e.g., ``dtype`` or ``units``) rather than directly on the + array type. + +We have left these out for now, because we don't know that they are necessary. +If we want to include them in the future, the easiest way to do so would be to +update the ``array_function_dispatch`` decorator to add them as function +attributes. Discussion ---------- -Various alternatives to this proposal were discussed in a few Github issues: +Various alternatives to this proposal were discussed in a few GitHub issues: 1. `pydata/sparse #1 <https://github.com/pydata/sparse/issues/1>`_ 2. `numpy/numpy #11129 <https://github.com/numpy/numpy/issues/11129>`_ @@ -613,6 +813,12 @@ it was discussed at a `NumPy developer sprint <https://scisprints.github.io/#may-numpy-developer-sprint>`_ at the `UC Berkeley Institute for Data Science (BIDS) <https://bids.berkeley.edu/>`_. +Detailed discussion of this proposal itself can be found on the +`the mailing list <https://mail.python.org/pipermail/numpy-discussion/2018-June/078127.html>`_ and relvant pull requests +(`1 <https://github.com/numpy/numpy/pull/11189>`_, +`2 <https://github.com/numpy/numpy/pull/11303#issuecomment-396638175>`_, +`3 <https://github.com/numpy/numpy/pull/11374>`_) + Copyright --------- diff --git a/doc/neps/nep-0019-rng-policy.rst b/doc/neps/nep-0019-rng-policy.rst index de9164bba..a2cc80262 100644 --- a/doc/neps/nep-0019-rng-policy.rst +++ b/doc/neps/nep-0019-rng-policy.rst @@ -91,23 +91,12 @@ those contributors simply walked away. Implementation -------------- -We propose first freezing ``RandomState`` as it is and developing a new RNG -subsystem alongside it. This allows anyone who has been relying on our old -stream-compatibility guarantee to have plenty of time to migrate. -``RandomState`` will be considered deprecated, but with a long deprecation -cycle, at least a few years. Deprecation warnings will start silent but become -increasingly noisy over time. Bugs in the current state of the code will *not* -be fixed if fixing them would impact the stream. However, if changes in the -rest of ``numpy`` would break something in the ``RandomState`` code, we will -fix ``RandomState`` to continue working (for example, some change in the -C API). No new features will be added to ``RandomState``. Users should -migrate to the new subsystem as they are able to. - -Work on a proposed `new PRNG subsystem -<https://github.com/bashtage/randomgen>`_ is already underway. The specifics -of the new design are out of scope for this NEP and up for much discussion, but -we will discuss general policies that will guide the evolution of whatever code -is adopted. +Work on a proposed new PRNG subsystem is already underway in the randomgen_ +project. The specifics of the new design are out of scope for this NEP and up +for much discussion, but we will discuss general policies that will guide the +evolution of whatever code is adopted. We will also outline just a few of the +requirements that such a new system must have to support the policy proposed in +this NEP. First, we will maintain API source compatibility just as we do with the rest of ``numpy``. If we *must* make a breaking change, we will only do so with an @@ -116,66 +105,156 @@ appropriate deprecation period and warnings. Second, breaking stream-compatibility in order to introduce new features or improve performance will be *allowed* with *caution*. Such changes will be considered features, and as such will be no faster than the standard release -cadence of features (i.e. on ``X.Y`` releases, never ``X.Y.Z``). Slowness is -not a bug. Correctness bug fixes that break stream-compatibility can happen on -bugfix releases, per usual, but developers should consider if they can wait -until the next feature release. We encourage developers to strongly weight -user’s pain from the break in stream-compatibility against the improvements. -One example of a worthwhile improvement would be to change algorithms for -a significant increase in performance, for example, moving from the `Box-Muller -transform <https://en.wikipedia.org/wiki/Box%E2%80%93Muller_transform>`_ method -of Gaussian variate generation to the faster `Ziggurat algorithm -<https://en.wikipedia.org/wiki/Ziggurat_algorithm>`_. An example of an -unworthy improvement would be tweaking the Ziggurat tables just a little bit. +cadence of features (i.e. on ``X.Y`` releases, never ``X.Y.Z``). Slowness will +not be considered a bug for this purpose. Correctness bug fixes that break +stream-compatibility can happen on bugfix releases, per usual, but developers +should consider if they can wait until the next feature release. We encourage +developers to strongly weight user’s pain from the break in +stream-compatibility against the improvements. One example of a worthwhile +improvement would be to change algorithms for a significant increase in +performance, for example, moving from the `Box-Muller transform +<https://en.wikipedia.org/wiki/Box%E2%80%93Muller_transform>`_ method of +Gaussian variate generation to the faster `Ziggurat algorithm +<https://en.wikipedia.org/wiki/Ziggurat_algorithm>`_. An example of a +discouraged improvement would be tweaking the Ziggurat tables just a little bit +for a small performance improvement. Any new design for the RNG subsystem will provide a choice of different core -uniform PRNG algorithms. We will be more strict about a select subset of -methods on these core PRNG objects. They MUST guarantee stream-compatibility -for a minimal, specified set of methods which are chosen to make it easier to -compose them to build other distributions. Namely, +uniform PRNG algorithms. A promising design choice is to make these core +uniform PRNGs their own lightweight objects with a minimal set of methods +(randomgen_ calls them “basic RNGs”). The broader set of non-uniform +distributions will be its own class that holds a reference to one of these core +uniform PRNG objects and simply delegates to the core uniform PRNG object when +it needs uniform random numbers. To borrow an example from randomgen_, the +class ``MT19937`` is a basic RNG that implements the classic Mersenne Twister +algorithm. The class ``RandomGenerator`` wraps around the basic RNG to provide +all of the non-uniform distribution methods:: + + # This is not the only way to instantiate this object. + # This is just handy for demonstrating the delegation. + >>> brng = MT19937(seed) + >>> rg = RandomGenerator(brng) + >>> x = rg.standard_normal(10) + +We will be more strict about a select subset of methods on these basic RNG +objects. They MUST guarantee stream-compatibility for a specified set +of methods which are chosen to make it easier to compose them to build other +distributions and which are needed to abstract over the implementation details +of the variety of core PRNG algorithms. Namely, * ``.bytes()`` * ``.random_uintegers()`` * ``.random_sample()`` -Furthermore, the new design should also provide one generator class (we shall -call it ``StableRandom`` for discussion purposes) that provides a slightly -broader subset of distribution methods for which stream-compatibility is -*guaranteed*. The point of ``StableRandom`` is to provide something that can -be used in unit tests so projects that currently have tests which rely on the -precise stream can be migrated off of ``RandomState``. For the best -transition, ``StableRandom`` should use as its core uniform PRNG the current -MT19937 algorithm. As best as possible, the API for the distribution methods -that are provided on ``StableRandom`` should match their counterparts on -``RandomState``. They should provide the same stream that the current version -of ``RandomState`` does. Because their intended use is for unit tests, we do -not need the performance improvements from the new algorithms that will be -introduced by the new subsystem. - -The list of ``StableRandom`` methods should be chosen to support unit tests: - - * ``.randint()`` - * ``.uniform()`` - * ``.normal()`` - * ``.standard_normal()`` - * ``.choice()`` - * ``.shuffle()`` - * ``.permutation()`` - - -Not Versioning --------------- +The distributions class (``RandomGenerator``) SHOULD have all of the same +distribution methods as ``RandomState`` with close-enough function signatures +such that almost all code that currently works with ``RandomState`` instances +will work with ``RandomGenerator`` instances (ignoring the precise stream +values). Some variance will be allowed for integer distributions: in order to +avoid some of the cross-platform problems described above, these SHOULD be +rewritten to work with ``uint64`` numbers on all platforms. + +.. _randomgen: https://github.com/bashtage/randomgen + + +Supporting Unit Tests +::::::::::::::::::::: + +Because we did make a strong stream-compatibility guarantee early in numpy’s +life, reliance on stream-compatibility has grown beyond reproducible +simulations. One use case that remains for stream-compatibility across numpy +versions is to use pseudorandom streams to generate test data in unit tests. +With care, many of the cross-platform instabilities can be avoided in the +context of small unit tests. + +The new PRNG subsystem MUST provide a second, legacy distributions class that +uses the same implementations of the distribution methods as the current +version of ``numpy.random.RandomState``. The methods of this class will keep +the same strict stream-compatibility guarantees. It is intended that this +class will no longer be modified, except to keep it working when numpy +internals change. All new development should go into the primary distributions +class. The purpose of ``RandomState`` will be documented as providing certain +fixed functionality for backwards compatibility and stable numbers for the +limited purpose of unit testing, and not making whole programs reproducible +across numpy versions. + +This legacy distributions class MUST be accessible under the name +``numpy.random.RandomState`` for backwards compatibility. All current ways of +instantiating ``numpy.random.RandomState`` with a given state should +instantiate the Mersenne Twister basic RNG with the same state. The legacy +distributions class MUST be capable of accepting other basic RNGs. The purpose +here is to ensure that one can write a program with a consistent basic RNG +state with a mixture of libraries that may or may not have upgraded from +``RandomState``. Instances of the legacy distributions class MUST respond +``True`` to ``isinstance(rg, numpy.random.RandomState)`` because there is +current utility code that relies on that check. Similarly, old pickles of +``numpy.random.RandomState`` instances MUST unpickle correctly. + + +``numpy.random.*`` +:::::::::::::::::: + +The preferred best practice for getting reproducible pseudorandom numbers is to +instantiate a generator object with a seed and pass it around. The implicit +global ``RandomState`` behind the ``numpy.random.*`` convenience functions can +cause problems, especially when threads or other forms of concurrency are +involved. Global state is always problematic. We categorically recommend +avoiding using the convenience functions when reproducibility is involved. + +That said, people do use them and use ``numpy.random.seed()`` to control the +state underneath them. It can be hard to categorize and count API usages +consistently and usefully, but a very common usage is in unit tests where many +of the problems of global state are less likely. + +This NEP does not propose removing these functions or changing them to use the +less-stable ``RandomGenerator`` distribution implementations. Future NEPs +might. + +Specifically, the initial release of the new PRNG subsystem SHALL leave these +convenience functions as aliases to the methods on a global ``RandomState`` +that is initialized with a Mersenne Twister basic RNG object. A call to +``numpy.random.seed()`` will be forwarded to that basic RNG object. In +addition, the global ``RandomState`` instance MUST be accessible in this +initial release by the name ``numpy.random.mtrand._rand``: Robert Kern long ago +promised ``scikit-learn`` that this name would be stable. Whoops. + +In order to allow certain workarounds, it MUST be possible to replace the basic +RNG underneath the global ``RandomState`` with any other basic RNG object (we +leave the precise API details up to the new subsystem). Calling +``numpy.random.seed()`` thereafter SHOULD just pass the given seed to the +current basic RNG object and not attempt to reset the basic RNG to the Mersenne +Twister. The set of ``numpy.random.*`` convenience functions SHALL remain the +same as they currently are. They SHALL be aliases to the ``RandomState`` +methods and not the new less-stable distributions class (``RandomGenerator``, +in the examples above). Users who want to get the fastest, best distributions +can follow best practices and instantiate generator objects explicitly. + +This NEP does not propose that these requirements remain in perpetuity. After +we have experience with the new PRNG subsystem, we can and should revisit these +issues in future NEPs. + + +Alternatives +------------ + +Versioning +:::::::::: For a long time, we considered that the way to allow algorithmic improvements while maintaining the stream was to apply some form of versioning. That is, every time we make a stream change in one of the distributions, we increment some version number somewhere. ``numpy.random`` would keep all past versions -of the code, and there would be a way to get the old versions. Proposals of -how to do this exactly varied widely, but we will not exhaustively list them -here. We spent years going back and forth on these designs and were not able -to find one that sufficed. Let that time lost, and more importantly, the -contributors that we lost while we dithered, serve as evidence against the -notion. +of the code, and there would be a way to get the old versions. + +We will not be doing this. If one needs to get the exact bit-for-bit results +from a given version of ``numpy``, whether one uses random numbers or not, one +should use the exact version of ``numpy``. + +Proposals of how to do RNG versioning varied widely, and we will not +exhaustively list them here. We spent years going back and forth on these +designs and were not able to find one that sufficed. Let that time lost, and +more importantly, the contributors that we lost while we dithered, serve as +evidence against the notion. Concretely, adding in versioning makes maintenance of ``numpy.random`` difficult. Necessarily, we would be keeping lots of versions of the same code @@ -195,11 +274,49 @@ is to pin the release of ``numpy`` as a whole, versioning ``RandomState`` alone is superfluous. +``StableRandom`` +:::::::::::::::: + +A previous version of this NEP proposed to leave ``RandomState`` completely +alone for a deprecation period and build the new subsystem alongside with new +names. To satisfy the unit testing use case, it proposed introducing a small +distributions class nominally called ``StableRandom``. It would have provided +a small subset of distribution methods that were considered most useful in unit +testing, but not the full set such that it would be too likely to be used +outside of the testing context. + +During discussion about this proposal, it became apparent that there was no +satisfactory subset. At least some projects used a fairly broad selection of +the ``RandomState`` methods in unit tests. + +Downstream project owners would have been forced to modify their code to +accomodate the new PRNG subsystem. Some modifications might be simply +mechanical, but the bulk of the work would have been tedious churn for no +positive improvement to the downstream project, just avoiding being broken. + +Furthermore, under this old proposal, we would have had a quite lengthy +deprecation period where ``RandomState`` existed alongside the new system of +basic RNGs and distribution classes. Leaving the implementation of +``RandomState`` fixed meant that it could not use the new basic RNG state +objects. Developing programs that use a mixture of libraries that have and +have not upgraded would require managing two sets of PRNG states. This would +notionally have been time-limited, but we intended the deprecation to be very +long. + +The current proposal solves all of these problems. All current usages of +``RandomState`` will continue to work in perpetuity, though some may be +discouraged through documentation. Unit tests can continue to use the full +complement of ``RandomState`` methods. Mixed ``RandomState/RandomGenerator`` +code can safely share the common basic RNG state. Unmodified ``RandomState`` +code can make use of the new features of alternative basic RNGs like settable +streams. + + Discussion ---------- -- https://mail.python.org/pipermail/numpy-discussion/2018-January/077608.html -- https://github.com/numpy/numpy/pull/10124#issuecomment-350876221 +- `NEP discussion <https://mail.python.org/pipermail/numpy-discussion/2018-June/078126.html>`_ +- `Earlier discussion <https://mail.python.org/pipermail/numpy-discussion/2018-January/077608.html>`_ Copyright diff --git a/doc/neps/nep-0021-advanced-indexing.rst b/doc/neps/nep-0021-advanced-indexing.rst new file mode 100644 index 000000000..0279146be --- /dev/null +++ b/doc/neps/nep-0021-advanced-indexing.rst @@ -0,0 +1,658 @@ +========================================= +Simplified and explicit advanced indexing +========================================= + +:Author: Sebastian Berg +:Author: Stephan Hoyer <shoyer@google.com> +:Status: Draft +:Type: Standards Track +:Created: 2015-08-27 + + +Abstract +-------- + +NumPy's "advanced" indexing support for indexing array with other arrays is +one of its most powerful and popular features. Unfortunately, the existing +rules for advanced indexing with multiple array indices are typically confusing +to both new, and in many cases even old, users of NumPy. Here we propose an +overhaul and simplification of advanced indexing, including two new "indexer" +attributes ``oindex`` and ``vindex`` to facilitate explicit indexing. + +Background +---------- + +Existing indexing operations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +NumPy arrays currently support a flexible range of indexing operations: + +- "Basic" indexing involving only slices, integers, ``np.newaxis`` and ellipsis + (``...``), e.g., ``x[0, :3, np.newaxis]`` for selecting the first element + from the 0th axis, the first three elements from the 1st axis and inserting a + new axis of size 1 at the end. Basic indexing always return a view of the + indexed array's data. +- "Advanced" indexing, also called "fancy" indexing, includes all cases where + arrays are indexed by other arrays. Advanced indexing always makes a copy: + + - "Boolean" indexing by boolean arrays, e.g., ``x[x > 0]`` for + selecting positive elements. + - "Vectorized" indexing by one or more integer arrays, e.g., ``x[[0, 1]]`` + for selecting the first two elements along the first axis. With multiple + arrays, vectorized indexing uses broadcasting rules to combine indices along + multiple dimensions. This allows for producing a result of arbitrary shape + with arbitrary elements from the original arrays. + - "Mixed" indexing involving any combinations of the other advancing types. + This is no more powerful than vectorized indexing, but is sometimes more + convenient. + +For clarity, we will refer to these existing rules as "legacy indexing". +This is only a high-level summary; for more details, see NumPy's documentation +and and `Examples` below. + +Outer indexing +~~~~~~~~~~~~~~ + +One broadly useful class of indexing operations is not supported: + +- "Outer" or orthogonal indexing treats one-dimensional arrays equivalently to + slices for determining output shapes. The rule for outer indexing is that the + result should be equivalent to independently indexing along each dimension + with integer or boolean arrays as if both the indexed and indexing arrays + were one-dimensional. This form of indexing is familiar to many users of other + programming languages such as MATLAB, Fortran and R. + +The reason why NumPy omits support for outer indexing is that the rules for +outer and vectorized conflict. Consider indexing a 2D array by two 1D integer +arrays, e.g., ``x[[0, 1], [0, 1]]``: + +- Outer indexing is equivalent to combining multiple integer indices with + ``itertools.product()``. The result in this case is another 2D array with + all combinations of indexed elements, e.g., + ``np.array([[x[0, 0], x[0, 1]], [x[1, 0], x[1, 1]]])`` +- Vectorized indexing is equivalent to combining multiple integer indices with + ``zip()``. The result in this case is a 1D array containing the diagonal + elements, e.g., ``np.array([x[0, 0], x[1, 1]])``. + +This difference is a frequent stumbling block for new NumPy users. The outer +indexing model is easier to understand, and is a natural generalization of +slicing rules. But NumPy instead chose to support vectorized indexing, because +it is strictly more powerful. + +It is always possible to emulate outer indexing by vectorized indexing with +the right indices. To make this easier, NumPy includes utility objects and +functions such as ``np.ogrid`` and ``np.ix_``, e.g., +``x[np.ix_([0, 1], [0, 1])]``. However, there are no utilities for emulating +fully general/mixed outer indexing, which could unambiguously allow for slices, +integers, and 1D boolean and integer arrays. + +Mixed indexing +~~~~~~~~~~~~~~ + +NumPy's existing rules for combining multiple types of indexing in the same +operation are quite complex, involving a number of edge cases. + +One reason why mixed indexing is particularly confusing is that at first glance +the result works deceptively like outer indexing. Returning to our example of a +2D array, both ``x[:2, [0, 1]]`` and ``x[[0, 1], :2]`` return 2D arrays with +axes in the same order as the original array. + +However, as soon as two or more non-slice objects (including integers) are +introduced, vectorized indexing rules apply. The axes introduced by the array +indices are at the front, unless all array indices are consecutive, in which +case NumPy deduces where the user "expects" them to be. Consider indexing a 3D +array ``arr`` with shape ``(X, Y, Z)``: + +1. ``arr[:, [0, 1], 0]`` has shape ``(X, 2)``. +2. ``arr[[0, 1], 0, :]`` has shape ``(2, Z)``. +3. ``arr[0, :, [0, 1]]`` has shape ``(2, Y)``, not ``(Y, 2)``! + +These first two cases are intuitive and consistent with outer indexing, but +this last case is quite surprising, even to many higly experienced NumPy users. + +Mixed cases involving multiple array indices are also surprising, and only +less problematic because the current behavior is so useless that it is rarely +encountered in practice. When a boolean array index is mixed with another boolean or +integer array, boolean array is converted to integer array indices (equivalent +to ``np.nonzero()``) and then broadcast. For example, indexing a 2D array of +size ``(2, 2)`` like ``x[[True, False], [True, False]]`` produces a 1D vector +with shape ``(1,)``, not a 2D sub-matrix with shape ``(1, 1)``. + +Mixed indexing seems so tricky that it is tempting to say that it never should +be used. However, it is not easy to avoid, because NumPy implicitly adds full +slices if there are fewer indices than the full dimensionality of the indexed +array. This means that indexing a 2D array like `x[[0, 1]]`` is equivalent to +``x[[0, 1], :]``. These cases are not surprising, but they constrain the +behavior of mixed indexing. + +Indexing in other Python array libraries +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Indexing is a useful and widely recognized mechanism for accessing +multi-dimensional array data, so it is no surprise that many other libraries in +the scientific Python ecosystem also support array indexing. + +Unfortunately, the full complexity of NumPy's indexing rules mean that it is +both challenging and undesirable for other libraries to copy its behavior in all +of its nuance. The only full implementation of NumPy-style indexing is NumPy +itself. This includes projects like dask.array and h5py, which support *most* +types of array indexing in some form, and otherwise attempt to copy NumPy's API +exactly. + +Vectorized indexing in particular can be challenging to implement with array +storage backends not based on NumPy. In contrast, indexing by 1D arrays along +at least one dimension in the style of outer indexing is much more acheivable. +This has led many libraries (including dask and h5py) to attempt to define a +safe subset of NumPy-style indexing that is equivalent to outer indexing, e.g., +by only allowing indexing with an array along at most one dimension. However, +this is quite challenging to do correctly in a general enough way to be useful. +For example, the current versions of dask and h5py both handle mixed indexing +in case 3 above inconsistently with NumPy. This is quite likely to lead to +bugs. + +These inconsistencies, in addition to the broader challenge of implementing +every type of indexing logic, make it challenging to write high-level array +libraries like xarray or dask.array that can interchangeably index many types of +array storage. In contrast, explicit APIs for outer and vectorized indexing in +NumPy would provide a model that external libraries could reliably emulate, even +if they don't support every type of indexing. + +High level changes +------------------ + +Inspired by multiple "indexer" attributes for controlling different types +of indexing behavior in pandas, we propose to: + +1. Introduce ``arr.oindex[indices]`` which allows array indices, but + uses outer indexing logic. +2. Introduce ``arr.vindex[indices]`` which use the current + "vectorized"/broadcasted logic but with two differences from + legacy indexing: + + * Boolean indices are not supported. All indices must be integers, + integer arrays or slices. + * The integer index result dimensions are always the first axes + of the result array. No transpose is done, even for a single + integer array index. + +3. Plain indexing on arrays will start to give warnings and eventually + errors in cases where one of the explicit indexers should be preferred: + + * First, in all cases where legacy and outer indexing would give + different results. + * Later, potentially in all cases involving an integer array. + +These constraints are sufficient for making indexing generally consistent +with expectations and providing a less surprising learning curve with +``oindex``. + +Note that all things mentioned here apply both for assignment as well as +subscription. + +Understanding these details is *not* easy. The `Examples` section in the +discussion gives code examples. +And the hopefully easier `Motivational Example` provides some +motivational use-cases for the general ideas and is likely a good start for +anyone not intimately familiar with advanced indexing. + + +Detailed Description +-------------------- + +Proposed rules +~~~~~~~~~~~~~~ + +From the three problems noted above some expectations for NumPy can +be deduced: + +1. There should be a prominent outer/orthogonal indexing method such as + ``arr.oindex[indices]``. + +2. Considering how confusing vectorized/fancy indexing can be, it should + be possible to be made more explicitly (e.g. ``arr.vindex[indices]``). + +3. A new ``arr.vindex[indices]`` method, would not be tied to the + confusing transpose rules of fancy indexing, which is for example + needed for the simple case of a single advanced index. Thus, + no transposing should be done. The axes created by the integer array + indices are always inserted at the front, even for a single index. + +4. Boolean indexing is conceptionally outer indexing. Broadcasting + together with other advanced indices in the manner of legacy + indexing is generally not helpful or well defined. + A user who wishes the "``nonzero``" plus broadcast behaviour can thus + be expected to do this manually. Thus, ``vindex`` does not need to + support boolean index arrays. + +5. An ``arr.legacy_index`` attribute should be implemented to support + legacy indexing. This gives a simple way to update existing codebases + using legacy indexing, which will make the deprecation of plain indexing + behavior easier. The longer name ``legacy_index`` is intentionally chosen + to be explicit and discourage its use in new code. + +6. Plain indexing ``arr[...]`` should return an error for ambiguous cases. + For the beginning, this probably means cases where ``arr[ind]`` and + ``arr.oindex[ind]`` return different results give deprecation warnings. + This includes every use of vectorized indexing with multiple integer arrays. + Due to the transposing behaviour, this means that``arr[0, :, index_arr]`` + will be deprecated, but ``arr[:, 0, index_arr]`` will not for the time being. + +7. To ensure that existing subclasses of `ndarray` that override indexing + do not inadvertently revert to default behavior for indexing attributes, + these attribute should have explicit checks that disable them if + ``__getitem__`` or ``__setitem__`` has been overriden. + +Unlike plain indexing, the new indexing attributes are explicitly aimed +at higher dimensional indexing, several additional changes should be implemented: + +* The indexing attributes will enforce exact dimension and indexing match. + This means that no implicit ellipsis (``...``) will be added. Unless + an ellipsis is present the indexing expression will thus only work for + an array with a specific number of dimensions. + This makes the expression more explicit and safeguards against wrong + dimensionality of arrays. + There should be no implications for "duck typing" compatibility with + builtin Python sequences, because Python sequences only support a limited + form of "basic indexing" with integers and slices. + +* The current plain indexing allows for the use of non-tuples for + multi-dimensional indexing such as ``arr[[slice(None), 2]]``. + This creates some inconsistencies and thus the indexing attributes + should only allow plain python tuples for this purpose. + (Whether or not this should be the case for plain indexing is a + different issue.) + +* The new attributes should not use getitem to implement setitem, + since it is a cludge and not useful for vectorized + indexing. (not implemented yet) + + +Open Questions +~~~~~~~~~~~~~~ + +* The names ``oindex``, ``vindex`` and ``legacy_index`` are just suggestions at + the time of writing this, another name NumPy has used for something like + ``oindex`` is ``np.ix_``. See also below. + +* ``oindex`` and ``vindex`` could always return copies, even when no array + operation occurs. One argument for allowing a view return is that this way + ``oindex`` can be used as a general index replacement. + However, there is one argument for returning copies. It is possible for + ``arr.vindex[array_scalar, ...]``, where ``array_scalar`` should be + a 0-D array but is not, since 0-D arrays tend to be converted. + Copying always "fixes" this possible inconsistency. + +* The final state to morph plain indexing in is not fixed in this PEP. + It is for example possible that `arr[index]`` will be equivalent to + ``arr.oindex`` at some point in the future. + Since such a change will take years, it seems unnecessary to make + specific decisions at this time. + +* The proposed changes to plain indexing could be postponed indefinitely or + not taken in order to not break or force major fixes to existing code bases. + + +Alternative Names +~~~~~~~~~~~~~~~~~ + +Possible names suggested (more suggestions will be added). + +============== ============ ======== +**Orthogonal** oindex oix +**Vectorized** vindex vix +**Legacy** legacy_index l/findex +============== ============ ======== + + +Subclasses +~~~~~~~~~~ + +Subclasses are a bit problematic in the light of these changes. There are +some possible solutions for this. For most subclasses (those which do not +provide ``__getitem__`` or ``__setitem__``) the special attributes should +just work. Subclasses that *do* provide it must be updated accordingly +and should preferably not subclass working versions of these attributes. + +All subclasses will inherit the attributes, however, the implementation +of ``__getitem__`` on these attributes should test +``subclass.__getitem__ is ndarray.__getitem__``. If not, the +subclass has special handling for indexing and ``NotImplementedError`` +should be raised, requiring that the indexing attributes is also explicitly +overwritten. Likewise, implementations of ``__setitem__`` should check to see +if ``__setitem__`` is overriden. + +A further question is how to facilitate implementing the special attributes. +Also there is the weird functionality where ``__setitem__`` calls +``__getitem__`` for non-advanced indices. It might be good to avoid it for +the new attributes, but on the other hand, that may make it even more +confusing. + +To facilitate implementations we could provide functions similar to +``operator.itemgetter`` and ``operator.setitem`` for the attributes. +Possibly a mixin could be provided to help implementation. These improvements +are not essential to the initial implementation, so they are saved for +future work. + +Implementation +-------------- + +Implementation would start with writing special indexing objects available +through ``arr.oindex``, ``arr.vindex``, and ``arr.legacy_index`` to allow these +indexing operations. Also, we would need to start to deprecate those plain index +operations which are not ambiguous. +Furthermore, the NumPy code base will need to use the new attributes and +tests will have to be adapted. + + +Backward compatibility +---------------------- + +As a new feature, no backward compatibility issues with the new ``vindex`` +and ``oindex`` attributes would arise. To facilitate backwards compatibility +as much as possible, we expect a long deprecation cycle for legacy indexing +behavior and propose the new ``legacy_index`` attribute. +Some forward compatibility issues with subclasses that do not specifically +implement the new methods may arise. + + +Alternatives +------------ + +NumPy may not choose to offer these different type of indexing methods, or +choose to only offer them through specific functions instead of the proposed +notation above. + +We don't think that new functions are a good alternative, because indexing +notation ``[]`` offer some syntactic advantages in Python (i.e., direct +creation of slice objects) compared to functions. + +A more reasonable alternative would be write new wrapper objects for alternative +indexing with functions rather than methods (e.g., ``np.oindex(arr)[indices]`` +instead of ``arr.oindex[indices]``). Functionally, this would be equivalent, +but indexing is such a common operation that we think it is important to +minimize syntax and worth implementing it directly on `ndarray` objects +themselves. Indexing attributes also define a clear interface that is easier +for alternative array implementations to copy, nonwithstanding ongoing +efforts to make it easier to override NumPy functions [2]_. + +Discussion +---------- + +The original discussion about vectorized vs outer/orthogonal indexing arose +on the NumPy mailing list: + + * https://mail.python.org/pipermail/numpy-discussion/2015-April/072550.html + +Some discussion can be found on the original pull request for this NEP: + + * https://github.com/numpy/numpy/pull/6256 + +Python implementations of the indexing operations can be found at: + + * https://github.com/numpy/numpy/pull/5749 + * https://gist.github.com/shoyer/c700193625347eb68fee4d1f0dc8c0c8 + + +Examples +~~~~~~~~ + +Since the various kinds of indexing is hard to grasp in many cases, these +examples hopefully give some more insights. Note that they are all in terms +of shape. +In the examples, all original dimensions have 5 or more elements, +advanced indexing inserts smaller dimensions. +These examples may be hard to grasp without working knowledge of advanced +indexing as of NumPy 1.9. + +Example array:: + + >>> arr = np.ones((5, 6, 7, 8)) + + +Legacy fancy indexing +--------------------- + +Note that the same result can be achieved with ``arr.legacy_index``, but the +"future error" will still work in this case. + +Single index is transposed (this is the same for all indexing types):: + + >>> arr[[0], ...].shape + (1, 6, 7, 8) + >>> arr[:, [0], ...].shape + (5, 1, 7, 8) + + +Multiple indices are transposed *if* consecutive:: + + >>> arr[:, [0], [0], :].shape # future error + (5, 1, 8) + >>> arr[:, [0], :, [0]].shape # future error + (1, 5, 7) + + +It is important to note that a scalar *is* integer array index in this sense +(and gets broadcasted with the other advanced index):: + + >>> arr[:, [0], 0, :].shape + (5, 1, 8) + >>> arr[:, [0], :, 0].shape # future error (scalar is "fancy") + (1, 5, 7) + + +Single boolean index can act on multiple dimensions (especially the whole +array). It has to match (as of 1.10. a deprecation warning) the dimensions. +The boolean index is otherwise identical to (multiple consecutive) integer +array indices:: + + >>> # Create boolean index with one True value for the last two dimensions: + >>> bindx = np.zeros((7, 8), dtype=np.bool_) + >>> bindx[0, 0] = True + >>> arr[:, 0, bindx].shape + (5, 1) + >>> arr[0, :, bindx].shape + (1, 6) + + +The combination with anything that is not a scalar is confusing, e.g.:: + + >>> arr[[0], :, bindx].shape # bindx result broadcasts with [0] + (1, 6) + >>> arr[:, [0, 1], bindx].shape # IndexError + + +Outer indexing +-------------- + +Multiple indices are "orthogonal" and their result axes are inserted +at the same place (they are not broadcasted):: + + >>> arr.oindex[:, [0], [0, 1], :].shape + (5, 1, 2, 8) + >>> arr.oindex[:, [0], :, [0, 1]].shape + (5, 1, 7, 2) + >>> arr.oindex[:, [0], 0, :].shape + (5, 1, 8) + >>> arr.oindex[:, [0], :, 0].shape + (5, 1, 7) + + +Boolean indices results are always inserted where the index is:: + + >>> # Create boolean index with one True value for the last two dimensions: + >>> bindx = np.zeros((7, 8), dtype=np.bool_) + >>> bindx[0, 0] = True + >>> arr.oindex[:, 0, bindx].shape + (5, 1) + >>> arr.oindex[0, :, bindx].shape + (6, 1) + + +Nothing changed in the presence of other advanced indices since:: + + >>> arr.oindex[[0], :, bindx].shape + (1, 6, 1) + >>> arr.oindex[:, [0, 1], bindx].shape + (5, 2, 1) + + +Vectorized/inner indexing +------------------------- + +Multiple indices are broadcasted and iterated as one like fancy indexing, +but the new axes area always inserted at the front:: + + >>> arr.vindex[:, [0], [0, 1], :].shape + (2, 5, 8) + >>> arr.vindex[:, [0], :, [0, 1]].shape + (2, 5, 7) + >>> arr.vindex[:, [0], 0, :].shape + (1, 5, 8) + >>> arr.vindex[:, [0], :, 0].shape + (1, 5, 7) + + +Boolean indices results are always inserted where the index is, exactly +as in ``oindex`` given how specific they are to the axes they operate on:: + + >>> # Create boolean index with one True value for the last two dimensions: + >>> bindx = np.zeros((7, 8), dtype=np.bool_) + >>> bindx[0, 0] = True + >>> arr.vindex[:, 0, bindx].shape + (5, 1) + >>> arr.vindex[0, :, bindx].shape + (6, 1) + + +But other advanced indices are again transposed to the front:: + + >>> arr.vindex[[0], :, bindx].shape + (1, 6, 1) + >>> arr.vindex[:, [0, 1], bindx].shape + (2, 5, 1) + + +Motivational Example +~~~~~~~~~~~~~~~~~~~~ + +Imagine having a data acquisition software storing ``D`` channels and +``N`` datapoints along the time. She stores this into an ``(N, D)`` shaped +array. During data analysis, we needs to fetch a pool of channels, for example +to calculate a mean over them. + +This data can be faked using:: + + >>> arr = np.random.random((100, 10)) + +Now one may remember indexing with an integer array and find the correct code:: + + >>> group = arr[:, [2, 5]] + >>> mean_value = arr.mean() + +However, assume that there were some specific time points (first dimension +of the data) that need to be specially considered. These time points are +already known and given by:: + + >>> interesting_times = np.array([1, 5, 8, 10], dtype=np.intp) + +Now to fetch them, we may try to modify the previous code:: + + >>> group_at_it = arr[interesting_times, [2, 5]] + IndexError: Ambiguous index, use `.oindex` or `.vindex` + +An error such as this will point to read up the indexing documentation. +This should make it clear, that ``oindex`` behaves more like slicing. +So, out of the different methods it is the obvious choice +(for now, this is a shape mismatch, but that could possibly also mention +``oindex``):: + + >>> group_at_it = arr.oindex[interesting_times, [2, 5]] + +Now of course one could also have used ``vindex``, but it is much less +obvious how to achieve the right thing!:: + + >>> reshaped_times = interesting_times[:, np.newaxis] + >>> group_at_it = arr.vindex[reshaped_times, [2, 5]] + + +One may find, that for example our data is corrupt in some places. +So, we need to replace these values by zero (or anything else) for these +times. The first column may for example give the necessary information, +so that changing the values becomes easy remembering boolean indexing:: + + >>> bad_data = arr[:, 0] > 0.5 + >>> arr[bad_data, :] = 0 # (corrupts further examples) + +Again, however, the columns may need to be handled more individually (but in +groups), and the ``oindex`` attribute works well:: + + >>> arr.oindex[bad_data, [2, 5]] = 0 + +Note that it would be very hard to do this using legacy fancy indexing. +The only way would be to create an integer array first:: + + >>> bad_data_indx = np.nonzero(bad_data)[0] + >>> bad_data_indx_reshaped = bad_data_indx[:, np.newaxis] + >>> arr[bad_data_indx_reshaped, [2, 5]] + +In any case we can use only ``oindex`` to do all of this without getting +into any trouble or confused by the whole complexity of advanced indexing. + +But, some new features are added to the data acquisition. Different sensors +have to be used depending on the times. Let us assume we already have +created an array of indices:: + + >>> correct_sensors = np.random.randint(10, size=(100, 2)) + +Which lists for each time the two correct sensors in an ``(N, 2)`` array. + +A first try to achieve this may be ``arr[:, correct_sensors]`` and this does +not work. It should be clear quickly that slicing cannot achieve the desired +thing. But hopefully users will remember that there is ``vindex`` as a more +powerful and flexible approach to advanced indexing. +One may, if trying ``vindex`` randomly, be confused about:: + + >>> new_arr = arr.vindex[:, correct_sensors] + +which is neither the same, nor the correct result (see transposing rules)! +This is because slicing works still the same in ``vindex``. However, reading +the documentation and examples, one can hopefully quickly find the desired +solution:: + + >>> rows = np.arange(len(arr)) + >>> rows = rows[:, np.newaxis] # make shape fit with correct_sensors + >>> new_arr = arr.vindex[rows, correct_sensors] + +At this point we have left the straight forward world of ``oindex`` but can +do random picking of any element from the array. Note that in the last example +a method such as mentioned in the ``Related Questions`` section could be more +straight forward. But this approach is even more flexible, since ``rows`` +does not have to be a simple ``arange``, but could be ``intersting_times``:: + + >>> interesting_times = np.array([0, 4, 8, 9, 10]) + >>> correct_sensors_at_it = correct_sensors[interesting_times, :] + >>> interesting_times_reshaped = interesting_times[:, np.newaxis] + >>> new_arr_it = arr[interesting_times_reshaped, correct_sensors_at_it] + +Truly complex situation would arise now if you would for example pool ``L`` +experiments into an array shaped ``(L, N, D)``. But for ``oindex`` this should +not result into surprises. ``vindex``, being more powerful, will quite +certainly create some confusion in this case but also cover pretty much all +eventualities. + + +Copyright +--------- + +This document is placed under the CC0 1.0 Universell (CC0 1.0) Public Domain Dedication [1]_. + + +References and Footnotes +------------------------ + +.. [1] To the extent possible under law, the person who associated CC0 + with this work has waived all copyright and related or neighboring + rights to this work. The CC0 license may be found at + https://creativecommons.org/publicdomain/zero/1.0/ +.. [2] e.g., see NEP 18, + http://www.numpy.org/neps/nep-0018-array-function-protocol.html diff --git a/doc/neps/nep-template.rst b/doc/neps/nep-template.rst index 26515127d..e869ebae3 100644 --- a/doc/neps/nep-template.rst +++ b/doc/neps/nep-template.rst @@ -64,7 +64,7 @@ References and Footnotes .. [1] Each NEP must either be explicitly labeled as placed in the public domain (see this NEP as an example) or licensed under the `Open Publication License`_. -.. _Open Publication License: http://www.opencontent.org/openpub/ +.. _Open Publication License: https://www.opencontent.org/openpub/ Copyright diff --git a/doc/release/1.15.0-notes.rst b/doc/release/1.15.0-notes.rst index cc193530c..8961de300 100644 --- a/doc/release/1.15.0-notes.rst +++ b/doc/release/1.15.0-notes.rst @@ -74,8 +74,10 @@ Deprecations * Users of ``nditer`` should use the nditer object as a context manager anytime one of the iterator operands is writeable, so that numpy can manage writeback semantics, or should call ``it.close()``. A - `RuntimeWarning` will be emitted otherwise in these cases. Users of the C-API - should call ``NpyIter_Close`` before ``NpyIter_Deallocate``. + `RuntimeWarning` may be emitted otherwise in these cases. + +* The ``normed`` argument of ``np.histogram``, deprecated long ago in 1.6.0, + now emits a ``DeprecationWarning``. Future Changes @@ -167,9 +169,6 @@ in the user guide. C API changes ============= -* ``NpyIter_Close`` has been added and should be called before - ``NpyIter_Deallocate`` to resolve possible writeback-enabled arrays. - * Functions ``npy_get_floatstatus_barrier`` and ``npy_clear_floatstatus_barrier`` have been added and should be used in place of the ``npy_get_floatstatus``and ``npy_clear_status`` functions. Optimizing compilers like GCC 8.1 and Clang diff --git a/doc/release/1.16.0-notes.rst b/doc/release/1.16.0-notes.rst index 701ef1f6e..8df763b56 100644 --- a/doc/release/1.16.0-notes.rst +++ b/doc/release/1.16.0-notes.rst @@ -34,6 +34,22 @@ New Features Improvements ============ +``randint`` and ``choice`` now work on empty distributions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Even when no elements needed to be drawn, ``np.random.randint`` and +``np.random.choice`` raised an error when the arguments described an empty +distribution. This has been fixed so that e.g. +``np.random.choice([], 0) == np.array([], dtype=float64)``. Changes ======= + +Comparison ufuncs will now error rather than return NotImplemented +------------------------------------------------------------------ + +Previously, comparison ufuncs such as ``np.equal`` would return +`NotImplemented` if their arguments had structured dtypes, to help comparison +operators such as ``__eq__`` deal with those. This is no longer needed, as the +relevant logic has moved to the comparison operators proper (which thus do +continue to return `NotImplemented` as needed). Hence, like all other ufuncs, +the comparison ufuncs will now error on structured dtypes. diff --git a/doc/release/1.3.0-notes.rst b/doc/release/1.3.0-notes.rst index 3ec93e0b0..239714246 100644 --- a/doc/release/1.3.0-notes.rst +++ b/doc/release/1.3.0-notes.rst @@ -14,7 +14,7 @@ Python 2.6 support Python 2.6 is now supported on all previously supported platforms, including windows. -http://www.python.org/dev/peps/pep-0361/ +https://www.python.org/dev/peps/pep-0361/ Generalized ufuncs ------------------ @@ -235,7 +235,7 @@ This should make the porting to new platforms easier, and more robust. In particular, the configuration stage does not need to execute any code on the target platform, which is a first step toward cross-compilation. -http://numpy.github.io/neps/math_config_clean.html +https://www.numpy.org/neps/nep-0003-math_config_clean.html umath refactor -------------- @@ -247,7 +247,7 @@ Improvements to build warnings Numpy can now build with -W -Wall without warnings -http://numpy.github.io/neps/warnfix.html +https://www.numpy.org/neps/nep-0002-warnfix.html Separate core math library -------------------------- diff --git a/doc/release/1.7.0-notes.rst b/doc/release/1.7.0-notes.rst index 72aab4d4f..09e6924e6 100644 --- a/doc/release/1.7.0-notes.rst +++ b/doc/release/1.7.0-notes.rst @@ -101,7 +101,7 @@ to NumPy 1.6: The notes in `doc/source/reference/arrays.datetime.rst <https://github.com/numpy/numpy/blob/maintenance/1.7.x/doc/source/reference/arrays.datetime.rst>`_ (also available in the online docs at `arrays.datetime.html -<http://docs.scipy.org/doc/numpy/reference/arrays.datetime.html>`_) should be +<https://docs.scipy.org/doc/numpy/reference/arrays.datetime.html>`_) should be consulted for more details. Custom formatter for printing arrays diff --git a/doc/release/time_based_proposal.rst b/doc/release/time_based_proposal.rst index 555be6863..2eb13562d 100644 --- a/doc/release/time_based_proposal.rst +++ b/doc/release/time_based_proposal.rst @@ -123,7 +123,7 @@ References * Proposed schedule for Gnome from Havoc Pennington (one of the core GTK and Gnome manager): - http://mail.gnome.org/archives/gnome-hackers/2002-June/msg00041.html + https://mail.gnome.org/archives/gnome-hackers/2002-June/msg00041.html The proposed schedule is heavily based on this email - * http://live.gnome.org/ReleasePlanning/Freezes + * https://wiki.gnome.org/ReleasePlanning/Freezes diff --git a/doc/source/_templates/indexcontent.html b/doc/source/_templates/indexcontent.html index fbd8930ae..008eaaa7c 100644 --- a/doc/source/_templates/indexcontent.html +++ b/doc/source/_templates/indexcontent.html @@ -36,7 +36,7 @@ <td width="50%"> <p class="biglink"><a class="biglink" href="{{ pathto("bugs") }}">Reporting bugs</a></p> <p class="biglink"><a class="biglink" href="{{ pathto("about") }}">About NumPy</a></p> - <p class="biglink"><a class="biglink" href="http://www.numpy.org/neps/index.html"> + <p class="biglink"><a class="biglink" href="https://www.numpy.org/neps/index.html"> NumPy Enhancement Proposals</a><br/> </td><td width="50%"> <p class="biglink"><a class="biglink" href="{{ pathto("release") }}">Release Notes</a></p> diff --git a/doc/source/_templates/indexsidebar.html b/doc/source/_templates/indexsidebar.html index 9edb003af..51e7c4308 100644 --- a/doc/source/_templates/indexsidebar.html +++ b/doc/source/_templates/indexsidebar.html @@ -1,4 +1,4 @@ <h3>Resources</h3> <ul> - <li><a href="http://scipy.org/">Scipy.org website</a></li> + <li><a href="https://scipy.org/">Scipy.org website</a></li> </ul> diff --git a/doc/source/about.rst b/doc/source/about.rst index 0614c90af..5ac4facbb 100644 --- a/doc/source/about.rst +++ b/doc/source/about.rst @@ -30,13 +30,13 @@ even better, contact us and participate in fixing the problem. Our main means of communication are: -- `scipy.org website <http://scipy.org/>`__ +- `scipy.org website <https://scipy.org/>`__ -- `Mailing lists <http://scipy.org/Mailing_Lists>`__ +- `Mailing lists <https://scipy.org/scipylib/mailing-lists.html>`__ - `NumPy Issues <https://github.com/numpy/numpy/issues>`__ (bug reports go here) -- `Old NumPy Trac <http://projects.scipy.org/numpy>`__ (no longer used) +- `Old NumPy Trac <http://projects.scipy.org/numpy>`__ (dead link) More information about the development of NumPy can be found at our `Developer Zone <https://scipy.scipy.org/scipylib/dev-zone.html>`__. diff --git a/doc/source/bugs.rst b/doc/source/bugs.rst index 950934b14..304a4136a 100644 --- a/doc/source/bugs.rst +++ b/doc/source/bugs.rst @@ -5,7 +5,7 @@ Reporting bugs File bug reports or feature requests, and make contributions (e.g. code patches), by opening a "new issue" on GitHub: -- NumPy Issues: http://github.com/numpy/numpy/issues +- NumPy Issues: https://github.com/numpy/numpy/issues Please give as much information as you can in the ticket. It is extremely useful if you can supply a small self-contained code snippet that reproduces @@ -15,5 +15,5 @@ the milestone. Report bugs to the appropriate GitHub project (there is one for NumPy and a different one for SciPy). -More information can be found on the http://scipy.org/Developer_Zone -website. +More information can be found on the +https://www.scipy.org/scipylib/dev-zone.html website. diff --git a/doc/source/conf.py b/doc/source/conf.py index 1472f5155..93a5ea01e 100644 --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -104,8 +104,8 @@ if 'scipyorg' in tags: "edit_link": True, "sidebar": "right", "scipy_org_logo": True, - "rootlinks": [("http://scipy.org/", "Scipy.org"), - ("http://docs.scipy.org/", "Docs")] + "rootlinks": [("https://scipy.org/", "Scipy.org"), + ("https://docs.scipy.org/", "Docs")] } else: # Default build @@ -341,8 +341,8 @@ def linkcode_resolve(domain, info): fn = relpath(fn, start=dirname(numpy.__file__)) if 'dev' in numpy.__version__: - return "http://github.com/numpy/numpy/blob/master/numpy/%s%s" % ( + return "https://github.com/numpy/numpy/blob/master/numpy/%s%s" % ( fn, linespec) else: - return "http://github.com/numpy/numpy/blob/v%s/numpy/%s%s" % ( + return "https://github.com/numpy/numpy/blob/v%s/numpy/%s%s" % ( numpy.__version__, fn, linespec) diff --git a/doc/source/dev/gitwash/development_workflow.rst b/doc/source/dev/gitwash/development_workflow.rst index c6884a7cf..9561e25f7 100644 --- a/doc/source/dev/gitwash/development_workflow.rst +++ b/doc/source/dev/gitwash/development_workflow.rst @@ -374,7 +374,7 @@ Deleting a branch on github_ git push origin :my-unwanted-branch (Note the colon ``:`` before ``test-branch``. See also: -http://github.com/guides/remove-a-remote-branch +https://github.com/guides/remove-a-remote-branch Several people sharing a single repository @@ -387,7 +387,7 @@ share it via github_. First fork NumPy into your account, as from :ref:`forking`. Then, go to your forked repository github page, say -``http://github.com/your-user-name/numpy`` +``https://github.com/your-user-name/numpy`` Click on the 'Admin' button, and add anyone else to the repo as a collaborator: diff --git a/doc/source/dev/gitwash/git_links.inc b/doc/source/dev/gitwash/git_links.inc index 30532da99..cebbb3a67 100644 --- a/doc/source/dev/gitwash/git_links.inc +++ b/doc/source/dev/gitwash/git_links.inc @@ -9,57 +9,57 @@ nipy, NIPY, Nipy, etc... .. git stuff -.. _git: http://git-scm.com/ -.. _github: http://github.com -.. _github help: http://help.github.com -.. _msysgit: http://code.google.com/p/msysgit/downloads/list -.. _git-osx-installer: http://code.google.com/p/git-osx-installer/downloads/list +.. _git: https://git-scm.com/ +.. _github: https://github.com +.. _github help: https://help.github.com +.. _msysgit: https://code.google.com/p/msysgit/downloads/list +.. _git-osx-installer: https://code.google.com/p/git-osx-installer/downloads/list .. _subversion: http://subversion.tigris.org/ -.. _git cheat sheet: http://github.com/guides/git-cheat-sheet -.. _pro git book: http://progit.org/ -.. _git svn crash course: http://git-scm.com/course/svn.html -.. _learn.github: http://learn.github.com/ -.. _network graph visualizer: http://github.com/blog/39-say-hello-to-the-network-graph-visualizer -.. _git user manual: http://www.kernel.org/pub/software/scm/git/docs/user-manual.html -.. _git tutorial: http://www.kernel.org/pub/software/scm/git/docs/gittutorial.html -.. _git community book: http://book.git-scm.com/ +.. _git cheat sheet: http://cheat.errtheblog.com/s/git +.. _pro git book: https://git-scm.com/book/ +.. _git svn crash course: https://git-scm.com/course/svn.html +.. _learn.github: https://learn.github.com/ +.. _network graph visualizer: https://github.com/blog/39-say-hello-to-the-network-graph-visualizer +.. _git user manual: https://www.kernel.org/pub/software/scm/git/docs/user-manual.html +.. _git tutorial: https://www.kernel.org/pub/software/scm/git/docs/gittutorial.html +.. _git community book: https://book.git-scm.com/ .. _git ready: http://www.gitready.com/ .. _git casts: http://www.gitcasts.com/ .. _Fernando's git page: http://www.fperez.org/py4science/git.html .. _git magic: http://www-cs-students.stanford.edu/~blynn/gitmagic/index.html .. _git concepts: http://www.eecs.harvard.edu/~cduan/technical/git/ -.. _git clone: http://www.kernel.org/pub/software/scm/git/docs/git-clone.html -.. _git checkout: http://www.kernel.org/pub/software/scm/git/docs/git-checkout.html -.. _git commit: http://www.kernel.org/pub/software/scm/git/docs/git-commit.html -.. _git push: http://www.kernel.org/pub/software/scm/git/docs/git-push.html -.. _git pull: http://www.kernel.org/pub/software/scm/git/docs/git-pull.html -.. _git add: http://www.kernel.org/pub/software/scm/git/docs/git-add.html -.. _git status: http://www.kernel.org/pub/software/scm/git/docs/git-status.html -.. _git diff: http://www.kernel.org/pub/software/scm/git/docs/git-diff.html -.. _git log: http://www.kernel.org/pub/software/scm/git/docs/git-log.html -.. _git branch: http://www.kernel.org/pub/software/scm/git/docs/git-branch.html -.. _git remote: http://www.kernel.org/pub/software/scm/git/docs/git-remote.html -.. _git config: http://www.kernel.org/pub/software/scm/git/docs/git-config.html +.. _git clone: https://www.kernel.org/pub/software/scm/git/docs/git-clone.html +.. _git checkout: https://www.kernel.org/pub/software/scm/git/docs/git-checkout.html +.. _git commit: https://www.kernel.org/pub/software/scm/git/docs/git-commit.html +.. _git push: https://www.kernel.org/pub/software/scm/git/docs/git-push.html +.. _git pull: https://www.kernel.org/pub/software/scm/git/docs/git-pull.html +.. _git add: https://www.kernel.org/pub/software/scm/git/docs/git-add.html +.. _git status: https://www.kernel.org/pub/software/scm/git/docs/git-status.html +.. _git diff: https://www.kernel.org/pub/software/scm/git/docs/git-diff.html +.. _git log: https://www.kernel.org/pub/software/scm/git/docs/git-log.html +.. _git branch: https://www.kernel.org/pub/software/scm/git/docs/git-branch.html +.. _git remote: https://www.kernel.org/pub/software/scm/git/docs/git-remote.html +.. _git config: https://www.kernel.org/pub/software/scm/git/docs/git-config.html .. _why the -a flag?: http://www.gitready.com/beginner/2009/01/18/the-staging-area.html .. _git staging area: http://www.gitready.com/beginner/2009/01/18/the-staging-area.html -.. _tangled working copy problem: http://tomayko.com/writings/the-thing-about-git +.. _tangled working copy problem: https://tomayko.com/writings/the-thing-about-git .. _git management: http://kerneltrap.org/Linux/Git_Management -.. _linux git workflow: http://www.mail-archive.com/dri-devel@lists.sourceforge.net/msg39091.html +.. _linux git workflow: https://www.mail-archive.com/dri-devel@lists.sourceforge.net/msg39091.html .. _ipython git workflow: http://mail.python.org/pipermail/ipython-dev/2010-October/006746.html .. _git parable: http://tom.preston-werner.com/2009/05/19/the-git-parable.html .. _git foundation: http://matthew-brett.github.com/pydagogue/foundation.html .. _numpy/master: https://github.com/numpy/numpy .. _git cherry-pick: https://www.kernel.org/pub/software/scm/git/docs/git-cherry-pick.html .. _git blame: https://www.kernel.org/pub/software/scm/git/docs/git-blame.html -.. _this blog post: http://github.com/blog/612-introducing-github-compare-view -.. _this article on merging conflicts: http://git-scm.com/book/en/Git-Branching-Basic-Branching-and-Merging#Basic-Merge-Conflicts +.. _this blog post: https://github.com/blog/612-introducing-github-compare-view +.. _this article on merging conflicts: https://git-scm.com/book/en/Git-Branching-Basic-Branching-and-Merging#Basic-Merge-Conflicts .. _learn git: https://www.atlassian.com/git/tutorials/ .. _filing pull requests: https://help.github.com/articles/using-pull-requests/#initiating-the-pull-request .. _pull request review: https://help.github.com/articles/using-pull-requests/#reviewing-the-pull-request .. other stuff -.. _python: http://www.python.org -.. _NumPy: http://www.numpy.org -.. _`NumPy github`: http://github.com/numpy/numpy -.. _`NumPy mailing list`: http://scipy.org/Mailing_Lists +.. _python: https://www.python.org +.. _NumPy: https://www.numpy.org +.. _`NumPy github`: https://github.com/numpy/numpy +.. _`NumPy mailing list`: https://scipy.org/scipylib/mailing-lists.html diff --git a/doc/source/dev/gitwash_links.txt b/doc/source/dev/gitwash_links.txt index f9536828c..36ca0b65f 100644 --- a/doc/source/dev/gitwash_links.txt +++ b/doc/source/dev/gitwash_links.txt @@ -1,3 +1,3 @@ -.. _NumPy: http://www.numpy.org -.. _`NumPy github`: http://github.com/numpy/numpy -.. _`NumPy mailing list`: http://scipy.org/Mailing_Lists +.. _NumPy: https://www.numpy.org +.. _`NumPy github`: https://github.com/numpy/numpy +.. _`NumPy mailing list`: https://scipy.org/scipylib/mailing-lists.html diff --git a/doc/source/dev/index.rst b/doc/source/dev/index.rst index 543194119..04c84eb61 100644 --- a/doc/source/dev/index.rst +++ b/doc/source/dev/index.rst @@ -7,6 +7,7 @@ Contributing to NumPy gitwash/index development_environment + releasing governance/index For core developers: see :ref:`development-workflow`. diff --git a/doc/source/dev/releasing.rst b/doc/source/dev/releasing.rst new file mode 100644 index 000000000..61fa19514 --- /dev/null +++ b/doc/source/dev/releasing.rst @@ -0,0 +1,16 @@ +=================== +Releasing a Version +=================== + +------------------------ +How to Prepare a Release +------------------------ + +.. include:: ../../HOWTO_RELEASE.rst.txt + +----------------------- +Step-by-Step Directions +----------------------- + +.. include:: ../../RELEASE_WALKTHROUGH.rst.txt + diff --git a/doc/source/docs/howto_build_docs.rst b/doc/source/docs/howto_build_docs.rst index 383bed96d..cdf490c37 100644 --- a/doc/source/docs/howto_build_docs.rst +++ b/doc/source/docs/howto_build_docs.rst @@ -10,11 +10,11 @@ documentation for NumPy. You will need Sphinx 1.0.1 or newer. If you only want to get the documentation, note that pre-built versions can be found at - http://docs.scipy.org/ + https://docs.scipy.org/ in several different formats. -.. _Sphinx: http://sphinx.pocoo.org +.. _Sphinx: http://www.sphinx-doc.org/ Instructions @@ -66,11 +66,11 @@ which will rebuild NumPy, install it to a temporary location, and build the documentation in all formats. This will most likely again only work on Unix platforms. -The documentation for NumPy distributed at http://docs.scipy.org in html and +The documentation for NumPy distributed at https://docs.scipy.org in html and pdf format is also built with ``make dist``. See `HOWTO RELEASE`_ for details on -how to update http://docs.scipy.org. +how to update https://docs.scipy.org. -.. _Matplotlib: http://matplotlib.org/ +.. _Matplotlib: https://matplotlib.org/ .. _HOWTO RELEASE: https://github.com/numpy/numpy/blob/master/doc/HOWTO_RELEASE.rst.txt Sphinx extensions @@ -83,5 +83,5 @@ above), and are automatically enabled when building NumPy's documentation. If you want to make use of these extensions in third-party projects, they are available on PyPi_ as the numpydoc_ package. -.. _PyPi: http://python.org/pypi -.. _numpydoc: http://python.org/pypi/numpydoc +.. _PyPi: https://pypi.org/ +.. _numpydoc: https://python.org/pypi/numpydoc diff --git a/doc/source/docs/howto_document.rst b/doc/source/docs/howto_document.rst index de7d06cf8..2a97a100d 100644 --- a/doc/source/docs/howto_document.rst +++ b/doc/source/docs/howto_document.rst @@ -4,7 +4,7 @@ A Guide to NumPy/SciPy Documentation ==================================== -When using `Sphinx <http://sphinx.pocoo.org/>`__ in combination with the +When using `Sphinx <http://www.sphinx-doc.org/>`__ in combination with the numpy conventions, you should use the ``numpydoc`` extension so that your docstrings will be handled correctly. For example, Sphinx will extract the ``Parameters`` section from your docstring and convert it into a field @@ -19,7 +19,7 @@ Some features described in this document require a recent version of It is available from: -* `numpydoc on PyPI <http://pypi.python.org/pypi/numpydoc>`_ +* `numpydoc on PyPI <https://pypi.python.org/pypi/numpydoc>`_ * `numpydoc on GitHub <https://github.com/numpy/numpydoc/>`_ Note that for documentation within numpy, it is not necessary to do diff --git a/doc/source/f2py/index.rst b/doc/source/f2py/index.rst index 8b7d1453a..d6773a76f 100644 --- a/doc/source/f2py/index.rst +++ b/doc/source/f2py/index.rst @@ -26,6 +26,5 @@ from Python. distutils advanced -.. _Python: http://www.python.org/ -.. _NumPy: http://www.numpy.org/ -.. _SciPy: http://www.numpy.org/ +.. _Python: https://www.python.org/ +.. _NumPy: https://www.numpy.org/ diff --git a/doc/source/f2py/signature-file.rst b/doc/source/f2py/signature-file.rst index bd926f33c..8e5a9710c 100644 --- a/doc/source/f2py/signature-file.rst +++ b/doc/source/f2py/signature-file.rst @@ -303,7 +303,7 @@ Other statements: ``pymethoddef`` statement can be used only inside ``python module`` block. - __ http://www.python.org/doc/current/ext/ext.html + __ https://docs.python.org/extending/index.html Attributes ------------ diff --git a/doc/source/reference/arrays.interface.rst b/doc/source/reference/arrays.interface.rst index 4a5fe62bf..f361ccb06 100644 --- a/doc/source/reference/arrays.interface.rst +++ b/doc/source/reference/arrays.interface.rst @@ -22,7 +22,7 @@ The Array Interface described here. __ http://cython.org/ -__ http://wiki.cython.org/tutorials/numpy +__ https://github.com/cython/cython/wiki/tutorials-numpy :version: 3 diff --git a/doc/source/reference/c-api.coremath.rst b/doc/source/reference/c-api.coremath.rst index ba1c61e8e..691f73287 100644 --- a/doc/source/reference/c-api.coremath.rst +++ b/doc/source/reference/c-api.coremath.rst @@ -297,10 +297,10 @@ External Links: * `OpenGL Half Float Pixel Support`__ * `The OpenEXR image format`__. -__ http://ieeexplore.ieee.org/servlet/opac?punumber=4610933 -__ http://en.wikipedia.org/wiki/Half_precision_floating-point_format -__ http://www.opengl.org/registry/specs/ARB/half_float_pixel.txt -__ http://www.openexr.com/about.html +__ https://ieeexplore.ieee.org/document/4610935/ +__ https://en.wikipedia.org/wiki/Half-precision_floating-point_format +__ https://www.khronos.org/registry/OpenGL/extensions/ARB/ARB_half_float_pixel.txt +__ https://www.openexr.com/about.html .. c:var:: NPY_HALF_ZERO diff --git a/doc/source/reference/c-api.iterator.rst b/doc/source/reference/c-api.iterator.rst index 392dcb730..940452d3c 100644 --- a/doc/source/reference/c-api.iterator.rst +++ b/doc/source/reference/c-api.iterator.rst @@ -110,7 +110,6 @@ number of non-zero elements in an array. /* Increment the iterator to the next inner loop */ } while(iternext(iter)); - NpyIter_Close(iter) /* best practice, not strictly required in this case */ NpyIter_Deallocate(iter); return nonzero_count; @@ -195,7 +194,6 @@ is used to control the memory layout of the allocated result, typically ret = NpyIter_GetOperandArray(iter)[1]; Py_INCREF(ret); - NpyIter_Close(iter); if (NpyIter_Deallocate(iter) != NPY_SUCCEED) { Py_DECREF(ret); return NULL; @@ -495,7 +493,7 @@ Construction and Destruction per operand. Using ``NPY_ITER_READWRITE`` or ``NPY_ITER_WRITEONLY`` for a user-provided operand may trigger `WRITEBACKIFCOPY`` semantics. The data will be written back to the original array - when ``NpyIter_Close`` is called. + when ``NpyIter_Deallocate`` is called. .. c:var:: NPY_ITER_COPY @@ -507,13 +505,13 @@ Construction and Destruction Triggers :c:data:`NPY_ITER_COPY`, and when an array operand is flagged for writing and is copied, causes the data - in a copy to be copied back to ``op[i]`` when ``NpyIter_Close`` is - called. + in a copy to be copied back to ``op[i]`` when + ``NpyIter_Deallocate`` is called. If the operand is flagged as write-only and a copy is needed, an uninitialized temporary array will be created and then copied - to back to ``op[i]`` on calling ``NpyIter_Close``, instead of doing - the unnecessary copy operation. + to back to ``op[i]`` on calling ``NpyIter_Deallocate``, instead of + doing the unnecessary copy operation. .. c:var:: NPY_ITER_NBO .. c:var:: NPY_ITER_ALIGNED @@ -709,9 +707,7 @@ Construction and Destruction the functions will pass back errors through it instead of setting a Python exception. - :c:func:`NpyIter_Deallocate` must be called for each copy. One call to - :c:func:`NpyIter_Close` is sufficient to trigger writeback resolution for - all copies since they share buffers. + :c:func:`NpyIter_Deallocate` must be called for each copy. .. c:function:: int NpyIter_RemoveAxis(NpyIter* iter, int axis)`` @@ -763,23 +759,9 @@ Construction and Destruction Returns ``NPY_SUCCEED`` or ``NPY_FAIL``. -.. c:function:: int NpyIter_Close(NpyIter* iter) - - Resolves any needed writeback resolution. Should be called before - :c:func::`NpyIter_Deallocate`. After this call it is not safe to use the operands. - When using :c:func:`NpyIter_Copy`, only one call to :c:func:`NpyIter_Close` - is sufficient to resolve any writebacks, since the copies share buffers. - - Returns ``0`` or ``-1`` if unsuccessful. - .. c:function:: int NpyIter_Deallocate(NpyIter* iter) - Deallocates the iterator object. - - :c:func:`NpyIter_Close` should be called before this. If not, and if - writeback is needed, it will be performed at this point in order to maintain - backward-compatibility with older code, and a deprecation warning will be - emitted. Old code should be updated to call `NpyIter_Close` beforehand. + Deallocates the iterator object and resolves any needed writebacks. Returns ``NPY_SUCCEED`` or ``NPY_FAIL``. diff --git a/doc/source/reference/routines.io.rst b/doc/source/reference/routines.io.rst index 55489951f..8bb29b793 100644 --- a/doc/source/reference/routines.io.rst +++ b/doc/source/reference/routines.io.rst @@ -63,6 +63,7 @@ Text formatting options set_printoptions get_printoptions set_string_function + printoptions Base-n representations ---------------------- diff --git a/doc/source/reference/routines.ma.rst b/doc/source/reference/routines.ma.rst index 2408899b3..15f2ba0a4 100644 --- a/doc/source/reference/routines.ma.rst +++ b/doc/source/reference/routines.ma.rst @@ -126,6 +126,7 @@ Changing the number of dimensions ma.MaskedArray.squeeze + ma.stack ma.column_stack ma.concatenate ma.dstack @@ -141,6 +142,7 @@ Joining arrays .. autosummary:: :toctree: generated/ + ma.stack ma.column_stack ma.concatenate ma.append diff --git a/doc/source/reference/routines.statistics.rst b/doc/source/reference/routines.statistics.rst index e287fe9c8..c675b6090 100644 --- a/doc/source/reference/routines.statistics.rst +++ b/doc/source/reference/routines.statistics.rst @@ -56,4 +56,5 @@ Histograms histogram2d histogramdd bincount + histogram_bin_edges digitize diff --git a/doc/source/user/building.rst b/doc/source/user/building.rst index 76eb48487..d224951dd 100644 --- a/doc/source/user/building.rst +++ b/doc/source/user/building.rst @@ -16,7 +16,7 @@ Building NumPy requires the following software installed: On Debian and derivatives (Ubuntu): python, python-dev (or python3-dev) On Windows: the official python installer at - `www.python.org <http://www.python.org>`_ is enough + `www.python.org <https://www.python.org>`_ is enough Make sure that the Python package distutils is installed before continuing. For example, in Debian GNU/Linux, installing python-dev diff --git a/doc/source/user/c-info.beyond-basics.rst b/doc/source/user/c-info.beyond-basics.rst index 5c321088d..aee68f6e7 100644 --- a/doc/source/user/c-info.beyond-basics.rst +++ b/doc/source/user/c-info.beyond-basics.rst @@ -481,7 +481,7 @@ type(s). In particular, to create a sub-type in C follow these steps: module dictionary so it can be accessed from Python. More information on creating sub-types in C can be learned by reading -PEP 253 (available at http://www.python.org/dev/peps/pep-0253). +PEP 253 (available at https://www.python.org/dev/peps/pep-0253). Specific features of ndarray sub-typing diff --git a/doc/source/user/c-info.how-to-extend.rst b/doc/source/user/c-info.how-to-extend.rst index 22c3b6e90..9738168d2 100644 --- a/doc/source/user/c-info.how-to-extend.rst +++ b/doc/source/user/c-info.how-to-extend.rst @@ -36,7 +36,7 @@ into Python as if it were a standard python file. It will contain objects and methods that have been defined and compiled in C code. The basic steps for doing this in Python are well-documented and you can find more information in the documentation for Python itself available -online at `www.python.org <http://www.python.org>`_ . +online at `www.python.org <https://www.python.org>`_ . In addition to the Python C-API, there is a full and rich C-API for NumPy allowing sophisticated manipulations on a C-level. However, for diff --git a/doc/source/user/c-info.python-as-glue.rst b/doc/source/user/c-info.python-as-glue.rst index 0152ac549..750fdddf0 100644 --- a/doc/source/user/c-info.python-as-glue.rst +++ b/doc/source/user/c-info.python-as-glue.rst @@ -405,8 +405,8 @@ interface between Python and Fortran. There is decent documentation for f2py found in the numpy/f2py/docs directory where-ever NumPy is installed on your system (usually under site-packages). There is also more information on using f2py (including how to use it to wrap C -codes) at http://www.scipy.org/Cookbook under the "Using NumPy with -Other Languages" heading. +codes) at https://scipy-cookbook.readthedocs.io under the "Interfacing +With Other Languages" heading. The f2py method of linking compiled code is currently the most sophisticated and integrated approach. It allows clean separation of diff --git a/doc/source/user/c-info.ufunc-tutorial.rst b/doc/source/user/c-info.ufunc-tutorial.rst index 5818ff182..788a3429f 100644 --- a/doc/source/user/c-info.ufunc-tutorial.rst +++ b/doc/source/user/c-info.ufunc-tutorial.rst @@ -17,7 +17,7 @@ Creating a new universal function Before reading this, it may help to familiarize yourself with the basics of C extensions for Python by reading/skimming the tutorials in Section 1 of `Extending and Embedding the Python Interpreter -<http://docs.python.org/extending/index.html>`_ and in :doc:`How to extend +<https://docs.python.org/extending/index.html>`_ and in :doc:`How to extend NumPy <c-info.how-to-extend>` The umath module is a computer-generated C-module that creates many diff --git a/doc/source/user/install.rst b/doc/source/user/install.rst index dd7543645..52586f3d7 100644 --- a/doc/source/user/install.rst +++ b/doc/source/user/install.rst @@ -4,7 +4,7 @@ Installing NumPy In most use cases the best way to install NumPy on your system is by using a pre-built package for your operating system. Please see -http://scipy.org/install.html for links to available options. +https://scipy.org/install.html for links to available options. For instructions on building for source package, see :doc:`building`. This information is useful mainly for advanced users. diff --git a/doc/source/user/numpy-for-matlab-users.rst b/doc/source/user/numpy-for-matlab-users.rst index 475c68c04..399237c21 100644 --- a/doc/source/user/numpy-for-matlab-users.rst +++ b/doc/source/user/numpy-for-matlab-users.rst @@ -618,9 +618,9 @@ initial element of a sequence has index 0. Confusion and flamewars arise because each has advantages and disadvantages. One based indexing is consistent with common human language usage, where the "first" element of a sequence has index 1. Zero based indexing `simplifies -indexing <http://groups.google.com/group/comp.lang.python/msg/1bf4d925dfbf368?q=g:thl3498076713d&hl=en>`__. +indexing <https://groups.google.com/group/comp.lang.python/msg/1bf4d925dfbf368?q=g:thl3498076713d&hl=en>`__. See also `a text by prof.dr. Edsger W. -Dijkstra <http://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html>`__. +Dijkstra <https://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html>`__. \ **RANGES**: In MATLAB®, ``0:5`` can be used as both a range literal and a 'slice' index (inside parentheses); however, in Python, constructs @@ -715,6 +715,6 @@ See http://mathesaurus.sf.net/ for another MATLAB®/NumPy cross-reference. An extensive list of tools for scientific work with python can be -found in the `topical software page <http://scipy.org/topical-software.html>`__. +found in the `topical software page <https://scipy.org/topical-software.html>`__. MATLAB® and SimuLink® are registered trademarks of The MathWorks. diff --git a/doc/source/user/quickstart.rst b/doc/source/user/quickstart.rst index 57a7004cc..5ef8b145f 100644 --- a/doc/source/user/quickstart.rst +++ b/doc/source/user/quickstart.rst @@ -14,11 +14,11 @@ Prerequisites Before reading this tutorial you should know a bit of Python. If you would like to refresh your memory, take a look at the `Python -tutorial <http://docs.python.org/tut/>`__. +tutorial <https://docs.python.org/tutorial/>`__. If you wish to work the examples in this tutorial, you must also have some software installed on your computer. Please see -http://scipy.org/install.html for instructions. +https://scipy.org/install.html for instructions. The Basics ========== @@ -569,7 +569,7 @@ first axis:: However, if one wants to perform an operation on each element in the array, one can use the ``flat`` attribute which is an -`iterator <https://docs.python.org/2/tutorial/classes.html#iterators>`__ +`iterator <https://docs.python.org/tutorial/classes.html#iterators>`__ over all the elements of the array:: >>> for element in b.flat: @@ -1191,7 +1191,7 @@ This property can be very useful in assignments:: You can look at the following example to see how to use boolean indexing to generate an image of the `Mandelbrot -set <http://en.wikipedia.org/wiki/Mandelbrot_set>`__: +set <https://en.wikipedia.org/wiki/Mandelbrot_set>`__: .. plot:: @@ -1462,8 +1462,8 @@ that ``pylab.hist`` plots the histogram automatically, while Further reading =============== -- The `Python tutorial <http://docs.python.org/tutorial/>`__ +- The `Python tutorial <https://docs.python.org/tutorial/>`__ - :ref:`reference` - `SciPy Tutorial <https://docs.scipy.org/doc/scipy/reference/tutorial/index.html>`__ -- `SciPy Lecture Notes <http://www.scipy-lectures.org>`__ +- `SciPy Lecture Notes <https://www.scipy-lectures.org>`__ - A `matlab, R, IDL, NumPy/SciPy dictionary <http://mathesaurus.sf.net/>`__ |