diff options
| -rw-r--r-- | .gitignore | 1 | ||||
| -rw-r--r-- | doc/HOWTO_RELEASE.rst | 283 | ||||
| -rw-r--r-- | doc/Makefile | 85 | ||||
| -rw-r--r-- | doc/RELEASE_WALKTHROUGH.rst | 66 |
4 files changed, 90 insertions, 345 deletions
diff --git a/.gitignore b/.gitignore index cb234664c..e2ee6a013 100644 --- a/.gitignore +++ b/.gitignore @@ -68,6 +68,7 @@ _build # setup.py dist directory dist doc/build +doc/docenv doc/cdoc/build # Egg metadata *.egg-info diff --git a/doc/HOWTO_RELEASE.rst b/doc/HOWTO_RELEASE.rst index da1ed5235..830e86791 100644 --- a/doc/HOWTO_RELEASE.rst +++ b/doc/HOWTO_RELEASE.rst @@ -69,9 +69,10 @@ Tool chain ========== We build all our wheels on cloud infrastructure - so this list of compilers is for information and debugging builds locally. See the ``.travis.yml`` script -in the `numpy wheels`_ repo for the definitive source -of the build recipes. Packages that are available using pip are noted. +in the `numpy wheels`_ repo for an outdated source of the build recipes using +multibuild. +.. _numpy wheels : https://github.com/MacPython/numpy-wheels Compilers --------- @@ -82,7 +83,7 @@ each platform. At the moment this means: for OSX >= 10.6 can be safely built from the travis-ci OSX 10.9 VMs when building against the Python from the Python.org installers; - Windows builds use the `mingw-w64 toolchain`_; -- Manylinux1 wheels use the gcc provided on the Manylinux docker images. +- Manylinux2014 wheels use the gcc provided on the Manylinux docker images. You will need Cython for building the binaries. Cython compiles the ``.pyx`` files in the NumPy distribution to ``.c`` files. @@ -101,35 +102,30 @@ collisions with other OpenBLAS shared objects that may exist in the filesystem. Building source archives and wheels ----------------------------------- -You will need write permission for numpy-wheels in order to trigger wheel -builds. - -- Python(s) from `python.org <https://python.org>`_ or linux distro. -- cython (pip) -- virtualenv (pip) -- Paver (pip) -- pandoc `pandoc.org <https://www.pandoc.org>`_ or linux distro. -- numpy-wheels `<https://github.com/MacPython/numpy-wheels>`_ (clone) +The NumPy wheels and sdist are now built using cibuildwheel with +github actions. Building docs ------------- -Building the documents requires a number of latex ``.sty`` files. Install them -all to avoid aggravation. +We are no longer building ``PDF`` files. All that will be needed is + +- virtualenv (pip). -- Sphinx (pip) -- numpydoc (pip) -- Matplotlib -- Texlive (or MikTeX on Windows) +The other requirements will be filled automatically during the documentation +build process. Uploading to PyPI ----------------- -- terryfy `<https://github.com/MacPython/terryfy>`_ (clone). -- beautifulsoup4 (pip) -- delocate (pip) -- auditwheel (pip) -- twine (pip) +The only application needed for uploading is + +- twine (pip). + +You will also need a PyPI token, which is best kept on a keyring. See the +twine keyring_ documentation for how to do that. + +.. _keyring: https://twine.readthedocs.io/en/stable/#keyring-support Generating author/pr lists @@ -145,7 +141,7 @@ so that scripts can access the github NumPy repository. Virtualenv ---------- Virtualenv is a very useful tool to keep several versions of packages around. -It is also used in the Paver script to build the docs. +It is also used when building the documentation. What is released @@ -155,13 +151,9 @@ Wheels ------ We currently support Python 3.8-3.10 on Windows, OSX, and Linux -* Windows: 32-bit and 64-bit wheels built using Appveyor; -* OSX: x64_86 OSX wheels built using travis-ci; -* Linux: 32-bit and 64-bit Manylinux1 wheels built using travis-ci. - -See the `numpy wheels`_ building repository for more detail. - -.. _numpy wheels : https://github.com/MacPython/numpy-wheels +* Windows: 32-bit and 64-bit wheels built using Github actions; +* OSX: x64_86 and arm64 OSX wheels built using Github actions; +* Linux: x64_86 and aarch64 Manylinux2014 wheels built using Github actions. Other @@ -172,7 +164,7 @@ Other Source distribution ------------------- -We build source releases in both .zip and .tar.gz formats. +We build source releases in the .tar.gz format. Release process @@ -190,14 +182,16 @@ Milestones. Make sure current branch builds a package correctly --------------------------------------------------- -:: +The CI builds wheels when a PR header begins with ``REL``. Your last +PR before releasing should be so marked and all the tests should pass. +You can also do:: - git clean -fxd + git clean -fxdq python setup.py bdist_wheel python setup.py sdist -For details of the build process itself, it is -best to read the pavement.py script. +For details of the build process itself, it is best to read the +Step-by-Step Directions below. .. note:: The following steps are repeated for the beta(s), release candidates(s) and the final release. @@ -272,220 +266,3 @@ following: - outlook for the near future .. _towncrier: https://pypi.org/project/towncrier/ - - -Update the release status and create a release "tag" ----------------------------------------------------- -Identify the commit hash of the release, e.g. 1b2e1d63ff:: - - git co 1b2e1d63ff # gives warning about detached head - -First, change/check the following variables in ``pavement.py`` depending on the -release version:: - - RELEASE_NOTES = 'doc/release/1.7.0-notes.rst' - LOG_START = 'v1.6.0' - LOG_END = 'maintenance/1.7.x' - -Do any other changes. When you are ready to release, do the following -changes:: - - diff --git a/setup.py b/setup.py - index b1f53e3..8b36dbe 100755 - --- a/setup.py - +++ b/setup.py - @@ -57,7 +57,7 @@ PLATFORMS = ["Windows", "Linux", "Solaris", "Mac OS- - MAJOR = 1 - MINOR = 7 - MICRO = 0 - -ISRELEASED = False - +ISRELEASED = True - VERSION = '%d.%d.%drc1' % (MAJOR, MINOR, MICRO) - - # Return the git revision as a string - -And make sure the ``VERSION`` variable is set properly. - -Now you can make the release commit and tag. We recommend you don't push -the commit or tag immediately, just in case you need to do more cleanup. We -prefer to defer the push of the tag until we're confident this is the exact -form of the released code (see: :ref:`push-tag-and-commit`): - - git commit -s -m "REL: Release." setup.py - git tag -s <version> - -The ``-s`` flag makes a PGP (usually GPG) signed tag. Please do sign the -release tags. - -The release tag should have the release number in the annotation (tag -message). Unfortunately, the name of a tag can be changed without breaking the -signature, the contents of the message cannot. - -See: https://github.com/scipy/scipy/issues/4919 for a discussion of signing -release tags, and https://keyring.debian.org/creating-key.html for instructions -on creating a GPG key if you do not have one. - -To make your key more readily identifiable as you, consider sending your key -to public keyservers, with a command such as:: - - gpg --send-keys <yourkeyid> - - -Update the version of the main branch -------------------------------------- -Increment the release number in setup.py. Release candidates should have "rc1" -(or "rc2", "rcN") appended to the X.Y.Z format. - -Also create a new version hash in cversions.txt and a corresponding version -define NPY_x_y_API_VERSION in numpyconfig.h - - -Trigger the wheel builds ------------------------- -See the `numpy wheels`_ repository. - -In that repository edit the files: - -- ``azure/posix.yml`` -- ``azure/windows.yml``. - -In both cases, set the ``BUILD_COMMIT`` variable to the current release tag - -e.g. ``v1.19.0``:: - - $ gvim azure/posix.yml azure/windows.yml - $ git commit -a - $ git push upstream HEAD - -Make sure that the release tag has been pushed. - -Trigger a build by pushing a commit of your edits to the repository. Note that -you can do this on a branch, but it must be pushed upstream to the -``MacPython/numpy-wheels`` repository to trigger uploads since only -that repo has the appropriate tokens to allow uploads. - -The wheels, once built, appear at https://anaconda.org/multibuild-wheels-staging/numpy - -Make the release ----------------- -Build the changelog and notes for upload with:: - - paver write_release - - -Build and archive documentation -------------------------------- -Do:: - - cd doc/ - make dist - -to check that the documentation is in a buildable state. Then, after tagging, -create an archive of the documentation in the numpy/doc repo:: - - # This checks out github.com/numpy/doc and adds (``git add``) the - # documentation to the checked out repo. - make merge-doc - # Now edit the ``index.html`` file in the repo to reflect the new content. - # If the documentation is for a non-patch release (e.g. 1.19 -> 1.20), - # make sure to update the ``stable`` symlink to point to the new directory. - ln -sfn <latest_stable_directory> stable - # Commit the changes - git -C build/merge commit -am "Add documentation for <version>" - # Push to numpy/doc repo - git -C build/merge push - - -Update PyPI ------------ -The wheels and source should be uploaded to PyPI. - -You should upload the wheels first, and the source formats last, to make sure -that pip users don't accidentally get a source install when they were -expecting a binary wheel. - -You can do this automatically using the ``wheel-uploader`` script from -https://github.com/MacPython/terryfy. Here is the recommended incantation for -downloading all the Windows, Manylinux, OSX wheels and uploading to PyPI. :: - - NPY_WHLS=~/wheelhouse # local directory to cache wheel downloads - CDN_URL=https://anaconda.org/multibuild-wheels-staging/numpy/files - wheel-uploader -u $CDN_URL -w $NPY_WHLS -v -s -t win numpy 1.11.1rc1 - wheel-uploader -u $CDN_URL -w warehouse -v -s -t macosx numpy 1.11.1rc1 - wheel-uploader -u $CDN_URL -w warehouse -v -s -t manylinux1 numpy 1.11.1rc1 - -The ``-v`` flag gives verbose feedback, ``-s`` causes the script to sign the -wheels with your GPG key before upload. Don't forget to upload the wheels -before the source tarball, so there is no period for which people switch from -an expected binary install to a source install from PyPI. - -There are two ways to update the source release on PyPI, the first one is:: - - $ git clean -fxd # to be safe - $ python setup.py sdist --formats=gztar,zip # to check - # python setup.py sdist --formats=gztar,zip upload --sign - -This will ask for your key PGP passphrase, in order to sign the built source -packages. - -The second way is to upload the PKG_INFO file inside the sdist dir in the -web interface of PyPI. The source tarball can also be uploaded through this -interface. - -.. _push-tag-and-commit: - - -Push the release tag and commit -------------------------------- -Finally, now you are confident this tag correctly defines the source code that -you released you can push the tag and release commit up to github:: - - git push # Push release commit - git push upstream <version> # Push tag named <version> - -where ``upstream`` points to the main https://github.com/numpy/numpy.git -repository. - - -Update scipy.org ----------------- -A release announcement with a link to the download site should be placed in the -sidebar of the front page of scipy.org. - -The scipy.org should be a PR at https://github.com/scipy/scipy.org. The file -that needs modification is ``www/index.rst``. Search for ``News``. - - -Update oldest-supported-numpy ------------------------------ -If this release is the first one to support a new Python version, or the first -to provide wheels for a new platform or PyPy version, the version pinnings -in https://github.com/scipy/oldest-supported-numpy should be updated. -Either submit a PR with changes to ``setup.cfg`` there, or open an issue with -info on needed changes. - - -Announce to the lists ---------------------- -The release should be announced on the mailing lists of -NumPy and SciPy, to python-announce, and possibly also those of -Matplotlib, IPython and/or Pygame. - -During the beta/RC phase, an explicit request for testing the binaries with -several other libraries (SciPy/Matplotlib/Pygame) should be posted on the -mailing list. - - -Announce to Linux Weekly News ------------------------------ -Email the editor of LWN to let them know of the release. Directions at: -https://lwn.net/op/FAQ.lwn#contact - - -After the final release ------------------------ -After the final release is announced, a few administrative tasks are left to be -done: - - - Forward port changes in the release branch to release notes and release - scripts, if any, to main branch. - - Update the Milestones in Trac. diff --git a/doc/Makefile b/doc/Makefile index 97ff709ba..6d3c93203 100644 --- a/doc/Makefile +++ b/doc/Makefile @@ -26,26 +26,25 @@ FILES= PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter ALLSPHINXOPTS = -WT --keep-going -d build/doctrees $(PAPEROPT_$(PAPER)) \ - $(SPHINXOPTS) source + $(SPHINXOPTS) source -.PHONY: help clean html web pickle htmlhelp latex changes linkcheck \ - dist dist-build gitwash-update version-check html-build latex-build \ - merge-doc show +.PHONY: help clean html web htmlhelp latex changes linkcheck \ + dist dist-build gitwash-update version-check html-build latex-build \ + merge-doc show docenv #------------------------------------------------------------------------------ help: @echo "Please use \`make <target>' where <target> is one of" + @echo " clean to remove generated doc files and start fresh" + @echo " docenv make a virtual environment in which to build docs" @echo " html to make standalone HTML files" - @echo " html-scipyorg to make standalone HTML files with scipy.org theming" - @echo " pickle to make pickle files (usable by e.g. sphinx-web)" @echo " htmlhelp to make HTML files and a HTML help project" @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" @echo " changes to make an overview over all changed/added/deprecated items" @echo " linkcheck to check all external links for integrity" @echo " dist PYVER=... to make a distribution-ready tree" @echo " gitwash-update GITWASH=path/to/gitwash update gitwash developer docs" - @echo " upload USERNAME=... RELEASE=... to upload built docs to docs.scipy.org" @echo " merge-doc TAG=... to clone numpy/doc and archive documentation into it" @echo " show to show the html output in a browser" @@ -53,6 +52,16 @@ clean: -rm -rf build/* find . -name generated -type d -prune -exec rm -rf "{}" ";" +docenv: + $(PYTHON) -mvenv docenv + ( \ + . docenv/bin/activate; \ + pip install -q --upgrade pip; \ + pip install -q -r ../test_requirements.txt; \ + pip install -q -r ../doc_requirements.txt; \ + pip install -q ..; \ + ) + gitwash-update: rm -rf source/dev/gitwash install -d source/dev/gitwash @@ -69,17 +78,9 @@ gitwash-update: # We have to be careful of some issues: # # - Everything must be done using the same Python version -# - We must use eggs (otherwise they might override PYTHONPATH on import). -# - Different versions of easy_install install to different directories (!) # - -INSTALL_DIR = $(CURDIR)/build/inst-dist -INSTALL_PPH = $(INSTALL_DIR)/lib/python$(PYVER)/site-packages:$(INSTALL_DIR)/local/lib/python$(PYVER)/site-packages:$(INSTALL_DIR)/lib/python$(PYVER)/dist-packages:$(INSTALL_DIR)/local/lib/python$(PYVER)/dist-packages -UPLOAD_DIR=/srv/docs_scipy_org/doc/numpy-$(RELEASE) - -DIST_VARS=SPHINXBUILD="LANG=C PYTHONPATH=$(INSTALL_PPH) python$(PYVER) `which sphinx-build`" PYTHON="PYTHONPATH=$(INSTALL_PPH) python$(PYVER)" - +#SPHINXBUILD="LANG=C sphinx-build" NUMPYVER:=$(shell $(PYTHON) -c "import numpy; print(numpy.version.git_revision[:10])" 2>/dev/null) GITVER ?= $(shell cd ..; $(PYTHON) -c "import versioneer as v; print(v.get_versions()['full-revisionid'][:10])") @@ -102,45 +103,15 @@ endif dist: build/dist.tar.gz -build/dist.tar.gz: - make $(DIST_VARS) real-dist +build/dist.tar.gz: real-dist -real-dist: dist-build html-build - test -d build/latex || make latex-build - make -C build/latex all-pdf - -rm -rf build/dist +real-dist: html-build cp -r build/html build/dist cd build/html && zip -9r ../dist/numpy-html.zip . - cp build/latex/numpy-ref.pdf build/dist - cp build/latex/numpy-user.pdf build/dist cd build/dist && tar czf ../dist.tar.gz * chmod ug=rwX,o=rX -R build/dist find build/dist -type d -print0 | xargs -0r chmod g+s -dist-build: - rm -f ../dist/*.egg - cd .. && $(PYTHON) setup.py bdist_egg - install -d $(subst :, ,$(INSTALL_PPH)) - $(PYTHON) `which easy_install` --prefix=$(INSTALL_DIR) ../dist/*.egg - -upload: build/dist.tar.gz - # SSH must be correctly configured for this to work. - # Assumes that ``make dist`` was already run - # Example usage: ``make upload USERNAME=rgommers RELEASE=1.10.1`` - ssh $(USERNAME)@docs.scipy.org mkdir $(UPLOAD_DIR) - scp build/dist.tar.gz $(USERNAME)@docs.scipy.org:$(UPLOAD_DIR) - ssh $(USERNAME)@docs.scipy.org tar xvC $(UPLOAD_DIR) \ - -zf $(UPLOAD_DIR)/dist.tar.gz - ssh $(USERNAME)@docs.scipy.org mv $(UPLOAD_DIR)/numpy-ref.pdf \ - $(UPLOAD_DIR)/numpy-ref-$(RELEASE).pdf - ssh $(USERNAME)@docs.scipy.org mv $(UPLOAD_DIR)/numpy-user.pdf \ - $(UPLOAD_DIR)/numpy-user-$(RELEASE).pdf - ssh $(USERNAME)@docs.scipy.org mv $(UPLOAD_DIR)/numpy-html.zip \ - $(UPLOAD_DIR)/numpy-html-$(RELEASE).zip - ssh $(USERNAME)@docs.scipy.org rm $(UPLOAD_DIR)/dist.tar.gz - ssh $(USERNAME)@docs.scipy.org ln -snf numpy-$(RELEASE) /srv/docs_scipy_org/doc/numpy - - merge-doc: build/dist.tar.gz ifeq "$(TAG)" "" echo tag "$(TAG)" not of the form 1.18; @@ -189,22 +160,6 @@ endif @echo @echo "Build finished. The HTML pages are in build/html." -html-scipyorg: - mkdir -p build/html build/doctrees - $(SPHINXBUILD) -t scipyorg -b html $(ALLSPHINXOPTS) build/html-scipyorg $(FILES) - @echo - @echo "Build finished. The HTML pages are in build/html-scipyorg." - -pickle: generate version-check - mkdir -p build/pickle build/doctrees - $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) build/pickle $(FILES) - @echo - @echo "Build finished; now you can process the pickle files or run" - @echo " sphinx-web build/pickle" - @echo "to start the sphinx-web server." - -web: pickle - htmlhelp: generate version-check mkdir -p build/htmlhelp build/doctrees $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) build/htmlhelp $(FILES) @@ -248,7 +203,6 @@ linkcheck: generate version-check @echo @echo "Link check complete; look for any errors in the above output " \ "or in build/linkcheck/output.txt." - texinfo: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) build/texinfo @echo @@ -262,5 +216,6 @@ info: make -C build/texinfo info @echo "makeinfo finished; the Info files are in build/texinfo." + show: @python -c "import webbrowser; webbrowser.open_new_tab('file://$(PWD)/build/html/index.html')" diff --git a/doc/RELEASE_WALKTHROUGH.rst b/doc/RELEASE_WALKTHROUGH.rst index 139bf0d59..863b6dab3 100644 --- a/doc/RELEASE_WALKTHROUGH.rst +++ b/doc/RELEASE_WALKTHROUGH.rst @@ -115,9 +115,28 @@ repository:: $ git tag -a -s v1.21.0 -m"NumPy 1.21.0 release" $ git push upstream v1.21.0 +Build wheels via cibuildwheel (preferred) +----------------------------------------- +Tagging the build at the beginning of this process will trigger a wheel build +via cibuildwheel and upload wheels and an sdist to the staging area. The CI run +on github actions (for all x86-based and macOS arm64 wheels) takes about 1 1/4 +hours. The CI run on travis (for aarch64) takes less time. + +If you wish to manually trigger a wheel build, you can do so: + +- On github actions -> `Wheel builder`_ there is a "Run workflow" button, click + on it and choose the tag to build +- On travis_ there is a "More Options" button, click on it and choose a branch + to build. There does not appear to be an option to build a tag. + +.. _`Wheel builder`: https://github.com/numpy/numpy/actions/workflows/wheels.yml +.. _travis : https://app.travis-ci.com/github/numpy/numpy + +Build wheels with multibuild (outdated) +--------------------------------------- Build source releases ---------------------- +~~~~~~~~~~~~~~~~~~~~~ Paver is used to build the source releases. It will create the ``release`` and ``release/installers`` directories and put the ``*.zip`` and ``*.tar.gz`` @@ -127,7 +146,7 @@ source releases in the latter. :: Build wheels via MacPython/numpy-wheels ---------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Trigger the wheels build by pointing the numpy-wheels repository at this commit. This can take up to an hour. The numpy-wheels repository is cloned from @@ -163,23 +182,6 @@ Note that sometimes builds, like tests, fail for unrelated reasons and you will need to rerun them. You will need to be logged in under 'numpy' to do this on azure. -Build wheels via cibuildwheel ------------------------------ -Tagging the build at the beginning of this process will trigger a wheel build -via cibuildwheel and upload wheels and an sdist to the staging area. The CI run -on github actions (for all x86-based and macOS arm64 wheels) takes about 1 1/4 -hours. The CI run on travis (for aarch64) takes less time. - -If you wish to manually trigger a wheel build, you can do so: - -- On github actions -> `Wheel builder`_ there is a "Run workflow" button, click - on it and choose the tag to build -- On travis_ there is a "More Options" button, click on it and choose a branch - to build. There does not appear to be an option to build a tag. - -.. _`Wheel builder`: https://github.com/numpy/numpy/actions/workflows/wheels.yml -.. _travis : https://app.travis-ci.com/github/numpy/numpy - Download wheels --------------- @@ -266,14 +268,12 @@ Upload documents to numpy.org (skip for prereleases) This step is only needed for final releases and can be skipped for pre-releases and most patch releases. ``make merge-doc`` clones the ``numpy/doc`` repo into -``doc/build/merge`` and updates it with the new documentation. If you already -have a numpy installed, you need to locally install the new NumPy version so -that document generation will use the correct NumPy. This is because ``make -dist`` does not correctly set up the path. Note that Python 3.10 cannot be used -for generating the docs as it has no ``easy_install``, use 3.9 or 3.8 instead:: +``doc/build/merge`` and updates it with the new documentation:: + $ git clean -xdfq + $ git co v1.21.0 $ pushd doc - $ make dist + $ make docenv && source docenv/bin/activate $ make merge-doc $ pushd build/merge @@ -287,8 +287,9 @@ update the version marked `(stable)`:: $ gvim _static/versions.json -Otherwise, only the ``zip`` and ``pdf`` links should be updated with the -new tag name:: +Otherwise, only the ``zip`` link should be updated with the new tag name. Since +we are no longer generating ``pdf`` files, remove the line for the ``pdf`` +files if present:: $ gvim index.html +/'tag v1.21' @@ -307,6 +308,7 @@ Once everything seems satisfactory, update, commit and upload the changes:: $ python3 update.py $ git commit -a -m"Add documentation for v1.21.0" $ git push + $ deactivate $ popd $ popd @@ -360,3 +362,13 @@ Checkout main and forward port the documentation changes:: $ git push origin HEAD Go to GitHub and make a PR. + +Update oldest-supported-numpy +----------------------------- + +If this release is the first one to support a new Python version, or the first +to provide wheels for a new platform or PyPy version, the version pinnings +in https://github.com/scipy/oldest-supported-numpy should be updated. +Either submit a PR with changes to ``setup.cfg`` there, or open an issue with +info on needed changes. + |