path: root/doc/HOWTO_RELEASE.rst.txt

This file gives an overview of what is necessary to build binary releases for
NumPy on OS X. Windows binaries are built here using Wine, they can of course
also be built on Windows itself. Building OS X binaries on another platform is
not possible.

Current build and release info
==============================
The current info on building and releasing NumPy and SciPy is scattered in
several places. It should be summarized in one place, updated and where
necessary described in more detail. The sections below list all places where
useful info can be found.

Source tree
-----------
* INSTALL.txt
* release.sh
* pavement.py

NumPy Docs
----------
* https://github.com/numpy/numpy/blob/master/doc/HOWTO_RELEASE.rst.txt
* http://projects.scipy.org/numpy/wiki/MicrosoftToolchainSupport

SciPy.org wiki
--------------
* http://www.scipy.org/Installing_SciPy and links on that page.
* http://new.scipy.org/building/windows.html

Doc wiki
--------
* http://docs.scipy.org/numpy/docs/numpy-docs/user/install.rst/

Release Scripts
---------------
* https://github.com/certik/numpy-vendor


Supported platforms and versions
================================
Python 2.6-2.7 and >=3.2 are the currently supported versions on all platforms.

OS X
----
OS X versions >= 10.5 are supported.  Note that there are currently still
issues with compiling on 10.7, due to Apple moving to gcc-llvm.
Only the Python from `python.org <http://python.org>`_ is supported. Binaries
do *not* support Apple Python.

Windows
-------
Windows XP, Vista and 7 are supported.

Linux
-----
Many distributions include NumPy. Building from source is also relatively
straightforward. Only tarballs are created for Linux, no specific binary
installers are provided (yet).

BSD / Solaris
-------------
No binaries are provided, but succesful builds on Solaris and BSD have been
reported.


Tool chain
==========
Compilers
---------
The same gcc version is used as the one with which Python itself is built on
each platform. At the moment this means:

* OS X uses gcc-4.0 (since that is what Python itself is built with) up to
  Python 2.6. Python 2.7 comes in two flavors; the 32-bit version is built with
  gcc-4.0 and the 64-bit version with gcc-4.2.  The "release.sh" script
  sets environment variables to pick the right compiler.
  All binaries should be built on OS X 10.5, with the exception of the 64-bit
  Python 2.7 one which should be built on 10.6.
* Windows builds use MinGW 3.4.5.  Updating this to a more recent MinGW with
  GCC 4.x is desired, but there are still practical difficulties in building
  the binary installers.

Cython is not needed for building the binaries, because generated C files from
Cython sources are checked in at the moment. It is worth keeping an eye on what
Cython versions have been used to generate all current C files, it should be
the same and most recent version (0.16 as of now).

Fortran: on OS X gfortran from `this site <http://r.research.att.com/tools/>`_
is used. On Windows g77 (included in MinGW) is the current default, in the future
this may shift to gfortran as well.

Python
------
* Python(s) from `python.org <http://python.org>`_
* virtualenv
* paver
* bdist_mpkg from https://github.com/rgommers/bdist_mpkg (has a necessary
  patch, don't use the unsupported version on PyPi).

Python itself should be installed multiple times - each version a binary is
built for should be installed. The other dependencies only have to be installed
for the default Python version on the system. The same applies to the doc-build
dependencies below.

Building docs
-------------
* Sphinx
* numpydoc
* Matplotlib
* Texlive (or MikTeX on Windows)

Wine
----
For building Windows binaries on OS X Wine can be used. In Wine the following
needs to be installed:

* Python 2.5-2.7 and 3.1-3.2
* MakeNsis
* CpuId plugin for MakeNsis : this can be found in the NumPy source tree under
  tools/win32build/cpucaps and has to be built with MinGW (see SConstruct file in
  that dir for details)
* MinGW
* ATLAS, 3x ([No SSE, SSE2, SSE3] for superpack installer) : ATLAS does not
  compile under wine or on Windows out of the box. Binaries for ATLAS can be
  found in the vendor repository on GitHub (http://github.com/numpy/vendor).

To install Wine on OS X Snow Leopard the current options are to compile a
current unstable version ,`<http://wiki.winehq.org/MacOSX/Building>`_, or to use
an install script from `here <http://code.google.com/p/osxwinebuilder/>`_. For
me, the former option did not work (everything compiled, but after installing
Python the command ``import tempfile`` resulted in an exception. The latter
option did work.

After successful installation and an invocation of the wine executable, a
~/.wine folder exists - new programs will be installed there in
~/.wine/drive_c. Installing Windows programs with .exe executables is done by
running

  $ wine yourprog.exe

and MSI installers can be installed with

  $ msiexec /i yourprog.msi

For the above to work you probably need to put the wine-1.x.x/bin directory in
your PATH.

To install MinGW, the easiest option is to use the automated installer on the
MinGW download page. This will give you (at this moment) GCC 3.4.5; GCC 4.x is
still not supported officially by MinGW.

To be able to use gcc and MakeNsis in Wine, the locations of gcc.exe and
makensis.exe should be added to the Windows environment variable PATH. This can
easily be done by running

  $ wine regedit

add adding a PATH variable in HKEY_CURRENT_USER/Environment.

Virtualenv
----------
Virtualenv is a very useful tool to keep several versions of packages around.
It is also used in the Paver script to build the docs.


What is released
================

Binaries
--------
Windows binaries in "superpack" form for Python 2.6/2.7/3.2/3.3.
A superpack contains three builds, for SSE2, SSE3 and no SSE.

OS X binaries are made in dmg format, targeting only the Python from
`python.org <http://python.org>`_


Other
-----
* Release Notes
* Changelog

Source distribution
-------------------
A source release in both .zip and .tar.gz formats is released.


Release process
===============

Agree on a release schedule
---------------------------
A typical release schedule is one beta, two release candidates and a final
release.  It's best to discuss the timing on the mailing list first, in order
for people to get their commits in on time, get doc wiki edits merged, etc.
After a date is set, create a new maintenance/x.y.z branch, add new empty
release notes for the next version in the master branch and update the Trac
Milestones.

Handle test warnings
--------------------
The default behavior of the test suite in the master branch is to report errors
for DeprecationWarnings and RuntimeWarnings that are issued.  For a released
version this is not desired.  Therefore any known warnings should be solved or
explicitly silenced before making the release branch, then when the branch is
made, the default behavior should be switched to not raise errors.  This is
done in the constructor of the NoseTester class in numpy/testing/nosetester.py,
by replacing ``raise_warnings="develop"`` with ``raise_warnings="release"``.

Make sure current trunk builds a package correctly
--------------------------------------------------
::

    python setup.py bdist
    python setup.py sdist

To actually build the binaries after everything is set up correctly, the
release.sh script can be used. For details of the build process itself it is
best to read the pavement.py script.

.. note:: The following steps are repeated for the beta(s), release
   candidates(s) and the final release.

Merge doc wiki edits
--------------------
The edits in the documentation wiki suitable for merging should be merged,
ideally just before making the release branch. How to do this is described in
detail in doc/HOWTO_MERGE_WIKI_DOCS.txt.

Check that docs can be built
----------------------------
Do::

    cd doc/
    make dist

to check that the documentation is in a buildable state.

Check deprecations
------------------
Before the release branch is made, it should be checked that all deprecated
code that should be removed is actually removed, and all new deprecations say
in the docstring or deprecation warning at what version the code will be
removed.

Check the C API version number
------------------------------
The C API version number is recorded in core/code_generators/cversions.txt.
Before every release this number should be updated by running
core/code_generators/cversions.py, and main changes noted in a comment.  If
this is done correctly, compiling the release should not give a warning "API
mismatch detected ..." at the beginning of the build.

The C ABI version number should only be updated for a major release.

Check the release notes
-----------------------
Check that the release notes are up-to-date, and mention at least the
following:

  - major new features
  - deprecated and removed features
  - supported Python versions
  - for SciPy, supported NumPy version(s)
  - outlook for the near future

Also make sure that as soon as the branch is made, there is a new release notes
file in trunk for the next release.

Update the release status and create a release "tag"
----------------------------------------------------
Identify the commit hash of the release, e.g. 1b2e1d63ff.

::
    git co 1b2e1d63ff # gives warning about detached head

Now, set ``release=True`` in setup.py, then

::

    git commit -m "REL: Release." setup.py
    git tag <version>
    git push origin <version>

Update the version of the master branch
---------------------------------------
Increment the release number in setup.py. Release candidates should have "rc1"
(or "rc2", "rcN") appended to the X.Y.Z format.

Also create a new version hash in cversions.txt and a corresponding version
define NPY_x_y_API_VERSION in numpyconfig.h

Make the release
----------------
The tar-files and binary releases for distribution should be uploaded to SourceForge,
together with the Release Notes and the Changelog. Uploading can be done
through a web interface or, more efficiently, through scp/sftp/rsync as
described in the SourceForge
`upload guide <https://sourceforge.net/apps/trac/sourceforge/wiki/Release%20files%20for%20download>`_.
For example::

  scp <filename> <username>,numpy@frs.sourceforge.net:/home/frs/project/n/nu/numpy/NumPy/<releasedir>/

Update PyPi
-----------
The final release (not betas or release candidates) should be uploaded to PyPi.
There are two ways to update PyPi, the first one is::

  $ python setup.py sdist upload

and the second one is to upload the PKG_INFO file inside the sdist dir in the
web interface of PyPi. The source tarball can also be uploaded through this
interface. A simple binary installer for windows, created with
``bdist_wininst``, should also be uploaded to PyPi so ``easy_install numpy``
works.

Update docs.scipy.org
---------------------
Do the following (or ask the doc people to take care of it):

Rebuild and upload documentation:

- ``cd numpy/doc``
- ``make dist``
- Check that the built documentation is OK.
- ``touch output-is-fine``
- ``make upload UPLOAD_TARGET=USERNAME@docs.scipy.org:/home/docserver/www-root/doc/numpy-1.5.x/``

where USERNAME should be replaced by your account on
``docs.scipy.org``, and ``numpy-1.5.x`` by the version number of the
*release series*. For instance, for Numpy 1.5.1, it should be
``numpy-1.5.x`` and for Numpy 2.0.0 ``numpy-2.0.x``.

Rebuild and upload ``docs.scipy.org`` front page, if the release
series is a new one. The front page sources are located in the Scipy
repository:

- ``cd scipy/doc/frontpage``
- Edit ``_templates/indexcontent.html`` to add links to the new release series.
- ``make dist``
- Check that the built documentation is OK.
- ``touch output-is-fine``
- ``make upload USER=USERNAME``


Update scipy.org
----------------
A release announcement with a link to the download site should be placed in the
sidebar of the front page of scipy.org.

Announce to the lists
---------------------
The release should be announced on the mailing lists of
NumPy and SciPy, to python-announce, and possibly also those of
Matplotlib,IPython and/or Pygame.

During the beta/RC phase an explicit request for testing the binaries with
several other libraries (SciPy/Matplotlib/Pygame) should be posted on the
mailing list.

After the final release
-----------------------
After the final release is announced, a few administrative tasks are left to be
done:

  - Forward port changes in the release branch to release notes and release
    scripts, if any, to trunk.
  - Update the Milestones in Trac.