Merge pull request #20875 from rgommers/deprecate-npdistutils

DEP: deprecate `numpy.distutils`, and add a migration guide

6 files changed, 171 insertions, 0 deletions

diff --git a/doc/release/upcoming_changes/20875.deprecation.rst b/doc/release/upcoming_changes/20875.deprecation.rst
new file mode 100644
index 000000000..4951f6ab3
--- /dev/null
+++ b/doc/release/upcoming_changes/20875.deprecation.rst
@@ -0,0 +1,9 @@
+Deprecation of ``numpy.distutils``
+----------------------------------
+
+``numpy.distutils`` has been deprecated, as a result of ``distutils``
+itself being deprecated. It will not be present in NumPy for Python >= 3.12,
+and will be removed completely 2 years after the release of Python 3.12
+
+For more details, see :ref:`distutils-status-migration`.
+
diff --git a/doc/source/reference/distutils.rst b/doc/source/reference/distutils.rst
index 9db757c89..ff1ba3b0d 100644
--- a/doc/source/reference/distutils.rst
+++ b/doc/source/reference/distutils.rst
@@ -4,6 +4,11 @@ Packaging (:mod:`numpy.distutils`)
 
 .. module:: numpy.distutils
 
+.. warning::
+
+   ``numpy.distutils`` is deprecated, and will be removed for
+   Python >= 3.12. For more details, see :ref:`distutils-status-migration`
+
 NumPy provides enhanced distutils functionality to make it easier to
 build and install sub-packages, auto-generate code, and extension
 modules that use Fortran-compiled libraries. To use features of NumPy
diff --git a/doc/source/reference/distutils_guide.rst b/doc/source/reference/distutils_guide.rst
index 081719d16..5bb4c2878 100644
--- a/doc/source/reference/distutils_guide.rst
+++ b/doc/source/reference/distutils_guide.rst
@@ -3,5 +3,11 @@
 NumPy Distutils - Users Guide
 =============================
 
+.. warning::
+
+   ``numpy.distutils`` is deprecated, and will be removed for
+   Python >= 3.12. For more details, see :ref:`distutils-status-migration`
+
+
 .. include:: ../../DISTUTILS.rst.txt
    :start-line: 6
diff --git a/doc/source/reference/distutils_status_migration.rst b/doc/source/reference/distutils_status_migration.rst
new file mode 100644
index 000000000..9ef5f7232
--- /dev/null
+++ b/doc/source/reference/distutils_status_migration.rst
@@ -0,0 +1,138 @@
+.. _distutils-status-migration:
+
+Status of ``numpy.distutils`` and migration advice
+==================================================
+
+`numpy.distutils` has been deprecated in NumPy ``1.23.0``. It will be removed
+for Python 3.12; for Python <= 3.11 it will not be removed until 2 years after
+the Python 3.12 release (Oct 2025).
+
+
+.. warning::
+
+   ``numpy.distutils`` is only tested with ``setuptools < 60.0``, newer
+   versions may break. See :ref:`numpy-setuptools-interaction` for details.
+
+
+Migration advice
+----------------
+
+It is **not necessary** to migrate immediately - the release date for Python 3.12
+is October 2023. It may be beneficial to wait with migrating until there are
+examples from other projects to follow (see below).
+
+There are several build systems which are good options to migrate to. Assuming
+you have compiled code in your package (if not, we recommend using Flit_) and
+you want to be using a well-designed, modern and reliable build system, we
+recommend:
+
+1. Meson_
+2. CMake_ (or scikit-build_ as an interface to CMake)
+
+If you have modest needs (only simple Cython/C extensions, and perhaps nested
+``setup.py`` files) and have been happy with ``numpy.distutils`` so far, you
+can also consider switching to ``setuptools``. Note that most functionality of
+``numpy.disutils`` is unlikely to be ported to ``setuptools``.
+
+
+Moving to Meson
+```````````````
+
+SciPy is moving to Meson for its 1.9.0 release, planned for July 2022. During
+this process, any remaining issues with Meson's Python support and achieving
+feature parity with ``numpy.distutils`` will be resolved. *Note: parity means a
+large superset, but right now some BLAS/LAPACK support is missing and there are
+a few open issues related to Cython.* SciPy uses almost all functionality that
+``numpy.distutils`` offers, so if SciPy has successfully made a release with
+Meson as the build system, there should be no blockers left to migrate, and
+SciPy will be a good reference for other packages who are migrating.
+For more details about the SciPy migration, see:
+
+- `RFC: switch to Meson as a build system <https://github.com/scipy/scipy/issues/13615>`__
+- `Tracking issue for Meson support <https://github.com/rgommers/scipy/issues/22>`__
+
+NumPy itself will very likely migrate to Meson as well, once the SciPy
+migration is done.
+
+
+Moving to CMake / scikit-build
+``````````````````````````````
+
+See the `scikit-build documentation <https://scikit-build.readthedocs.io/en/latest/>`__
+for how to use scikit-build. Please note that as of Feb 2022, scikit-build
+still relies on setuptools, so it's probably not quite ready yet for a
+post-distutils world. How quickly this changes depends on funding, the current
+(Feb 2022) estimate is that if funding arrives then a viable ``numpy.distutils``
+replacement will be ready at the end of 2022, and a very polished replacement
+mid-2023.  For more details on this, see
+`this blog post by Henry Schreiner <https://iscinumpy.gitlab.io/post/scikit-build-proposal/>`__.
+
+
+Moving to ``setuptools``
+````````````````````````
+
+For projects that only use ``numpy.distutils`` for historical reasons, and do
+not actually use features beyond those that ``setuptools`` also supports,
+moving to ``setuptools`` is likely the solution which costs the least effort.
+To assess that, there are the ``numpy.distutils`` features that are *not*
+present in ``setuptools``:
+
+- Nested ``setup.py`` files
+- Fortran build support
+- BLAS/LAPACK library support (OpenBLAS, MKL, ATLAS, Netlib LAPACK/BLAS, BLIS, 64-bit ILP interface, etc.)
+- Support for a few other scientific libraries, like FFTW and UMFPACK
+- Better MinGW support
+- Per-compiler build flag customization (e.g. `-O3` and `SSE2` flags are default)
+- a simple user build config system, see [site.cfg.example](https://github.com/numpy/numpy/blob/master/site.cfg.example)
+- SIMD intrinsics support
+
+The most widely used feature is nested ``setup.py`` files. This feature will
+likely be ported to ``setuptools`` (see
+`gh-18588 <https://github.com/numpy/numpy/issues/18588>`__ for status).
+Projects only using that feature could move to ``setuptools`` after that is
+done. In case a project uses only a couple of ``setup.py`` files, it also could
+make sense to simply aggregate all the content of those files into a single
+``setup.py`` file and then move to ``setuptools``. This involves dropping all
+``Configuration`` instances, and using ``Extension`` instead. E.g.,::
+
+    from distutils.core import setup
+    from distutils.extension import Extension
+    setup(name='foobar',
+          version='1.0',
+          ext_modules=[
+              Extension('foopkg.foo', ['foo.c']),
+              Extension('barpkg.bar', ['bar.c']),
+              ],
+          )
+
+For more details, see the
+`setuptools documentation <https://setuptools.pypa.io/en/latest/setuptools.html>`__
+
+
+.. _numpy-setuptools-interaction:
+
+Interaction of ``numpy.disutils`` with ``setuptools``
+-----------------------------------------------------
+
+It is recommended to use ``setuptools < 60.0``. Newer versions may work, but
+are not guaranteed to. The reason for this is that ``setuptools`` 60.0 enabled
+a vendored copy of ``distutils``, including backwards incompatible changes that
+affect some functionality in ``numpy.distutils``.
+
+If you are using only simple Cython or C extensions with minimal use of
+``numpy.distutils`` functionality beyond nested ``setup.py`` files (its most
+popular feature, see :class:`Configuration <numpy.distutils.misc_util.Configuration>`),
+then latest ``setuptools`` is likely to continue working. In case of problems,
+you can also try ``SETUPTOOLS_USE_DISTUTILS=stdlib`` to avoid the backwards
+incompatible changes in ``setuptools``.
+
+Whatever you do, it is recommended to put an upper bound on your ``setuptools``
+build requirement in ``pyproject.toml`` to avoid future breakage - see
+:ref:`for-downstream-package-authors`.
+
+
+.. _Flit: https://flit.readthedocs.io
+.. _CMake: https://cmake.org/
+.. _Meson: https://mesonbuild.com/
+.. _scikit-build: https://scikit-build.readthedocs.io/
+
diff --git a/doc/source/reference/index.rst b/doc/source/reference/index.rst
index 00c929d66..66bb45d6a 100644
--- a/doc/source/reference/index.rst
+++ b/doc/source/reference/index.rst
@@ -25,6 +25,7 @@ For learning how to use NumPy, see the :ref:`complete documentation <numpy_docs_
    global_state
    distutils
    distutils_guide
+   distutils_status_migration
    c-api/index
    simd/index
    swig
diff --git a/numpy/distutils/__init__.py b/numpy/distutils/__init__.py
index 79974d1c2..3be3372ff 100644
--- a/numpy/distutils/__init__.py
+++ b/numpy/distutils/__init__.py
@@ -19,6 +19,8 @@ LAPACK, and for setting include paths and similar build options, please see
 
 """
 
+import warnings
+
 # Must import local ccompiler ASAP in order to get
 # customized CCompiler.spawn effective.
 from . import ccompiler
@@ -26,6 +28,16 @@ from . import unixccompiler
 
 from .npy_pkg_config import *
 
+warnings.warn("\n\n"
+    "  `numpy.distutils` is deprecated since NumPy 1.23.0, as a result\n"
+    "  of the deprecation of `distutils` itself. It will be removed for\n"
+    "  Python >= 3.12. For older Python versions it will remain present.\n"
+    "  It is recommended to use `setuptools < 60.0` for those Python versions.\n"
+    "  For more details, see: TODO! \n\n",
+    DeprecationWarning, stacklevel=2
+)
+del warnings
+
 # If numpy is installed, add distutils.test()
 try:
     from . import __config__