ENH: implement NEP-35's `like=` argument (gh-16935)

This PR adds the implementation of NEP-35's like= argument, allowing dispatch of array creation functions with __array_function__ based on a reference array.

* ENH: Add like= kwarg via __array_function__ dispatcher to asarray

* ENH: Add new function for __array_function__ dispatching from C

This new function allows dispatching from C directly, while also
implementing the new `like=` argument, requiring only minimal
changes to existing array creation functions that need to add
support for that argument.

* ENH: Add like= support to numpy.array

The implementation uses array_implement_c_array_function, thus
introducing minimal complexity to the original _array_fromobject
code.

* BUG: Fix like= dispatcher for np.full

* ENH: Remove np.asarray like= dispatcher via Python

np.asarray can rely on np.array's C dispatcher instead.

* TST: Add some tests for like= argument

Tests comprise some of the functions that have been implemented already:
* np.array (C dispatcher)
* np.asarray (indirect C dispatcher via np.array)
* np.full (Python dispatcher)
* np.ones (Python dispatcher)

* ENH: Remove like= argument during array_implement_array_function

* ENH: Add like= kwarg to ones and full

* BUG: prevent duplicate removal of `like=` argument

* ENH: Make `like=` a keyword-only argument

* ENH: Use PyUnicode_InternFromString in arrayfunction_override

Replace PyUnicode_FromString by PyUnicode_InternFromString to cache
"like" string.

* ENH: Check for arrayfunction_override import errors

Check and handle errors on importing NumPy's Python functions

* BUG: Fix array_implement_c_array_function error handling

* ENH: Handle exceptions with C implementation of `like=`

* ENH: Add `like=` dispatch for all asarray functions

Using Python dispatcher for all of them. Using the C dispatcher
directly on the `np.array` call can result in incorrect behavior.
Incorrect behavior may happen if the downstream library's
implementation is different or if not all keyword arguments are
supported.

* ENH: Simplify handling of exceptions with `like=`

* TST: Add test for exception handling with `like=`

* ENH: Add support for `like=` to `np.empty` and `np.zeros`

* TST: Add `like=` tests for `np.empty` and `np.zeros`

* ENH: Add `like=` to remaining multiarraymodule.c functions

Functions are:
* np.arange
* np.frombuffer
* np.fromfile
* np.fromiter
* np.fromstring

* TST: Add tests for multiarraymodule.c functions with like=

Functions are:
* np.arange
* np.frombuffer
* np.fromfile
* np.fromiter
* np.fromstring

* ENH: Add `like=` support to more creation functions

Support for the following functions is added:
* np.eye
* np.fromfunction
* np.genfromtxt
* np.identity
* np.loadtxt
* np.tri

* TST: Add `like=` tests for multiple functions

Tests for the following functions are added:
* np.eye
* np.fromfunction
* np.genfromtxt
* np.identity
* np.loadtxt
* np.tri

* TST: Reduce code duplication in `like=` tests

* DOC: Document `like=` in functions that support it

Add documentations for the following functions:
* np.array
* np.arange
* np.asarray
* np.asanyarray
* np.ascontiguousarray
* np.asfortranarray
* np.require
* np.empty
* np.full
* np.ones
* np.zeros
* np.identity
* np.eye
* np.tri
* np.frombuffer
* np.fromfile
* np.fromiter
* np.fromstring
* np.loadtxt
* np.genfromtxt

* ENH: Add `like=` to numpy/__init__.pyi stubs

* BUG: Remove duplicate `like=` dispatching in as*array

Functions `np.asanyarray`, `np.contiguousarray` and `np.fortranarray`
were dispatching both via their definitions and `np.array` calls,
the latter should be avoided.

* BUG: Fix missing check in array_implement_array_function

* BUG: Add missing keyword-only markers in stubs

* BUG: Fix duplicate keyword-only marker in array stub

* BUG: Fix syntax error in numpy/__init__.pyi

* BUG: Fix more syntax errors in numpy/__init__.pyi

* ENH: Intern arrayfunction_override strings in multiarraymodule

* STY: Add missing brackets to arrayfunction_override.c

* MAINT: Remove arrayfunction_override dict check for kwarg

* TST: Assert that 'like' is not in TestArrayLike kwargs

* MAINT: Rename array_implement_c_array_function(_creation)

This is done to be more explicit as to its usage being intended for
array creation functions only.

* MAINT: Use NotImplemented to indicate fallback to default

* TST: Test that results with `like=np.array` are correct

* TST: Avoid duplicating MyArray code in TestArrayLike

* TST: Don't delete temp file, it may cause issues with Windows

* TST: Don't rely on eval in TestArrayLike

* TST: Use lambda with StringIO in TestArrayLike

* ENH: Avoid unnecessary Py_XDECREF in arrayfunction_override

* TST: Make TestArrayLike more readable

* ENH: Cleaner error handling in arrayfunction_override

* ENH: Simplify array_implement_c_array_function_creation

* STY: Add missing spaces to multiarraymodule.c

* STY: C99 declaration style in arrayfunction_override.c

* ENH: Simplify arrayfunction_override.c further

Remove cleanup label from array_implementation_c_array_function,
simplifying the code. Fix unitialized variable warning in
array_implementation_array_function_internal.

* DOC: Use string replacement for `like=` documentation

Avoid repeating the full text for the `like=` argument by storing it as
a variable and using `replace` on each docstring.

* DOC: Update `like=` docstring

* TST: Test like= with array not implementing __array_function__

* TST: Add missing asanyarray to TestArrayLike

* ENH: Use helper function for like= dispatching

Avoid dispatching like= from Python implementation functions to improve
their performance. This is achieved by only calling a dispatcher
function when like is passed by the users.

* ENH: Rename array_function_dispatch kwarg to public_api

* BUG: Add accidentally removed decorator for np.eye back

* DOC: Add set_array_function_like_doc function

The function keeps Python files cleaner and resolve errors when __doc__
is not defined due to PYTHONOPTIMIZE or -OO .

* DOC: Add mention to like= kwarg being experimental

* TST: Test like= with not implemented downstream function

* DOC: Fix like= docstring reference to NEP 35.

* ENH: Prevent silent errors if public_api is not callable

* ENH: Make set_array_function_like_doc a decorator

* ENH: Simplify `_*_with_like` functions

* BUG: Fix multiple `like=` dispatching in `require`

* MAINT: Remove now unused public_api from array_function_dispatch

Co-authored-by: Sebastian Berg <sebastian@sipsolutions.net>

1 files changed, 62 insertions, 3 deletions

diff --git a/numpy/lib/npyio.py b/numpy/lib/npyio.py
index 58affc2fc..cc3465cc6 100644
--- a/numpy/lib/npyio.py
+++ b/numpy/lib/npyio.py
@@ -14,7 +14,7 @@ from . import format
 from ._datasource import DataSource
 from numpy.core import overrides
 from numpy.core.multiarray import packbits, unpackbits
-from numpy.core.overrides import set_module
+from numpy.core.overrides import set_array_function_like_doc, set_module
 from numpy.core._internal import recursive
 from ._iotools import (
     LineSplitter, NameValidator, StringConverter, ConverterError,
@@ -790,10 +790,17 @@ def _getconv(dtype):
 _loadtxt_chunksize = 50000
 
 
+def _loadtxt_dispatcher(fname, dtype=None, comments=None, delimiter=None,
+                        converters=None, skiprows=None, usecols=None, unpack=None,
+                        ndmin=None, encoding=None, max_rows=None, *, like=None):
+    return (like,)
+
+
+@set_array_function_like_doc
 @set_module('numpy')
 def loadtxt(fname, dtype=float, comments='#', delimiter=None,
             converters=None, skiprows=0, usecols=None, unpack=False,
-            ndmin=0, encoding='bytes', max_rows=None):
+            ndmin=0, encoding='bytes', max_rows=None, *, like=None):
     r"""
     Load data from a text file.
 
@@ -860,6 +867,9 @@ def loadtxt(fname, dtype=float, comments='#', delimiter=None,
         is to read all the lines.
 
         .. versionadded:: 1.16.0
+    ${ARRAY_FUNCTION_LIKE}
+
+        .. versionadded:: 1.20.0
 
     Returns
     -------
@@ -917,6 +927,14 @@ def loadtxt(fname, dtype=float, comments='#', delimiter=None,
            [-17.57,  63.94]])
     """
 
+    if like is not None:
+        return _loadtxt_with_like(
+            fname, dtype=dtype, comments=comments, delimiter=delimiter,
+            converters=converters, skiprows=skiprows, usecols=usecols,
+            unpack=unpack, ndmin=ndmin, encoding=encoding,
+            max_rows=max_rows, like=like
+        )
+
     # - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
     # Nested functions used by loadtxt.
     # - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
@@ -1201,6 +1219,11 @@ def loadtxt(fname, dtype=float, comments='#', delimiter=None,
         return X
 
 
+_loadtxt_with_like = array_function_dispatch(
+    _loadtxt_dispatcher
+)(loadtxt)
+
+
 def _savetxt_dispatcher(fname, X, fmt=None, delimiter=None, newline=None,
                         header=None, footer=None, comments=None,
                         encoding=None):
@@ -1554,6 +1577,18 @@ def fromregex(file, regexp, dtype, encoding=None):
 #####--------------------------------------------------------------------------
 
 
+def _genfromtxt_dispatcher(fname, dtype=None, comments=None, delimiter=None,
+                           skip_header=None, skip_footer=None, converters=None,
+                           missing_values=None, filling_values=None, usecols=None,
+                           names=None, excludelist=None, deletechars=None,
+                           replace_space=None, autostrip=None, case_sensitive=None,
+                           defaultfmt=None, unpack=None, usemask=None, loose=None,
+                           invalid_raise=None, max_rows=None, encoding=None, *,
+                           like=None):
+    return (like,)
+
+
+@set_array_function_like_doc
 @set_module('numpy')
 def genfromtxt(fname, dtype=float, comments='#', delimiter=None,
                skip_header=0, skip_footer=0, converters=None,
@@ -1562,7 +1597,8 @@ def genfromtxt(fname, dtype=float, comments='#', delimiter=None,
                deletechars=''.join(sorted(NameValidator.defaultdeletechars)),
                replace_space='_', autostrip=False, case_sensitive=True,
                defaultfmt="f%i", unpack=None, usemask=False, loose=True,
-               invalid_raise=True, max_rows=None, encoding='bytes'):
+               invalid_raise=True, max_rows=None, encoding='bytes', *,
+               like=None):
     """
     Load data from a text file, with missing values handled as specified.
 
@@ -1659,6 +1695,9 @@ def genfromtxt(fname, dtype=float, comments='#', delimiter=None,
         to None the system default is used. The default value is 'bytes'.
 
         .. versionadded:: 1.14.0
+    ${ARRAY_FUNCTION_LIKE}
+
+        .. versionadded:: 1.20.0
 
     Returns
     -------
@@ -1737,6 +1776,21 @@ def genfromtxt(fname, dtype=float, comments='#', delimiter=None,
       dtype=[('f0', 'S12'), ('f1', 'S12')])
 
     """
+
+    if like is not None:
+        return _genfromtxt_with_like(
+            fname, dtype=dtype, comments=comments, delimiter=delimiter,
+            skip_header=skip_header, skip_footer=skip_footer,
+            converters=converters, missing_values=missing_values,
+            filling_values=filling_values, usecols=usecols, names=names,
+            excludelist=excludelist, deletechars=deletechars,
+            replace_space=replace_space, autostrip=autostrip,
+            case_sensitive=case_sensitive, defaultfmt=defaultfmt,
+            unpack=unpack, usemask=usemask, loose=loose,
+            invalid_raise=invalid_raise, max_rows=max_rows, encoding=encoding,
+            like=like
+        )
+
     if max_rows is not None:
         if skip_footer:
             raise ValueError(
@@ -2250,6 +2304,11 @@ def genfromtxt(fname, dtype=float, comments='#', delimiter=None,
     return output.squeeze()
 
 
+_genfromtxt_with_like = array_function_dispatch(
+    _genfromtxt_dispatcher
+)(genfromtxt)
+
+
 def ndfromtxt(fname, **kwargs):
     """
     Load ASCII data stored in a file and return it as a single array.