ENH: implement NEP-35's `like=` argument (gh-16935)

This PR adds the implementation of NEP-35's like= argument, allowing dispatch of array creation functions with __array_function__ based on a reference array.

* ENH: Add like= kwarg via __array_function__ dispatcher to asarray

* ENH: Add new function for __array_function__ dispatching from C

This new function allows dispatching from C directly, while also
implementing the new `like=` argument, requiring only minimal
changes to existing array creation functions that need to add
support for that argument.

* ENH: Add like= support to numpy.array

The implementation uses array_implement_c_array_function, thus
introducing minimal complexity to the original _array_fromobject
code.

* BUG: Fix like= dispatcher for np.full

* ENH: Remove np.asarray like= dispatcher via Python

np.asarray can rely on np.array's C dispatcher instead.

* TST: Add some tests for like= argument

Tests comprise some of the functions that have been implemented already:
* np.array (C dispatcher)
* np.asarray (indirect C dispatcher via np.array)
* np.full (Python dispatcher)
* np.ones (Python dispatcher)

* ENH: Remove like= argument during array_implement_array_function

* ENH: Add like= kwarg to ones and full

* BUG: prevent duplicate removal of `like=` argument

* ENH: Make `like=` a keyword-only argument

* ENH: Use PyUnicode_InternFromString in arrayfunction_override

Replace PyUnicode_FromString by PyUnicode_InternFromString to cache
"like" string.

* ENH: Check for arrayfunction_override import errors

Check and handle errors on importing NumPy's Python functions

* BUG: Fix array_implement_c_array_function error handling

* ENH: Handle exceptions with C implementation of `like=`

* ENH: Add `like=` dispatch for all asarray functions

Using Python dispatcher for all of them. Using the C dispatcher
directly on the `np.array` call can result in incorrect behavior.
Incorrect behavior may happen if the downstream library's
implementation is different or if not all keyword arguments are
supported.

* ENH: Simplify handling of exceptions with `like=`

* TST: Add test for exception handling with `like=`

* ENH: Add support for `like=` to `np.empty` and `np.zeros`

* TST: Add `like=` tests for `np.empty` and `np.zeros`

* ENH: Add `like=` to remaining multiarraymodule.c functions

Functions are:
* np.arange
* np.frombuffer
* np.fromfile
* np.fromiter
* np.fromstring

* TST: Add tests for multiarraymodule.c functions with like=

Functions are:
* np.arange
* np.frombuffer
* np.fromfile
* np.fromiter
* np.fromstring

* ENH: Add `like=` support to more creation functions

Support for the following functions is added:
* np.eye
* np.fromfunction
* np.genfromtxt
* np.identity
* np.loadtxt
* np.tri

* TST: Add `like=` tests for multiple functions

Tests for the following functions are added:
* np.eye
* np.fromfunction
* np.genfromtxt
* np.identity
* np.loadtxt
* np.tri

* TST: Reduce code duplication in `like=` tests

* DOC: Document `like=` in functions that support it

Add documentations for the following functions:
* np.array
* np.arange
* np.asarray
* np.asanyarray
* np.ascontiguousarray
* np.asfortranarray
* np.require
* np.empty
* np.full
* np.ones
* np.zeros
* np.identity
* np.eye
* np.tri
* np.frombuffer
* np.fromfile
* np.fromiter
* np.fromstring
* np.loadtxt
* np.genfromtxt

* ENH: Add `like=` to numpy/__init__.pyi stubs

* BUG: Remove duplicate `like=` dispatching in as*array

Functions `np.asanyarray`, `np.contiguousarray` and `np.fortranarray`
were dispatching both via their definitions and `np.array` calls,
the latter should be avoided.

* BUG: Fix missing check in array_implement_array_function

* BUG: Add missing keyword-only markers in stubs

* BUG: Fix duplicate keyword-only marker in array stub

* BUG: Fix syntax error in numpy/__init__.pyi

* BUG: Fix more syntax errors in numpy/__init__.pyi

* ENH: Intern arrayfunction_override strings in multiarraymodule

* STY: Add missing brackets to arrayfunction_override.c

* MAINT: Remove arrayfunction_override dict check for kwarg

* TST: Assert that 'like' is not in TestArrayLike kwargs

* MAINT: Rename array_implement_c_array_function(_creation)

This is done to be more explicit as to its usage being intended for
array creation functions only.

* MAINT: Use NotImplemented to indicate fallback to default

* TST: Test that results with `like=np.array` are correct

* TST: Avoid duplicating MyArray code in TestArrayLike

* TST: Don't delete temp file, it may cause issues with Windows

* TST: Don't rely on eval in TestArrayLike

* TST: Use lambda with StringIO in TestArrayLike

* ENH: Avoid unnecessary Py_XDECREF in arrayfunction_override

* TST: Make TestArrayLike more readable

* ENH: Cleaner error handling in arrayfunction_override

* ENH: Simplify array_implement_c_array_function_creation

* STY: Add missing spaces to multiarraymodule.c

* STY: C99 declaration style in arrayfunction_override.c

* ENH: Simplify arrayfunction_override.c further

Remove cleanup label from array_implementation_c_array_function,
simplifying the code. Fix unitialized variable warning in
array_implementation_array_function_internal.

* DOC: Use string replacement for `like=` documentation

Avoid repeating the full text for the `like=` argument by storing it as
a variable and using `replace` on each docstring.

* DOC: Update `like=` docstring

* TST: Test like= with array not implementing __array_function__

* TST: Add missing asanyarray to TestArrayLike

* ENH: Use helper function for like= dispatching

Avoid dispatching like= from Python implementation functions to improve
their performance. This is achieved by only calling a dispatcher
function when like is passed by the users.

* ENH: Rename array_function_dispatch kwarg to public_api

* BUG: Add accidentally removed decorator for np.eye back

* DOC: Add set_array_function_like_doc function

The function keeps Python files cleaner and resolve errors when __doc__
is not defined due to PYTHONOPTIMIZE or -OO .

* DOC: Add mention to like= kwarg being experimental

* TST: Test like= with not implemented downstream function

* DOC: Fix like= docstring reference to NEP 35.

* ENH: Prevent silent errors if public_api is not callable

* ENH: Make set_array_function_like_doc a decorator

* ENH: Simplify `_*_with_like` functions

* BUG: Fix multiple `like=` dispatching in `require`

* MAINT: Remove now unused public_api from array_function_dispatch

Co-authored-by: Sebastian Berg <sebastian@sipsolutions.net>

1 files changed, 34 insertions, 3 deletions

diff --git a/numpy/lib/twodim_base.py b/numpy/lib/twodim_base.py
index cd7484241..2b4cbdfbb 100644
--- a/numpy/lib/twodim_base.py
+++ b/numpy/lib/twodim_base.py
@@ -8,7 +8,7 @@ from numpy.core.numeric import (
     asarray, where, int8, int16, int32, int64, empty, promote_types, diagonal,
     nonzero
     )
-from numpy.core.overrides import set_module
+from numpy.core.overrides import set_array_function_like_doc, set_module
 from numpy.core import overrides
 from numpy.core import iinfo
 
@@ -149,8 +149,13 @@ def flipud(m):
     return m[::-1, ...]
 
 
+def _eye_dispatcher(N, M=None, k=None, dtype=None, order=None, *, like=None):
+    return (like,)
+
+
+@set_array_function_like_doc
 @set_module('numpy')
-def eye(N, M=None, k=0, dtype=float, order='C'):
+def eye(N, M=None, k=0, dtype=float, order='C', *, like=None):
     """
     Return a 2-D array with ones on the diagonal and zeros elsewhere.
 
@@ -171,6 +176,9 @@ def eye(N, M=None, k=0, dtype=float, order='C'):
         column-major (Fortran-style) order in memory.
 
         .. versionadded:: 1.14.0
+    ${ARRAY_FUNCTION_LIKE}
+
+        .. versionadded:: 1.20.0
 
     Returns
     -------
@@ -194,6 +202,8 @@ def eye(N, M=None, k=0, dtype=float, order='C'):
            [0.,  0.,  0.]])
 
     """
+    if like is not None:
+        return _eye_with_like(N, M=M, k=k, dtype=dtype, order=order, like=like)
     if M is None:
         M = N
     m = zeros((N, M), dtype=dtype, order=order)
@@ -207,6 +217,11 @@ def eye(N, M=None, k=0, dtype=float, order='C'):
     return m
 
 
+_eye_with_like = array_function_dispatch(
+    _eye_dispatcher
+)(eye)
+
+
 def _diag_dispatcher(v, k=None):
     return (v,)
 
@@ -343,8 +358,13 @@ def diagflat(v, k=0):
     return wrap(res)
 
 
+def _tri_dispatcher(N, M=None, k=None, dtype=None, *, like=None):
+    return (like,)
+
+
+@set_array_function_like_doc
 @set_module('numpy')
-def tri(N, M=None, k=0, dtype=float):
+def tri(N, M=None, k=0, dtype=float, *, like=None):
     """
     An array with ones at and below the given diagonal and zeros elsewhere.
 
@@ -361,6 +381,9 @@ def tri(N, M=None, k=0, dtype=float):
         and `k` > 0 is above.  The default is 0.
     dtype : dtype, optional
         Data type of the returned array.  The default is float.
+    ${ARRAY_FUNCTION_LIKE}
+
+        .. versionadded:: 1.20.0
 
     Returns
     -------
@@ -381,6 +404,9 @@ def tri(N, M=None, k=0, dtype=float):
            [1.,  1.,  0.,  0.,  0.]])
 
     """
+    if like is not None:
+        return _tri_with_like(N, M=M, k=k, dtype=dtype, like=like)
+
     if M is None:
         M = N
 
@@ -393,6 +419,11 @@ def tri(N, M=None, k=0, dtype=float):
     return m
 
 
+_tri_with_like = array_function_dispatch(
+    _tri_dispatcher
+)(tri)
+
+
 def _trilu_dispatcher(m, k=None):
     return (m,)