diff options
| -rw-r--r-- | doc/neps/nep-0037-array-module.rst | 80 | ||||
| -rw-r--r-- | numpy/core/src/multiarray/descriptor.c | 95 |
2 files changed, 78 insertions, 97 deletions
diff --git a/doc/neps/nep-0037-array-module.rst b/doc/neps/nep-0037-array-module.rst index 9c89e26ec..387356490 100644 --- a/doc/neps/nep-0037-array-module.rst +++ b/doc/neps/nep-0037-array-module.rst @@ -12,7 +12,7 @@ NEP 37 — A dispatch protocol for NumPy-like modules Abstract -------- -NEP-18's `__array_function__` has been a mixed success. Some projects (e.g., +NEP-18's ``__array_function__`` has been a mixed success. Some projects (e.g., dask, CuPy, xarray, sparse, Pint) have enthusiastically adopted it. Others (e.g., PyTorch, JAX, SciPy) have been more reluctant. Here we propose a new protocol, ``__array_module__``, that we expect could eventually subsume most @@ -29,47 +29,47 @@ There are two broad ways in which NEP-18 has fallen short of its goals: 1. **Maintainability concerns**. `__array_function__` has significant implications for libraries that use it: - - Projects like `PyTorch - <https://github.com/pytorch/pytorch/issues/22402>`_, `JAX - <https://github.com/google/jax/issues/1565>`_ and even `scipy.sparse - <https://github.com/scipy/scipy/issues/10362>`_ have been reluctant to - implement `__array_function__` in part because they are concerned about - **breaking existing code**: users expect NumPy functions like - ``np.concatenate`` to return NumPy arrays. This is a fundamental - limitation of the ``__array_function__`` design, which we chose to allow - overriding the existing ``numpy`` namespace. - - ``__array_function__`` currently requires an "all or nothing" approach to - implementing NumPy's API. There is no good pathway for **incremental - adoption**, which is particularly problematic for established projects - for which adopting ``__array_function__`` would result in breaking - changes. - - It is no longer possible to use **aliases to NumPy functions** within - modules that support overrides. For example, both CuPy and JAX set - ``result_type = np.result_type``. - - Implementing **fall-back mechanisms** for unimplemented NumPy functions - by using NumPy's implementation is hard to get right (but see the - `version from dask <https://github.com/dask/dask/pull/5043>`_), because - ``__array_function__`` does not present a consistent interface. - Converting all arguments of array type requires recursing into generic - arguments of the form ``*args, **kwargs``. + - Projects like `PyTorch + <https://github.com/pytorch/pytorch/issues/22402>`_, `JAX + <https://github.com/google/jax/issues/1565>`_ and even `scipy.sparse + <https://github.com/scipy/scipy/issues/10362>`_ have been reluctant to + implement `__array_function__` in part because they are concerned about + **breaking existing code**: users expect NumPy functions like + ``np.concatenate`` to return NumPy arrays. This is a fundamental + limitation of the ``__array_function__`` design, which we chose to allow + overriding the existing ``numpy`` namespace. + - ``__array_function__`` currently requires an "all or nothing" approach to + implementing NumPy's API. There is no good pathway for **incremental + adoption**, which is particularly problematic for established projects + for which adopting ``__array_function__`` would result in breaking + changes. + - It is no longer possible to use **aliases to NumPy functions** within + modules that support overrides. For example, both CuPy and JAX set + ``result_type = np.result_type``. + - Implementing **fall-back mechanisms** for unimplemented NumPy functions + by using NumPy's implementation is hard to get right (but see the + `version from dask <https://github.com/dask/dask/pull/5043>`_), because + ``__array_function__`` does not present a consistent interface. + Converting all arguments of array type requires recursing into generic + arguments of the form ``*args, **kwargs``. 2. **Limitations on what can be overridden.** ``__array_function__`` has some important gaps, most notably array creation and coercion functions: - - **Array creation** routines (e.g., ``np.arange`` and those in - ``np.random``) need some other mechanism for indicating what type of - arrays to create. `NEP 36 <https://github.com/numpy/numpy/pull/14715>`_ - proposed adding optional ``like=`` arguments to functions without - existing array arguments. However, we still lack any mechanism to - override methods on objects, such as those needed by - ``np.random.RandomState``. - - **Array conversion** can't reuse the existing coercion functions like - ``np.asarray``, because ``np.asarray`` sometimes means "convert to an - exact ``np.ndarray``" and other times means "convert to something _like_ - a NumPy array." This led to the `NEP 30 - <https://numpy.org/neps/nep-0030-duck-array-protocol.html>`_ proposal for - a separate ``np.duckarray`` function, but this still does not resolve how - to cast one duck array into a type matching another duck array. + - **Array creation** routines (e.g., ``np.arange`` and those in + ``np.random``) need some other mechanism for indicating what type of + arrays to create. `NEP 36 <https://github.com/numpy/numpy/pull/14715>`_ + proposed adding optional ``like=`` arguments to functions without + existing array arguments. However, we still lack any mechanism to + override methods on objects, such as those needed by + ``np.random.RandomState``. + - **Array conversion** can't reuse the existing coercion functions like + ``np.asarray``, because ``np.asarray`` sometimes means "convert to an + exact ``np.ndarray``" and other times means "convert to something _like_ + a NumPy array." This led to the `NEP 30 + <https://numpy.org/neps/nep-0030-duck-array-protocol.html>`_ proposal for + a separate ``np.duckarray`` function, but this still does not resolve how + to cast one duck array into a type matching another duck array. ``get_array_module`` and the ``__array_module__`` protocol ---------------------------------------------------------- @@ -196,7 +196,7 @@ general because such fall-back behavior can be error prone): return getattr(base_module, name, getattr(numpy, name)) Subclassing from ``numpy.ndarray`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ All of the same guidance about well-defined type casting hierarchies from NEP-18 still applies. ``numpy.ndarray`` itself contains a matching @@ -251,7 +251,7 @@ equivalent to this Python code: return implementing_arrays, types Relationship with ``__array_ufunc__`` and ``__array_function__`` ---------------------------------------------------------------- +---------------------------------------------------------------- These older protocols have distinct use-cases and should remain =============================================================== diff --git a/numpy/core/src/multiarray/descriptor.c b/numpy/core/src/multiarray/descriptor.c index 7a94929dd..1f151fd58 100644 --- a/numpy/core/src/multiarray/descriptor.c +++ b/numpy/core/src/multiarray/descriptor.c @@ -43,6 +43,24 @@ static PyArray_Descr * _use_inherit(PyArray_Descr *type, PyObject *newobj, int *errflag); static PyArray_Descr * +_arraydescr_run_converter(PyObject *arg, int align) +{ + PyArray_Descr *type = NULL; + if (align) { + if (PyArray_DescrAlignConverter(arg, &type) == NPY_FAIL) { + return NULL; + } + } + else { + if (PyArray_DescrConverter(arg, &type) == NPY_FAIL) { + return NULL; + } + } + assert(type != NULL); + return type; +} + +static PyArray_Descr * _arraydescr_from_ctypes_type(PyTypeObject *type) { PyObject *_numpy_dtype_ctypes; @@ -232,15 +250,9 @@ _convert_from_tuple(PyObject *obj, int align) if (PyTuple_GET_SIZE(obj) != 2) { return NULL; } - if (align) { - if (!PyArray_DescrAlignConverter(PyTuple_GET_ITEM(obj, 0), &type)) { - return NULL; - } - } - else { - if (!PyArray_DescrConverter(PyTuple_GET_ITEM(obj, 0), &type)) { - return NULL; - } + type = _arraydescr_run_converter(PyTuple_GET_ITEM(obj, 0), align); + if (type == NULL) { + return NULL; } val = PyTuple_GET_ITEM(obj,1); /* try to interpret next item as a type */ @@ -411,7 +423,6 @@ static PyArray_Descr * _convert_from_array_descr(PyObject *obj, int align) { int n, i, totalsize; - int ret; PyObject *fields, *item, *newobj; PyObject *name, *tup, *title; PyObject *nameslist; @@ -491,31 +502,22 @@ _convert_from_array_descr(PyObject *obj, int align) /* Process rest */ if (PyTuple_GET_SIZE(item) == 2) { - if (align) { - ret = PyArray_DescrAlignConverter(PyTuple_GET_ITEM(item, 1), - &conv); - } - else { - ret = PyArray_DescrConverter(PyTuple_GET_ITEM(item, 1), &conv); + conv = _arraydescr_run_converter(PyTuple_GET_ITEM(item, 1), align); + if (conv == NULL) { + goto fail; } } else if (PyTuple_GET_SIZE(item) == 3) { newobj = PyTuple_GetSlice(item, 1, 3); - if (align) { - ret = PyArray_DescrAlignConverter(newobj, &conv); - } - else { - ret = PyArray_DescrConverter(newobj, &conv); - } + conv = _arraydescr_run_converter(newobj, align); Py_DECREF(newobj); + if (conv == NULL) { + goto fail; + } } else { goto fail; } - if (ret == NPY_FAIL) { - goto fail; - } - if ((PyDict_GetItem(fields, name) != NULL) || (title && PyBaseString_Check(title) @@ -616,7 +618,6 @@ _convert_from_list(PyObject *obj, int align) PyArray_Descr *new; PyObject *key, *tup; PyObject *nameslist = NULL; - int ret; int maxalign = 0; /* Types with fields need the Python C API for field access */ char dtypeflags = NPY_NEEDS_PYAPI; @@ -643,13 +644,8 @@ _convert_from_list(PyObject *obj, int align) for (i = 0; i < n; i++) { tup = PyTuple_New(2); key = PyUString_FromFormat("f%d", i); - if (align) { - ret = PyArray_DescrAlignConverter(PyList_GET_ITEM(obj, i), &conv); - } - else { - ret = PyArray_DescrConverter(PyList_GET_ITEM(obj, i), &conv); - } - if (ret == NPY_FAIL) { + conv = _arraydescr_run_converter(PyList_GET_ITEM(obj, i), align); + if (conv == NULL) { Py_DECREF(tup); Py_DECREF(key); goto fail; @@ -1091,7 +1087,7 @@ _convert_from_dict(PyObject *obj, int align) totalsize = 0; for (i = 0; i < n; i++) { PyObject *tup, *descr, *ind, *title, *name, *off; - int len, ret, _align = 1; + int len, _align = 1; PyArray_Descr *newdescr; /* Build item to insert (descr, offset, [title])*/ @@ -1115,14 +1111,9 @@ _convert_from_dict(PyObject *obj, int align) Py_DECREF(ind); goto fail; } - if (align) { - ret = PyArray_DescrAlignConverter(descr, &newdescr); - } - else { - ret = PyArray_DescrConverter(descr, &newdescr); - } + newdescr = _arraydescr_run_converter(descr, align); Py_DECREF(descr); - if (ret == NPY_FAIL) { + if (newdescr == NULL) { Py_DECREF(tup); Py_DECREF(ind); goto fail; @@ -1168,7 +1159,9 @@ _convert_from_dict(PyObject *obj, int align) "not divisible by the field alignment %d " "with align=True", offset, newdescr->alignment); - ret = NPY_FAIL; + Py_DECREF(ind); + Py_DECREF(tup); + goto fail; } else if (offset + newdescr->elsize > totalsize) { totalsize = offset + newdescr->elsize; @@ -1181,11 +1174,6 @@ _convert_from_dict(PyObject *obj, int align) PyTuple_SET_ITEM(tup, 1, PyInt_FromLong(totalsize)); totalsize += newdescr->elsize; } - if (ret == NPY_FAIL) { - Py_DECREF(ind); - Py_DECREF(tup); - goto fail; - } if (len == 3) { PyTuple_SET_ITEM(tup, 2, title); } @@ -1223,9 +1211,6 @@ _convert_from_dict(PyObject *obj, int align) } } Py_DECREF(tup); - if (ret == NPY_FAIL) { - goto fail; - } dtypeflags |= (newdescr->flags & NPY_FROM_FIELDS); } @@ -2281,12 +2266,8 @@ arraydescr_new(PyTypeObject *NPY_UNUSED(subtype), return NULL; } - if (align) { - if (!PyArray_DescrAlignConverter(odescr, &conv)) { - return NULL; - } - } - else if (!PyArray_DescrConverter(odescr, &conv)) { + conv = _arraydescr_run_converter(odescr, align); + if (conv == NULL) { return NULL; } |