DEP, ENH: deprecate UPDATEIFCOPY (except for nditer) and replace with WRITEBACKIFCOPY (#9639)

* ENH: add API to resolve UPDATEONCOPY outside dealloc, test and use

* BUG: Fix usage of keyword "from" as argument name for "can_cast".

Also removed inconsistency between the second argument name
between documentation ("totype") and code ("to").

* UPDATEIFCOPY -> WRITEBACKIFCOPY, documentation

* fixes for review

* review2, fix new test

* fix new test for using self.assert_deprecated

* change deprecation logic as per review

* new logic exposed places where PyArray_ResolveWritebackIfCopy not called

* deprecate PyArray_XDECREF_ERR in favor of PyArray_DiscardWritebackIfCopy

* code review changes

* clean up merge cruft

* fix from review

* fixes from review

* extend the release note

6 files changed, 88 insertions, 37 deletions

diff --git a/doc/source/reference/arrays.indexing.rst b/doc/source/reference/arrays.indexing.rst
index ae95517b6..c41a8df56 100644
--- a/doc/source/reference/arrays.indexing.rst
+++ b/doc/source/reference/arrays.indexing.rst
@@ -170,7 +170,7 @@ concepts to remember include:
 
 .. data:: newaxis
 
-   The :const:`newaxis` object can be used in all slicing operations to 
+   The :const:`newaxis` object can be used in all slicing operations to
    create an axis of length one. :const:`newaxis` is an alias for
    'None', and 'None' can be used in place of this with the same result.
 
@@ -503,7 +503,7 @@ dictionary-like.
 Indexing ``x['field-name']`` returns a new :term:`view` to the array,
 which is of the same shape as *x* (except when the field is a
 sub-array) but of data type ``x.dtype['field-name']`` and contains
-only the part of the data in the specified field. Also 
+only the part of the data in the specified field. Also
 :ref:`record array <arrays.classes.rec>` scalars can be "indexed" this way.
 
 Indexing into a structured array can also be done with a list of field names,
diff --git a/doc/source/reference/c-api.array.rst b/doc/source/reference/c-api.array.rst
index 7e42d3c5f..d3aa7cafb 100644
--- a/doc/source/reference/c-api.array.rst
+++ b/doc/source/reference/c-api.array.rst
@@ -76,9 +76,10 @@ sub-types).
     your own memory, you should use the function :c:func:`PyArray_SetBaseObject`
     to set the base to an object which owns the memory.
 
-    If the :c:data:`NPY_ARRAY_UPDATEIFCOPY` flag is set, it has a different
+    If the (deprecated) :c:data:`NPY_ARRAY_UPDATEIFCOPY` or the
+    :c:data:`NPY_ARRAY_WRITEBACKIFCOPY` flags are set, it has a different
     meaning, namely base is the array into which the current array will
-    be copied upon destruction. This overloading of the base property
+    be copied upon copy resolution. This overloading of the base property
     for two functions is likely to change in a future version of NumPy.
 
 .. c:function:: PyArray_Descr *PyArray_DESCR(PyArrayObject* arr)
@@ -217,9 +218,9 @@ From scratch
     can be non-zero to indicate a Fortran-style contiguous array. If
     *data* is not ``NULL``, then it is assumed to point to the memory
     to be used for the array and the *flags* argument is used as the
-    new flags for the array (except the state of :c:data:`NPY_OWNDATA`
-    and :c:data:`NPY_ARRAY_UPDATEIFCOPY` flags of the new array will
-    be reset).
+    new flags for the array (except the state of :c:data:`NPY_OWNDATA`,
+    :c:data:`NPY_ARRAY_WRITEBACKIFCOPY` and :c:data:`NPY_ARRAY_UPDATEIFCOPY`
+    flags of the new array will be reset).
 
     In addition, if *data* is non-NULL, then *strides* can
     also be provided. If *strides* is ``NULL``, then the array strides
@@ -444,19 +445,26 @@ From other objects
         safely.  Without this flag, a data cast will occur only if it
         can be done safely, otherwise an error is raised.
 
-    .. c:var:: NPY_ARRAY_UPDATEIFCOPY
+    .. c:var:: NPY_ARRAY_WRITEBACKIFCOPY
 
         If *op* is already an array, but does not satisfy the
         requirements, then a copy is made (which will satisfy the
         requirements). If this flag is present and a copy (of an object
         that is already an array) must be made, then the corresponding
-        :c:data:`NPY_ARRAY_UPDATEIFCOPY` flag is set in the returned
-        copy and *op* is made to be read-only. When the returned copy
-        is deleted (presumably after your calculations are complete),
-        its contents will be copied back into *op* and the *op* array
+        :c:data:`NPY_ARRAY_WRITEBACKIFCOPY` flag is set in the returned
+        copy and *op* is made to be read-only. You must be sure to call
+        :c:func:`PyArray_ResolveWritebackIfCopy` to copy the contents
+        back into *op* and the *op* array
         will be made writeable again. If *op* is not writeable to begin
         with, or if it is not already an array, then an error is raised.
 
+    .. c:var:: NPY_ARRAY_UPDATEIFCOPY
+
+        Deprecated. Use :c:data:`NPY_ARRAY_WRITEBACKIFCOPY`, which is similar.
+        This flag "automatically" copies the data back when the returned
+        array is deallocated, which is not supported in all python
+        implementations.
+
     .. c:var:: NPY_ARRAY_BEHAVED
 
         :c:data:`NPY_ARRAY_ALIGNED` \| :c:data:`NPY_ARRAY_WRITEABLE`
@@ -502,12 +510,14 @@ From other objects
     .. c:var:: NPY_ARRAY_INOUT_ARRAY
 
         :c:data:`NPY_ARRAY_C_CONTIGUOUS` \| :c:data:`NPY_ARRAY_WRITEABLE` \|
-        :c:data:`NPY_ARRAY_ALIGNED` \| :c:data:`NPY_ARRAY_UPDATEIFCOPY`
+        :c:data:`NPY_ARRAY_ALIGNED` \| :c:data:`NPY_ARRAY_WRITEBACKIFCOPY` \|
+        :c:data:`NPY_ARRAY_UPDATEIFCOPY`
 
     .. c:var:: NPY_ARRAY_INOUT_FARRAY
 
         :c:data:`NPY_ARRAY_F_CONTIGUOUS` \| :c:data:`NPY_ARRAY_WRITEABLE` \|
-        :c:data:`NPY_ARRAY_ALIGNED` \| :c:data:`NPY_ARRAY_UPDATEIFCOPY`
+        :c:data:`NPY_ARRAY_ALIGNED` \| :c:data:`NPY_ARRAY_WRITEBACKIFCOPY` \|
+        :c:data:`NPY_ARRAY_UPDATEIFCOPY`
 
 .. c:function:: int PyArray_GetArrayParamsFromObject( \
         PyObject* op, PyArray_Descr* requested_dtype, npy_bool writeable, \
@@ -752,7 +762,8 @@ From other objects
     :c:data:`NPY_ARRAY_C_CONTIGUOUS`, :c:data:`NPY_ARRAY_F_CONTIGUOUS`,
     :c:data:`NPY_ARRAY_ALIGNED`, :c:data:`NPY_ARRAY_WRITEABLE`,
     :c:data:`NPY_ARRAY_NOTSWAPPED`, :c:data:`NPY_ARRAY_ENSURECOPY`,
-    :c:data:`NPY_ARRAY_UPDATEIFCOPY`, :c:data:`NPY_ARRAY_FORCECAST`, and
+    :c:data:`NPY_ARRAY_WRITEBACKIFCOPY`, :c:data:`NPY_ARRAY_UPDATEIFCOPY`,
+    :c:data:`NPY_ARRAY_FORCECAST`, and
     :c:data:`NPY_ARRAY_ENSUREARRAY`. Standard combinations of flags can also
     be used:
 
@@ -1336,6 +1347,21 @@ Special functions for NPY_OBJECT
     decrement all the items in the object array prior to calling this
     function.
 
+.. c:function:: int PyArray_SetUpdateIfCopyBase(PyArrayObject* arr, PyArrayObject* base)
+
+    Precondition: ``arr`` is a copy of ``base`` (though possibly with different
+    strides, ordering, etc.) Set the UPDATEIFCOPY flag and ``arr->base`` so
+    that when ``arr`` is destructed, it will copy any changes back to ``base``.
+    DEPRECATED, use :c:func:`PyArray_SetWritebackIfCopyBase``.
+
+.. c:function:: int PyArray_SetWritebackIfCopyBase(PyArrayObject* arr, PyArrayObject* base)
+
+    Precondition: ``arr`` is a copy of ``base`` (though possibly with different
+    strides, ordering, etc.) Sets the WRITEBACKIFCOPY flag and ``arr->base``, and
+    set ``base`` to READONLY. Call PyArray_ResolveWritebackIfCopy before calling
+    :c:func:`PyArray_DECREF`` in order copy any changes back to ``base`` and
+    reset the READONLY flag.
+
 
 Array flags
 -----------
@@ -1415,24 +1441,33 @@ of the constant names is deprecated in 1.7.
     Notice that the above 3 flags are defined so that a new, well-
     behaved array has these flags defined as true.
 
-.. c:var:: NPY_ARRAY_UPDATEIFCOPY
+.. c:var:: NPY_ARRAY_WRITEBACKIFCOPY
 
     The data area represents a (well-behaved) copy whose information
-    should be transferred back to the original when this array is deleted.
+    should be transferred back to the original when
+    :c:func:`PyArray_ResolveWritebackIfCopy` is called.
 
     This is a special flag that is set if this array represents a copy
     made because a user required certain flags in
     :c:func:`PyArray_FromAny` and a copy had to be made of some other
     array (and the user asked for this flag to be set in such a
     situation). The base attribute then points to the "misbehaved"
-    array (which is set read_only). When the array with this flag set
-    is deallocated, it will copy its contents back to the "misbehaved"
+    array (which is set read_only). :c:func`PyArray_ResolveWritebackIfCopy`
+    will copy its contents back to the "misbehaved"
     array (casting if necessary) and will reset the "misbehaved" array
     to :c:data:`NPY_ARRAY_WRITEABLE`. If the "misbehaved" array was not
     :c:data:`NPY_ARRAY_WRITEABLE` to begin with then :c:func:`PyArray_FromAny`
-    would have returned an error because :c:data:`NPY_ARRAY_UPDATEIFCOPY`
+    would have returned an error because :c:data:`NPY_ARRAY_WRITEBACKIFCOPY`
     would not have been possible.
 
+.. c:var:: NPY_ARRAY_UPDATEIFCOPY
+
+    A deprecated version of :c:data:`NPY_ARRAY_WRITEBACKIFCOPY` which
+    depends upon ``dealloc`` to trigger the writeback. For backwards
+    compatibility, :c:func:`PyArray_ResolveWritebackIfCopy` is called at
+    ``dealloc`` but relying
+    on that behavior is deprecated and not supported in PyPy.
+
 :c:func:`PyArray_UpdateFlags` (obj, flags) will update the ``obj->flags``
 for ``flags`` which can be any of :c:data:`NPY_ARRAY_C_CONTIGUOUS`,
 :c:data:`NPY_ARRAY_F_CONTIGUOUS`, :c:data:`NPY_ARRAY_ALIGNED`, or
@@ -1513,7 +1548,8 @@ For all of these macros *arr* must be an instance of a (subclass of)
     combinations of the possible flags an array can have:
     :c:data:`NPY_ARRAY_C_CONTIGUOUS`, :c:data:`NPY_ARRAY_F_CONTIGUOUS`,
     :c:data:`NPY_ARRAY_OWNDATA`, :c:data:`NPY_ARRAY_ALIGNED`,
-    :c:data:`NPY_ARRAY_WRITEABLE`, :c:data:`NPY_ARRAY_UPDATEIFCOPY`.
+    :c:data:`NPY_ARRAY_WRITEABLE`, :c:data:`NPY_ARRAY_WRITEBACKIFCOPY`,
+    :c:data:`NPY_ARRAY_UPDATEIFCOPY`.
 
 .. c:function:: PyArray_IS_C_CONTIGUOUS(arr)
 
@@ -3432,13 +3468,25 @@ Miscellaneous Macros
 
     Returns the reference count of any Python object.
 
-.. c:function:: PyArray_XDECREF_ERR(PyObject \*obj)
+.. c:function:: PyArray_DiscardWritebackIfCopy(PyObject* obj)
+
+    Reset the :c:data:`NPY_ARRAY_WRITEBACKIFCOPY` and deprecated
+    :c:data:`NPY_ARRAY_UPDATEIFCOPY` flag. Also resets the
+    :c:data:`NPY_ARRAY_WRITEABLE` flag on the base object. This is
+    useful for recovering from an error condition when
+    writeback semantics are used, but will lead to wrong results.
+
+.. c:function:: PyArray_XDECREF_ERR(PyObject* obj)
+
+    Deprecated in 1.14, use :c:func:`PyArray_DiscardWritebackIfCopy`
+    followed by ``Py_XDECREF``
 
-    DECREF's an array object which may have the :c:data:`NPY_ARRAY_UPDATEIFCOPY`
+    DECREF's an array object which may have the (deprecated)
+    :c:data:`NPY_ARRAY_UPDATEIFCOPY` or :c:data:`NPY_ARRAY_WRITEBACKIFCOPY`
     flag set without causing the contents to be copied back into the
     original array. Resets the :c:data:`NPY_ARRAY_WRITEABLE` flag on the base
     object. This is useful for recovering from an error condition when
-    :c:data:`NPY_ARRAY_UPDATEIFCOPY` is used.
+    writeback semantics are used, but will lead to wrong results.
 
 
 Enumerated Types
diff --git a/doc/source/reference/c-api.types-and-structures.rst b/doc/source/reference/c-api.types-and-structures.rst
index 255c348f9..dcebd1ede 100644
--- a/doc/source/reference/c-api.types-and-structures.rst
+++ b/doc/source/reference/c-api.types-and-structures.rst
@@ -133,10 +133,11 @@ PyArray_Type
     is related to this array. There are two use cases: 1) If this array
     does not own its own memory, then base points to the Python object
     that owns it (perhaps another array object), 2) If this array has
-    the :c:data:`NPY_ARRAY_UPDATEIFCOPY` flag set, then this array is
-    a working copy of a "misbehaved" array. As soon as this array is
-    deleted, the array pointed to by base will be updated with the
-    contents of this array.
+    the (deprecated) :c:data:`NPY_ARRAY_UPDATEIFCOPY` or 
+    :c:data:NPY_ARRAY_WRITEBACKIFCOPY`: flag set, then this array is
+    a working copy of a "misbehaved" array. When 
+    ``PyArray_ResolveWritebackIfCopy`` is called, the array pointed to by base
+    will be updated with the contents of this array.
 
 .. c:member:: PyArray_Descr *PyArrayObject.descr
 
@@ -153,8 +154,8 @@ PyArray_Type
     Flags indicating how the memory pointed to by data is to be
     interpreted. Possible flags are :c:data:`NPY_ARRAY_C_CONTIGUOUS`,
     :c:data:`NPY_ARRAY_F_CONTIGUOUS`, :c:data:`NPY_ARRAY_OWNDATA`,
-    :c:data:`NPY_ARRAY_ALIGNED`, :c:data:`NPY_ARRAY_WRITEABLE`, and
-    :c:data:`NPY_ARRAY_UPDATEIFCOPY`.
+    :c:data:`NPY_ARRAY_ALIGNED`, :c:data:`NPY_ARRAY_WRITEABLE`,
+    :c:data:`NPY_ARRAY_WRITEBACKIFCOPY`, and :c:data:`NPY_ARRAY_UPDATEIFCOPY`.
 
 .. c:member:: PyObject *PyArrayObject.weakreflist
 
diff --git a/doc/source/reference/internals.code-explanations.rst b/doc/source/reference/internals.code-explanations.rst
index 94e827429..ca81e1676 100644
--- a/doc/source/reference/internals.code-explanations.rst
+++ b/doc/source/reference/internals.code-explanations.rst
@@ -368,8 +368,9 @@ The output arguments (if any) are then processed and any missing
 return arrays are constructed. If any provided output array doesn't
 have the correct type (or is mis-aligned) and is smaller than the
 buffer size, then a new output array is constructed with the special
-UPDATEIFCOPY flag set so that when it is DECREF'd on completion of the
-function, its contents will be copied back into the output array.
+:c:data:`WRITEBACKIFCOPY` flag set. At the end of the function,
+:c:func:`PyArray_ResolveWritebackIfCopy` is called so that 
+its contents will be copied back into the output array.
 Iterators for the output arguments are then processed.
 
 Finally, the decision is made about how to execute the looping
@@ -508,10 +509,11 @@ of a different shape depending on whether the method is reduce,
 accumulate, or reduceat. If an output array is already provided, then
 it's shape is checked. If the output array is not C-contiguous,
 aligned, and of the correct data type, then a temporary copy is made
-with the UPDATEIFCOPY flag set. In this way, the methods will be able
+with the WRITEBACKIFCOPY flag set. In this way, the methods will be able
 to work with a well-behaved output array but the result will be copied
-back into the true output array when the method computation is
-complete. Finally, iterators are set up to loop over the correct axis
+back into the true output array when :c:func:`PyArray_ResolveWritebackIfCopy`
+is called at function completion.
+Finally, iterators are set up to loop over the correct axis
 (depending on the value of axis provided to the method) and the setup
 routine returns to the actual computation routine.
 
diff --git a/doc/source/reference/routines.io.rst b/doc/source/reference/routines.io.rst
index 00cafebc2..5df590f17 100644
--- a/doc/source/reference/routines.io.rst
+++ b/doc/source/reference/routines.io.rst
@@ -14,7 +14,7 @@ NumPy binary files (NPY, NPZ)
    savez_compressed
 
 The format of these binary file types is documented in
-http://docs.scipy.org/doc/numpy/neps/npy-format.html  
+http://docs.scipy.org/doc/numpy/neps/npy-format.html
 
 Text files
 ----------
diff --git a/doc/source/reference/routines.set.rst b/doc/source/reference/routines.set.rst
index 0089fb3e9..b12d3d5f5 100644
--- a/doc/source/reference/routines.set.rst
+++ b/doc/source/reference/routines.set.rst
@@ -17,7 +17,7 @@ Boolean operations
 
    in1d
    intersect1d
-   isin   
+   isin
    setdiff1d
    setxor1d
    union1d