Merge branch 'pull-141'

* pull-141: (167 commits)
  ENH: missingdata: Make PyArray_Converter and PyArray_OutputConverter safer for legacy code
  DOC: missingdata: Add a mention of the design NEP, and masks vs bitpatterns
  DOC: missingdata: Updates from pull request feedback
  DOC: missingdata: Updates based on pull request feedback
  ENH: nditer: Change the Python nditer exposure to automatically add NPY_ITER_USE_MASKNA
  ENH: missingdata: Make comparisons with NA return NA(dtype='bool')
  BLD: core: onefile build fix and Python3 compatibility change
  DOC: Mention the update to np.all and np.any in the release notes
  TST: dtype: Adjust void dtype test to pass without raising a zero-size exception
  STY: Remove trailing whitespace
  TST: missingdata: Write some tests for the np.any and np.all NA behavior
  ENH: missingdata: Make numpy.all follow the NA && False == False rule
  ENH: missingdata: Make numpy.all follow the NA || True == True rule
  DOC: missingdata: Also show what assigning a non-NA value does in each case
  DOC: missingdata: Add introductory documentation for NA-masked arrays
  ENH: core: Rename PyArrayObject_fieldaccess to PyArrayObject_fields
  DOC: missingdata: Some tweaks to the NA mask documentation
  DOC: missingdata: Add example of a C-API function supporting NA masks
  DOC: missingdata: Documenting C API for NA-masked arrays
  ENH: missingdata: Finish adding C-API access to the NpyNA object
  ...

1 files changed, 227 insertions, 105 deletions

diff --git a/doc/source/reference/c-api.array.rst b/doc/source/reference/c-api.array.rst
index 4b57945e9..46a215a12 100644
--- a/doc/source/reference/c-api.array.rst
+++ b/doc/source/reference/c-api.array.rst
@@ -21,13 +21,30 @@ Array structure and data access
 -------------------------------
 
 These macros all access the :ctype:`PyArrayObject` structure members. The input
-argument, obj, can be any :ctype:`PyObject *` that is directly interpretable
+argument, arr, can be any :ctype:`PyObject *` that is directly interpretable
 as a :ctype:`PyArrayObject *` (any instance of the :cdata:`PyArray_Type` and its
 sub-types).
 
-.. cfunction:: void *PyArray_DATA(PyObject *obj)
+.. cfunction:: int PyArray_NDIM(PyArrayObject *arr)
 
-.. cfunction:: char *PyArray_BYTES(PyObject *obj)
+    The number of dimensions in the array.
+
+.. cfunction:: npy_intp *PyArray_DIMS(PyArrayObject *arr)
+
+    Returns a pointer to the dimensions/shape of the array. The
+    number of elements matches the number of dimensions
+    of the array.
+
+.. cfunction:: npy_intp *PyArray_SHAPE(PyArrayObject *arr)
+
+    .. versionadded:: 1.7
+
+    A synonym for PyArray_DIMS, named to be consistent with the
+    'shape' usage within Python.
+
+.. cfunction:: void *PyArray_DATA(PyArrayObject *arr)
+
+.. cfunction:: char *PyArray_BYTES(PyArrayObject *arr)
 
     These two macros are similar and obtain the pointer to the
     data-buffer for the array. The first macro can (and should be)
@@ -36,19 +53,21 @@ sub-types).
     array then be sure you understand how to access the data in the
     array to avoid memory and/or alignment problems.
 
-.. cfunction:: npy_intp *PyArray_DIMS(PyObject *arr)
+.. cfunction:: npy_intp *PyArray_STRIDES(PyArrayObject* arr)
 
-.. cfunction:: npy_intp *PyArray_STRIDES(PyObject* arr)
+    Returns a pointer to the strides of the array. The
+    number of elements matches the number of dimensions
+    of the array.
 
-.. cfunction:: npy_intp PyArray_DIM(PyObject* arr, int n)
+.. cfunction:: npy_intp PyArray_DIM(PyArrayObject* arr, int n)
 
     Return the shape in the *n* :math:`^{\textrm{th}}` dimension.
 
-.. cfunction:: npy_intp PyArray_STRIDE(PyObject* arr, int n)
+.. cfunction:: npy_intp PyArray_STRIDE(PyArrayObject* arr, int n)
 
     Return the stride in the *n* :math:`^{\textrm{th}}` dimension.
 
-.. cfunction:: PyObject *PyArray_BASE(PyObject* arr)
+.. cfunction:: PyObject *PyArray_BASE(PyArrayObject* arr)
 
     This returns the base object of the array. In most cases, this
     means the object which owns the memory the array is pointing at.
@@ -62,40 +81,97 @@ sub-types).
     be copied upon destruction. This overloading of the base property
     for two functions is likely to change in a future version of NumPy.
 
-.. cfunction:: PyArray_Descr *PyArray_DESCR(PyObject* arr)
+.. cfunction:: PyArray_Descr *PyArray_DESCR(PyArrayObject* arr)
+
+    Returns a borrowed reference to the dtype property of the array.
+
+.. cfunction:: PyArray_Descr *PyArray_DTYPE(PyArrayObject* arr)
+
+    .. versionadded:: 1.7
+
+    A synonym for PyArray_DESCR, named to be consistent with the
+    'dtype' usage within Python.
+
+.. cfunction:: npy_bool PyArray_HASMASKNA(PyArrayObject* arr)
+
+    .. versionadded:: 1.7
+
+    Returns true if the array has an NA-mask, false otherwise.
+
+.. cfunction:: PyArray_Descr *PyArray_MASKNA_DTYPE(PyArrayObject* arr)
+
+    .. versionadded:: 1.7
+
+    Returns a borrowed reference to the dtype property for the NA mask
+    of the array, or NULL if the array has no NA mask. This function does
+    not raise an exception when it returns NULL, it is simply returning
+    the appropriate field.
+
+.. cfunction:: char *PyArray_MASKNA_DATA(PyArrayObject* arr)
 
-.. cfunction:: int PyArray_FLAGS(PyObject* arr)
+    .. versionadded:: 1.7
 
-.. cfunction:: int PyArray_ITEMSIZE(PyObject* arr)
+    Returns a pointer to the raw data for the NA mask of the array,
+    or NULL if the array has no NA mask. This function does
+    not raise an exception when it returns NULL, it is simply returning
+    the appropriate field.
+
+.. cfunction:: npy_intp *PyArray_MASKNA_STRIDES(PyArrayObject* arr)
+
+    .. versionadded:: 1.7
+
+    Returns a pointer to strides of the NA mask of the array, If the
+    array has no NA mask, the values contained in the array will be
+    invalid. The shape of the NA mask is identical to the shape of the
+    array itself, so the number of strides is always the same as the
+    number of array dimensions.
+
+.. cfunction:: void PyArray_ENABLEFLAGS(PyArrayObject* arr, int flags)
+
+    .. versionadded:: 1.7
+
+    Enables the specified array flags. This function does no validation,
+    and assumes that you know what you're doing.
+
+.. cfunction:: void PyArray_CLEARFLAGS(PyArrayObject* arr, int flags)
+
+    .. versionadded:: 1.7
+
+    Clears the specified array flags. This function does no validation,
+    and assumes that you know what you're doing.
+
+.. cfunction:: int PyArray_FLAGS(PyArrayObject* arr)
+
+.. cfunction:: int PyArray_ITEMSIZE(PyArrayObject* arr)
 
     Return the itemsize for the elements of this array.
 
-.. cfunction:: int PyArray_TYPE(PyObject* arr)
+.. cfunction:: int PyArray_TYPE(PyArrayObject* arr)
 
     Return the (builtin) typenumber for the elements of this array.
 
-.. cfunction:: PyObject *PyArray_GETITEM(PyObject* arr, void* itemptr)
+.. cfunction:: PyObject *PyArray_GETITEM(PyArrayObject* arr, void* itemptr)
 
     Get a Python object from the ndarray, *arr*, at the location
     pointed to by itemptr. Return ``NULL`` on failure.
 
-.. cfunction:: int PyArray_SETITEM(PyObject* arr, void* itemptr, PyObject* obj)
+.. cfunction:: int PyArray_SETITEM(PyArrayObject* arr, void* itemptr, PyObject* obj)
 
     Convert obj and place it in the ndarray, *arr*, at the place
     pointed to by itemptr. Return -1 if an error occurs or 0 on
     success.
 
-.. cfunction:: npy_intp PyArray_SIZE(PyObject* arr)
+.. cfunction:: npy_intp PyArray_SIZE(PyArrayObject* arr)
 
     Returns the total size (in number of elements) of the array.
 
-.. cfunction:: npy_intp PyArray_Size(PyObject* obj)
+.. cfunction:: npy_intp PyArray_Size(PyArrayObject* obj)
 
     Returns 0 if *obj* is not a sub-class of bigndarray. Otherwise,
     returns the total number of elements in the array. Safer version
     of :cfunc:`PyArray_SIZE` (*obj*).
 
-.. cfunction:: npy_intp PyArray_NBYTES(PyObject* arr)
+.. cfunction:: npy_intp PyArray_NBYTES(PyArrayObject* arr)
 
     Returns the total number of bytes consumed by the array.
 
@@ -149,48 +225,64 @@ From scratch
 .. cfunction:: PyObject* PyArray_NewFromDescr(PyTypeObject* subtype, PyArray_Descr* descr, int nd, npy_intp* dims, npy_intp* strides, void* data, int flags, PyObject* obj)
 
     This is the main array creation function. Most new arrays are
-    created with this flexible function. The returned object is an
-    object of Python-type *subtype*, which must be a subtype of
-    :cdata:`PyArray_Type`. The array has *nd* dimensions, described by
-    *dims*. The data-type descriptor of the new array is *descr*. If
-    *subtype* is not :cdata:`&PyArray_Type` (*e.g.* a Python subclass of
-    the ndarray), then *obj* is the object to pass to the
-    :obj:`__array_finalize__` method of the subclass. If *data* is
-    ``NULL``, then new memory will be allocated and *flags* can be
-    non-zero to indicate a Fortran-style contiguous array. If *data*
-    is not ``NULL``, then it is assumed to point to the memory to be
-    used for the array and the *flags* argument is used as the new
-    flags for the array (except the state of :cdata:`NPY_OWNDATA` and
-    :cdata:`NPY_ARRAY_UPDATEIFCOPY` flags of the new array will be reset). In
-    addition, if *data* is non-NULL, then *strides* can also be
-    provided. If *strides* is ``NULL``, then the array strides are
-    computed as C-style contiguous (default) or Fortran-style
+    created with this flexible function.
+
+    The returned object is an object of Python-type *subtype*, which
+    must be a subtype of :cdata:`PyArray_Type`.  The array has *nd*
+    dimensions, described by *dims*. The data-type descriptor of the
+    new array is *descr*.
+
+    If *subtype* is of an array subclass instead of the base
+    :cdata:`&PyArray_Type`, then *obj* is the object to pass to
+    the :obj:`__array_finalize__` method of the subclass.
+
+    If *data* is ``NULL``, then new memory will be allocated and *flags*
+    can be non-zero to indicate a Fortran-style contiguous array. If
+    *data* is not ``NULL``, then it is assumed to point to the memory
+    to be used for the array and the *flags* argument is used as the
+    new flags for the array (except the state of :cdata:`NPY_OWNDATA`
+    and :cdata:`NPY_ARRAY_UPDATEIFCOPY` flags of the new array will
+    be reset).
+
+    In addition, if *data* is non-NULL, then *strides* can
+    also be provided. If *strides* is ``NULL``, then the array strides
+    are computed as C-style contiguous (default) or Fortran-style
     contiguous (*flags* is nonzero for *data* = ``NULL`` or *flags* &
-    :cdata:`NPY_ARRAY_F_CONTIGUOUS` is nonzero non-NULL *data*). Any provided
-    *dims* and *strides* are copied into newly allocated dimension and
-    strides arrays for the new array object.
+    :cdata:`NPY_ARRAY_F_CONTIGUOUS` is nonzero non-NULL *data*). Any
+    provided *dims* and *strides* are copied into newly allocated
+    dimension and strides arrays for the new array object.
+
+    Because the flags are ignored when *data* is NULL, you cannot
+    create a new array from scratch with an NA mask. If one is desired,
+    call the function :cfunc:`PyArray_AllocateMaskNA` after the array
+    is created.
 
 .. cfunction:: PyObject* PyArray_NewLikeArray(PyArrayObject* prototype, NPY_ORDER order, PyArray_Descr* descr, int subok)
 
     .. versionadded:: 1.6
 
-   This function steals a reference to *descr* if it is not NULL.
+    This function steals a reference to *descr* if it is not NULL.
+
+    This array creation routine allows for the convenient creation of
+    a new array matching an existing array's shapes and memory layout,
+    possibly changing the layout and/or data type.
 
-   This array creation routine allows for the convenient creation of
-   a new array matching an existing array's shapes and memory layout,
-   possibly changing the layout and/or data type.
+    When *order* is :cdata:`NPY_ANYORDER`, the result order is
+    :cdata:`NPY_FORTRANORDER` if *prototype* is a fortran array,
+    :cdata:`NPY_CORDER` otherwise.  When *order* is
+    :cdata:`NPY_KEEPORDER`, the result order matches that of *prototype*, even
+    when the axes of *prototype* aren't in C or Fortran order.
 
-   When *order* is :cdata:`NPY_ANYORDER`, the result order is
-   :cdata:`NPY_FORTRANORDER` if *prototype* is a fortran array,
-   :cdata:`NPY_CORDER` otherwise.  When *order* is
-   :cdata:`NPY_KEEPORDER`, the result order matches that of *prototype*, even
-   when the axes of *prototype* aren't in C or Fortran order.
+    If *descr* is NULL, the data type of *prototype* is used.
 
-   If *descr* is NULL, the data type of *prototype* is used.
+    If *subok* is 1, the newly created array will use the sub-type of
+    *prototype* to create the new array, otherwise it will create a
+    base-class array.
 
-   If *subok* is 1, the newly created array will use the sub-type of
-   *prototype* to create the new array, otherwise it will create a
-   base-class array.
+    The newly allocated array does not have an NA mask even if the
+    *prototype* provided does. If an NA mask is desired in the array,
+    call the function :cfunc:`PyArray_AllocateMaskNA` after the array
+    is created.
 
 .. cfunction:: PyObject* PyArray_New(PyTypeObject* subtype, int nd, npy_intp* dims, int type_num, npy_intp* strides, void* data, int itemsize, int flags, PyObject* obj)
 
@@ -278,7 +370,9 @@ From scratch
     increments of ``step``. Equivalent to arange( ``start``,
     ``stop``, ``step``, ``typenum`` ).
 
-.. cfunction:: int PyArray_SetBaseObject(PyArrayObject *arr, PyObject *obj)
+.. cfunction:: int PyArray_SetBaseObject(PyArrayObject* arr, PyObject* obj)
+
+    .. versionadded:: 1.7
 
     If you construct an array by passing in your own memory buffer as
     a parameter, you need to set the array's `base` property to ensure
@@ -377,6 +471,31 @@ From other objects
         with, then an error is raised. If *op* is not already an array,
         then this flag has no effect.
 
+    .. cvar:: NPY_ARRAY_MASKNA
+
+        .. versionadded:: 1.7
+
+        Make sure the array has an NA mask associated with its data.
+
+    .. cvar:: NPY_ARRAY_OWNMASKNA
+
+        .. versionadded:: 1.7
+
+        Make sure the array has an NA mask which it owns
+        associated with its data.
+
+    .. cvar:: NPY_ARRAY_ALLOWNA
+
+        .. versionadded:: 1.7
+
+        To prevent simple errors from slipping in, arrays with NA
+        masks are not permitted to pass through by default. Instead
+        an exception is raised indicating the operation doesn't support
+        NA masks yet. In order to enable NA mask support, this flag
+        must be passed in to allow the NA mask through, signalling that
+        the later code is written appropriately to handle NA mask
+        semantics.
+
     .. cvar:: NPY_ARRAY_BEHAVED
 
         :cdata:`NPY_ARRAY_ALIGNED` \| :cdata:`NPY_ARRAY_WRITEABLE`
@@ -1292,6 +1411,24 @@ or :cdata:`NPY_ARRAY_F_CONTIGUOUS` can be determined by the ``strides``,
     would have returned an error because :cdata:`NPY_ARRAY_UPDATEIFCOPY`
     would not have been possible.
 
+.. cvar:: NPY_ARRAY_MASKNA
+
+    If this flag is enabled, the array has an NA mask associated with
+    the data. C code which interacts with the NA mask must follow
+    specific semantic rules about when to overwrite data and when not
+    to. The mask can be accessed through the functions
+    :cfunc:`PyArray_MASKNA_DTYPE`, :cfunc:`PyArray_MASKNA_DATA`, and
+    :cfunc:`PyArray_MASKNA_STRIDES`.
+
+.. cvar:: NPY_ARRAY_OWNMASKNA
+
+    If this flag is enabled, the array owns its own NA mask. If it is not
+    enabled, the NA mask is a view into a different array's NA mask.
+
+    In order to ensure that an array owns its own NA mask, you can
+    call :cfunc:`PyArray_AllocateMaskNA` with the parameter *ownmaskna*
+    set to 1.
+
 :cfunc:`PyArray_UpdateFlags` (obj, flags) will update the ``obj->flags``
 for ``flags`` which can be any of :cdata:`NPY_ARRAY_C_CONTIGUOUS`,
 :cdata:`NPY_ARRAY_F_CONTIGUOUS`, :cdata:`NPY_ARRAY_ALIGNED`, or
@@ -1536,18 +1673,20 @@ Conversion
     copied into every location. A -1 is returned if an error occurs,
     otherwise 0 is returned.
 
-.. cfunction:: PyObject* PyArray_View(PyArrayObject* self, PyArray_Descr* dtype)
+.. cfunction:: PyObject* PyArray_View(PyArrayObject* self, PyArray_Descr* dtype, PyTypeObject *ptype)
 
-    Equivalent to :meth:`ndarray.view` (*self*, *dtype*). Return a new view of
-    the array *self* as possibly a different data-type, *dtype*. If
-    *dtype* is ``NULL``, then the returned array will have the same
-    data type as *self*. The new data-type must be consistent with
-    the size of *self*. Either the itemsizes must be identical, or
-    *self* must be single-segment and the total number of bytes must
-    be the same.  In the latter case the dimensions of the returned
-    array will be altered in the last (or first for Fortran-style
-    contiguous arrays) dimension. The data area of the returned array
-    and self is exactly the same.
+    Equivalent to :meth:`ndarray.view` (*self*, *dtype*). Return a new
+    view of the array *self* as possibly a different data-type, *dtype*,
+    and different array subclass *ptype*.
+
+    If *dtype* is ``NULL``, then the returned array will have the same
+    data type as *self*. The new data-type must be consistent with the
+    size of *self*. Either the itemsizes must be identical, or *self* must
+    be single-segment and the total number of bytes must be the same.
+    In the latter case the dimensions of the returned array will be
+    altered in the last (or first for Fortran-style contiguous arrays)
+    dimension. The data area of the returned array and self is exactly
+    the same.
 
 
 Shape Manipulation
@@ -2152,50 +2291,6 @@ an element copier function as a primitive.::
     A macro which calls the auxdata's clone function appropriately,
     returning a deep copy of the auxiliary data.
 
-Masks for Selecting Elements to Modify
---------------------------------------
-
-.. versionadded:: 1.7.0
-
-The array iterator, :ctype:`NpyIter`, has some new flags which
-allow control over which elements are intended to be modified,
-providing the ability to do masking even when doing casts to a buffer
-of a different type. Some inline functions have been added
-to facilitate consistent usage of these masks.
-
-A mask dtype can be one of three different possibilities. It can
-be :cdata:`NPY_BOOL`, :cdata:`NPY_MASK`, or a struct dtype whose
-fields are all mask dtypes.
-
-A mask of :cdata:`NPY_BOOL` can just indicate True, with underlying
-value 1, for an element that is exposed, and False, with underlying
-value 0, for an element that is hidden.
-
-A mask of :cdata:`NPY_MASK` can additionally carry a payload which
-is a value from 0 to 127. This allows for missing data implementations
-based on such masks to support multiple reasons for data being missing.
-
-A mask of a struct dtype can only pair up with another struct dtype
-with the same field names. In this way, each field of the mask controls
-the masking for the corresponding field in the associated data array.
-
-Inline functions to work with masks are as follows.
-
-.. cfunction:: npy_bool NpyMask_IsExposed(npy_mask mask)
-
-    Returns true if the data element corresponding to the mask element
-    can be modified, false if not.
-
-.. cfunction:: npy_uint8 NpyMask_GetPayload(npy_mask mask)
-
-    Returns the payload contained in the mask. The return value
-    is between 0 and 127.
-
-.. cfunction:: npy_mask NpyMask_Create(npy_bool exposed, npy_int8 payload)
-
-    Creates a mask from a flag indicating whether the element is exposed
-    or not and a payload value.
-
 Array Iterators
 ---------------
 
@@ -2442,6 +2537,9 @@ Array Scalars
     if so, returns the appropriate array scalar. It should be used
     whenever 0-dimensional arrays could be returned to Python.
 
+    If *arr* is a 0-dimensional NA-masked array with its value hidden,
+    an instance of :ctype:`NpyNA *` is returned.
+
 .. cfunction:: PyObject* PyArray_Scalar(void* data, PyArray_Descr* dtype, PyObject* itemsize)
 
     Return an array scalar object of the given enumerated *typenum*
@@ -2654,6 +2752,19 @@ to.
     . No matter what is returned, you must DECREF the object returned
     by this routine in *address* when you are done with it.
 
+    If the input is an array with NA support, this will either raise
+    an error if it contains any NAs, or will make a copy of the array
+    without NA support if it does not contain any NAs. Use the function
+    :cfunc:`PyArray_AllowNAConverter` to support NA-arrays directly
+    and more efficiently.
+
+.. cfunction:: int PyArray_AllowConverter(PyObject* obj, PyObject** address)
+
+    This is the same as :cfunc:`PyArray_Converter`, but allows arrays
+    with NA support to pass through untouched. This function was created
+    so that the existing converter could raise errors appropriately
+    for functions which have not been updated with NA support
+
 .. cfunction:: int PyArray_OutputConverter(PyObject* obj, PyArrayObject** address)
 
     This is a default converter for output arrays given to
@@ -2662,6 +2773,17 @@ to.
     *obj*) is TRUE then it is returned in *\*address* without
     incrementing its reference count.
 
+    If the output is an array with NA support, this will raise an error.
+    Use the function :cfunc:`PyArray_OutputAllowNAConverter` to support
+    NA-arrays directly.
+
+.. cfunction:: int PyArray_OutputAllowNAConverter(PyObject* obj, PyArrayObject** address)
+
+    This is the same as :cfunc:`PyArray_OutputConverter`, but allows arrays
+    with NA support to pass through. This function was created
+    so that the existing output converter could raise errors appropriately
+    for functions which have not been updated with NA support
+
 .. cfunction:: int PyArray_IntpConverter(PyObject* obj, PyArray_Dims* seq)
 
     Convert any Python sequence, *obj*, smaller than :cdata:`NPY_MAXDIMS`