diff options
author | Hameer Abbasi <einstein.edison@gmail.com> | 2019-10-10 10:27:03 +0500 |
---|---|---|
committer | Hameer Abbasi <einstein.edison@gmail.com> | 2019-10-10 10:27:03 +0500 |
commit | 7ad12e1d5821a3800f52f701729b3c3b39f49227 (patch) | |
tree | 451a41cdeda24743e12bf57f419118e776654ebc /doc/neps | |
parent | ff181c53352ba54b02bb3560797361c0b3c3ba74 (diff) | |
download | numpy-7ad12e1d5821a3800f52f701729b3c3b39f49227.tar.gz |
Respond to comments by Juan Nunez-Iglesias on the mailing list.
Diffstat (limited to 'doc/neps')
-rw-r--r-- | doc/neps/nep-0031-uarray.rst | 75 |
1 files changed, 56 insertions, 19 deletions
diff --git a/doc/neps/nep-0031-uarray.rst b/doc/neps/nep-0031-uarray.rst index b52cb09de..3519b6bc0 100644 --- a/doc/neps/nep-0031-uarray.rst +++ b/doc/neps/nep-0031-uarray.rst @@ -42,7 +42,36 @@ discussed in the references, and this NEP will not attempt to go into the details of why these are needed; but in short: It is necessary for library authors to be able to coerce arbitrary objects into arrays of their own types, such as CuPy needing to coerce to a CuPy array, for example, instead of -a NumPy array. +a NumPy array. In simpler words, one needs things like ``np.asarray(...)`` or +an alternative to "just work" and return duck-arrays. + +The primary end-goal of this NEP is to make the following possible: + +.. code:: python + + # On the library side + import numpy.overridable as unp + + def library_function(array): + array = unp.asarray(array) + # Code using unumpy as usual + return array + + # On the user side: + import numpy.overridable as unp + import uarray as ua + import dask.array as da + + ua.register_backend(da) # Can be done within Dask itself + + library_function(dask_array) # works and returns dask_array + + with unp.set_backend(da): + library_function([1, 2, 3, 4]) # actually returns a Dask array. + +Here, ``backend`` can be any compatible object defined either by NumPy or an +external library, such as Dask or CuPy. Ideally, it should be the module +``dask.array`` or ``cupy`` itself. These kinds of overrides are useful for both the end-user as well as library authors. End-users may have written or wish to write code that they then later @@ -72,32 +101,40 @@ Detailed description Using overrides ~~~~~~~~~~~~~~~ -The way we propose the overrides will be used by end users is:: +Here are a few examples of how an end-user would use overrides. - # On the library side - import numpy.overridable as unp +.. code:: python - def library_function(array): - array = unp.asarray(array) - # Code using unumpy as usual - return array + data = da.from_zarr('myfile.zarr') + # result should still be dask, all things being equal + result = library_function(data) + result.to_zarr('output.zarr') - # On the user side: - import numpy.overridable as unp - import uarray as ua - import dask.array as da +This would keep on working, assuming the Dask backend was either set or +registered. Registration can also be done at import-time. - ua.register_backend(da) +Now consider another function, and what would need to happen in order to +make this work: - library_function(dask_array) # works and returns dask_array +.. code:: python - with unp.set_backend(da): - library_function([1, 2, 3, 4]) # actually returns a Dask array. + from dask import array as da + from magic_library import pytorch_predict + data = da.from_zarr('myfile.zarr') + # normally here one would use e.g. data.map_overlap + result = pytorch_predict(data) + result.to_zarr('output.zarr') -Here, ``backend`` can be any compatible object defined either by NumPy or an -external library, such as Dask or CuPy. Ideally, it should be the module -``dask.array`` or ``cupy`` itself. +This would work in two scenarios: The first is that ``pytorch_predict`` was a +multimethod, and implemented by the Dask backend. Dask could provide utility +functions to allow external libraries to register implementations. + +The second, and perhaps more useful way, is that ``pytorch_predict`` was defined +in an idiomatic style true to NumPy in terms of other multimethods, and that Dask +implemented the required multimethods itself, e.g. ``np.convolve``. If this +happened, then the above example would work without either ``magic_library`` +or Dask having to do anything specific to the other. Composing backends ~~~~~~~~~~~~~~~~~~ |