MAINT: Implement API changes for randomgen-derived code

remove numpy.random.gen, BRNG.generator, pcg*, rand, randn
remove use_mask and Lemire's method, fix benchmarks for PCG removal
convert brng to bitgen (in C) and bit_generator (in python)
convert base R{NG,andom.*} to BitGenerator, fix last commit
randint -> integers, remove rand, randn, random_integers
RandomGenerator -> Generator, more "basic RNG" -> BitGenerator
random_sample -> random, jump -> jumped, resync with randomgen
Remove derived code from entropy
Port over changes accepted in upstream to protect log(0.0) where relevant
fix doctests for jumped, better document choice
Remove Python 2.7 shims
Use NPY_INLINE to simplify
Fix performance.py to work
Renam directory brng to bit_generators
Fix examples wiht new directory structure
Clarify relationship to historical RandomState
Remove references to .generator
Rename xoshiro256/512starstar

1 files changed, 101 insertions, 122 deletions

diff --git a/numpy/random/threefry.pyx b/numpy/random/threefry.pyx
index a8878c6bc..d008cd828 100644
--- a/numpy/random/threefry.pyx
+++ b/numpy/random/threefry.pyx
@@ -1,4 +1,3 @@
-from libc.stdlib cimport malloc, free
 from cpython.pycapsule cimport PyCapsule_New
 
 try:
@@ -9,7 +8,7 @@ except ImportError:
 import numpy as np
 
 from .common cimport *
-from .distributions cimport brng_t
+from .distributions cimport bitgen_t
 from .entropy import random_entropy, seed_by_array
 
 np.import_array()
@@ -67,56 +66,29 @@ cdef class ThreeFry:
         used.
     counter : {None, int, array_like}, optional
         Counter to use in the ThreeFry state. Can be either
-        a Python int (long in 2.x) in [0, 2**256) or a 4-element uint64 array.
+        a Python int in [0, 2**256) or a 4-element uint64 array.
         If not provided, the RNG is initialized at 0.
     key : {None, int, array_like}, optional
         Key to use in the ThreeFry state.  Unlike seed, which is run through
         another RNG before use, the value in key is directly set. Can be either
-        a Python int (long in 2.x) in [0, 2**256) or a 4-element uint64 array.
+        a Python int in [0, 2**256) or a 4-element uint64 array.
         key and seed cannot both be used.
 
     Notes
     -----
-    ThreeFry is a 64-bit PRNG that uses a counter-based design based on weaker
-    (and faster) versions of cryptographic functions [1]_. Instances using
-    different values of the key produce independent sequences.  ThreeFry has a
-    period of :math:`2^{256} - 1` and supports arbitrary advancing and jumping
-    the sequence in increments of :math:`2^{128}`. These features allow
+    ThreeFry is a 64-bit PRNG that uses a counter-based design based on
+    weaker (and faster) versions of cryptographic functions [1]_. Instances
+    using different values of the key produce independent sequences.  ``ThreeFry``
+    has a period of :math:`2^{256} - 1` and supports arbitrary advancing and
+    jumping the sequence in increments of :math:`2^{128}`. These features allow
     multiple non-overlapping sequences to be generated.
 
-    ``ThreeFry`` exposes no user-facing API except ``generator``,
-    ``state``, ``cffi`` and ``ctypes``. Designed for use in a
-    ``RandomGenerator`` object.
+    ``ThreeFry`` provides a capsule containing function pointers that produce
+    doubles, and unsigned 32 and 64- bit integers. These are not
+    directly consumable in Python and must be consumed by a ``Generator``
+    or similar object that supports low-level access.
 
-    **Compatibility Guarantee**
-
-    ``ThreeFry`` guarantees that a fixed seed will always produce the
-    same results.
-
-    See ``Philox`` for a closely related PRNG implementation.
-
-    **Parallel Features**
-
-    ``ThreeFry`` can be used in parallel applications by
-    calling the method ``jump`` which advances the state as-if
-    :math:`2^{128}` random numbers have been generated. Alternatively,
-    ``advance`` can be used to advance the counter for an any
-    positive step in [0, 2**256). When using ``jump``, all generators should
-    be initialized with the same seed to ensure that the segments come from
-    the same sequence. Alternatively, ``ThreeFry`` can be used
-    in parallel applications by using a sequence of distinct keys where each
-    instance uses different key.
-
-    >>> from numpy.random import RandomGenerator, ThreeFry
-    >>> rg = [RandomGenerator(ThreeFry(1234)) for _ in range(10)]
-    # Advance each ThreeFry instance by i jumps
-    >>> for i in range(10):
-    ...     rg[i].brng.jump(i)
-
-    Using distinct keys produces independent streams
-
-    >>> key = 2**196 + 2**132 + 2**65 + 2**33 + 2**17 + 2**9
-    >>> rg = [RandomGenerator(ThreeFry(key=key+i)) for i in range(10)]
+    See ``Philox`` for a closely related PRNG.
 
     **State and Seeding**
 
@@ -127,26 +99,44 @@ cdef class ThreeFry:
     sequences.
 
     ``ThreeFry`` is seeded using either a single 64-bit unsigned integer
-    or a vector of 64-bit unsigned integers.  In either case, the input seed is
-    used as an input (or inputs) for another simple random number generator,
-    Splitmix64, and the output of this PRNG function is used as the initial state.
+    or a vector of 64-bit unsigned integers.  In either case, the seed is
+    used as an input for a second random number generator,
+    SplitMix64, and the output of this PRNG function is used as the initial state.
     Using a single 64-bit value for the seed can only initialize a small range of
-    the possible initial state values.  When using an array, the SplitMix64 state
-    for producing the ith component of the initial state is XORd with the ith
-    value of the seed array until the seed array is exhausted. When using an array
-    the initial state for the SplitMix64 state is 0 so that using a single element
-    array and using the same value as a scalar will produce the same initial state.
+    the possible initial state values.
 
-    Examples
-    --------
-    >>> from numpy.random import RandomGenerator, ThreeFry
-    >>> rg = RandomGenerator(ThreeFry(1234))
-    >>> rg.standard_normal()
-    0.123  # random
+    **Parallel Features**
 
-    Identical method using only ThreeFry
+    ``ThreeFry`` can be used in parallel applications by calling the ``jumped``
+    method  to advances the state as-if :math:`2^{128}` random numbers have
+    been generated. Alternatively, ``advance`` can be used to advance the
+    counter for any positive step in [0, 2**256). When using ``jumped``, all
+    generators should be chained to ensure that the segments come from the same
+    sequence.
 
-    >>> rg = ThreeFry(1234).generator
+    >>> from numpy.random import Generator, ThreeFry
+    >>> bit_generator = ThreeFry(1234)
+    >>> rg = []
+    >>> for _ in range(10):
+    ...    rg.append(Generator(bit_generator))
+    ...    # Chain the BitGenerators
+    ...    bit_generator = bit_generator.jumped()
+
+    Alternatively, ``ThreeFry`` can be used in parallel applications by using
+    a sequence of distinct keys where each instance uses different key.
+
+    >>> key = 2**196 + 2**132 + 2**65 + 2**33 + 2**17 + 2**9
+    >>> rg = [Generator(ThreeFry(key=key+i)) for i in range(10)]
+
+    **Compatibility Guarantee**
+
+    ``ThreeFry`` makes a guarantee that a fixed seed and will always produce
+    the same random integer stream.
+
+    Examples
+    --------
+    >>> from numpy.random import Generator, ThreeFry
+    >>> rg = Generator(ThreeFry(1234))
     >>> rg.standard_normal()
     0.123  # random
 
@@ -157,34 +147,32 @@ cdef class ThreeFry:
            the International Conference for High Performance Computing,
            Networking, Storage and Analysis (SC11), New York, NY: ACM, 2011.
     """
-    cdef threefry_state *rng_state
-    cdef brng_t *_brng
+    cdef threefry_state rng_state
+    cdef threefry4x64_ctr_t threefry_ctr
+    cdef threefry4x64_key_t threefry_key
+    cdef bitgen_t _bitgen
     cdef public object capsule
     cdef object _ctypes
     cdef object _cffi
-    cdef object _generator
     cdef public object lock
 
     def __init__(self, seed=None, counter=None, key=None):
-        self.rng_state = <threefry_state *>malloc(sizeof(threefry_state))
-        self.rng_state.ctr = <threefry4x64_ctr_t *>malloc(sizeof(threefry4x64_ctr_t))
-        self.rng_state.key = <threefry4x64_key_t *>malloc(sizeof(threefry4x64_key_t))
-        self._brng = <brng_t *>malloc(sizeof(brng_t))
+        self.rng_state.ctr = &self.threefry_ctr
+        self.rng_state.key = &self.threefry_key
         self.seed(seed, counter, key)
         self.lock = Lock()
 
-        self._brng.state = <void *>self.rng_state
-        self._brng.next_uint64 = &threefry_uint64
-        self._brng.next_uint32 = &threefry_uint32
-        self._brng.next_double = &threefry_double
-        self._brng.next_raw = &threefry_uint64
+        self._bitgen.state = <void *>&self.rng_state
+        self._bitgen.next_uint64 = &threefry_uint64
+        self._bitgen.next_uint32 = &threefry_uint32
+        self._bitgen.next_double = &threefry_double
+        self._bitgen.next_raw = &threefry_uint64
 
         self._ctypes = None
         self._cffi = None
-        self._generator = None
 
-        cdef const char *name = 'BasicRNG'
-        self.capsule = PyCapsule_New(<void *>self._brng, name, NULL)
+        cdef const char *name = 'BitGenerator'
+        self.capsule = PyCapsule_New(<void *>&self._bitgen, name, NULL)
 
     # Pickling support:
     def __getstate__(self):
@@ -194,18 +182,8 @@ cdef class ThreeFry:
         self.state = state
 
     def __reduce__(self):
-        from ._pickle import __brng_ctor
-        return (__brng_ctor,
-                (self.state['brng'],),
-                self.state)
-
-    def __dealloc__(self):
-        if self.rng_state:
-            free(self.rng_state.ctr)
-            free(self.rng_state.key)
-            free(self.rng_state)
-        if self._brng:
-            free(self._brng)
+        from ._pickle import __bit_generator_ctor
+        return __bit_generator_ctor, (self.state['bit_generator'],), self.state
 
     cdef _reset_state_variables(self):
         self.rng_state.has_uint32 = 0
@@ -218,7 +196,7 @@ cdef class ThreeFry:
         """
         random_raw(self, size=None)
 
-        Return randoms as generated by the underlying BasicRNG
+        Return randoms as generated by the underlying BitGenerator
 
         Parameters
         ----------
@@ -243,10 +221,10 @@ cdef class ThreeFry:
 
         See the class docstring for the number of bits returned.
         """
-        return random_raw(self._brng, self.lock, size, output)
+        return random_raw(&self._bitgen, self.lock, size, output)
 
     def _benchmark(self, Py_ssize_t cnt, method=u'uint64'):
-        return benchmark(self._brng, self.lock, cnt, method)
+        return benchmark(&self._bitgen, self.lock, cnt, method)
 
     def seed(self, seed=None, counter=None, key=None):
         """
@@ -325,7 +303,7 @@ cdef class ThreeFry:
         for i in range(THREEFRY_BUFFER_SIZE):
             buffer[i] = self.rng_state.buffer[i]
         state = {'counter': ctr, 'key': key}
-        return {'brng': self.__class__.__name__,
+        return {'bit_generator': self.__class__.__name__,
                 'state': state,
                 'buffer': buffer,
                 'buffer_pos': self.rng_state.buffer_pos,
@@ -336,8 +314,8 @@ cdef class ThreeFry:
     def state(self, value):
         if not isinstance(value, dict):
             raise TypeError('state must be a dict')
-        brng = value.get('brng', '')
-        if brng != self.__class__.__name__:
+        bitgen = value.get('bit_generator', '')
+        if bitgen != self.__class__.__name__:
             raise ValueError('state must be for a {0} '
                              'PRNG'.format(self.__class__.__name__))
         for i in range(4):
@@ -349,28 +327,45 @@ cdef class ThreeFry:
         self.rng_state.uinteger = value['uinteger']
         self.rng_state.buffer_pos = value['buffer_pos']
 
-    def jump(self, np.npy_intp iter=1):
+    cdef jump_inplace(self, np.npy_intp iter):
         """
-        jump(iter=1)
+        Jump state in-place
 
-        Jumps the state as-if 2**128 random numbers have been generated.
+        Not part of public API
 
         Parameters
         ----------
         iter : integer, positive
             Number of times to jump the state of the rng.
+        """
+        self.advance(iter * 2 ** 128)
+
+    def jumped(self, np.npy_intp iter=1):
+        """
+        jumped(iter=1)
+
+        Returns a new bit generator with the state jumped
+
+        The state of the returned big generator is jumped as-if
+        2**(128 * iter) random numbers have been generated.
+
+        Parameters
+        ----------
+        iter : integer, positive
+            Number of times to jump the state of the bit generator returned
 
         Returns
         -------
-        self : ThreeFry
-            PRNG jumped iter times
-
-        Notes
-        -----
-        Jumping the rng state resets any pre-computed random numbers. This is
-        required to ensure exact reproducibility.
+        bit_generator : Xoroshiro128
+            New instance of generator jumped iter times
         """
-        return self.advance(iter * 2**128)
+        cdef ThreeFry bit_generator
+
+        bit_generator = self.__class__()
+        bit_generator.state = self.state
+        bit_generator.jump_inplace(iter)
+
+        return bit_generator
 
     def advance(self, delta):
         """
@@ -410,8 +405,7 @@ cdef class ThreeFry:
         """
         cdef np.ndarray delta_a
         delta_a = int_to_array(delta, 'step', 256, 64)
-        loc = 0
-        threefry_advance(<uint64_t *>delta_a.data, self.rng_state)
+        threefry_advance(<uint64_t *>delta_a.data, &self.rng_state)
         self._reset_state_variables()
         return self
 
@@ -430,10 +424,10 @@ cdef class ThreeFry:
             * next_uint64 - function pointer to produce 64 bit integers
             * next_uint32 - function pointer to produce 32 bit integers
             * next_double - function pointer to produce doubles
-            * brng - pointer to the Basic RNG struct
+            * bitgen - pointer to the BitGenerator struct
         """
         if self._ctypes is None:
-            self._ctypes = prepare_ctypes(self._brng)
+            self._ctypes = prepare_ctypes(&self._bitgen)
 
         return self._ctypes
 
@@ -452,24 +446,9 @@ cdef class ThreeFry:
             * next_uint64 - function pointer to produce 64 bit integers
             * next_uint32 - function pointer to produce 32 bit integers
             * next_double - function pointer to produce doubles
-            * brng - pointer to the Basic RNG struct
+            * bitgen - pointer to the BitGenerator struct
         """
         if self._cffi is not None:
             return self._cffi
-        self._cffi = prepare_cffi(self._brng)
+        self._cffi = prepare_cffi(&self._bitgen)
         return self._cffi
-
-    @property
-    def generator(self):
-        """
-        Return a RandomGenerator object
-
-        Returns
-        -------
-        gen : numpy.random.RandomGenerator
-            Random generator used this instance as the core PRNG
-        """
-        if self._generator is None:
-            from .generator import RandomGenerator
-            self._generator = RandomGenerator(self)
-        return self._generator