You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Python already has three functions to generate random bytes:
os.getrandom(): specific to Linux, not portable
os.urandom()
secrets.token_bytes()
These 3 functions are based on system entropy and they block on Linux until the kernel collected enough entropy: PEP-524.
While many users are fine with these functions, there are also use cases for simulation where the security doesn't matter, and it's more about being able to get reproducible experience from a seed. That's what random.Random is about.
If we have to have this, the method name should be differentiated from getrandbits() because the latter returns an integer. I suggest just random.bytes(n), the same as numpy.
Python already has three functions to generate random bytes:
Now, there will be four ;-)
vedgar commented on Apr 15, 2020
vedgarmannequin
Mannequin
More actions
I suggest just random.bytes(n), the same as numpy.
The problem with this is that people who from random import * (some schools insist on this, probably because most functions they need already start with rand) will shadow builtin bytes. Not that those schools do anything with bytes, but still, it might be inconvenient.
(The metaproblem is of course that some functions already do the "poor man's namespacing" in C-style by starting with rand, and some don't. I'm always for user control of namespacing, but I'm just saying that it doesn't correspond to how many beginners use random module.)
Do you have another name suggestion that doesn't have a parallelism problem with the existing name? The names getrandbytes() and getrandbits() suggest a parallelism that is incorrect.
vedgar commented on Apr 15, 2020
vedgarmannequin
Mannequin
More actions
I think that the "module owner";-P must decide whether the random module should follow the C-namespacing or not. Of course, I'm in the "not" camp, so I believe those two "rand..." functions (randrange is completely redundant with random.choice(range)) should be supplemented with random.int and random.float. And then random.bytes will be completely natural. And people might be gently nudged into the right direction when using Python module namespaces.
The randbytes() method needs to depend on genrandbits(). It is documented that custom generators can supply there own random() and genrandbits() methods and expect that the other downstream generators all follow. See the attached example which demonstrates that randbytes() bypasses this framework pattern.
Also, I don't want randbytes() in the C extension. We're tried to keep as much of the code as possible in pure Python and only have the MersenneTwister specific code in the C module. The improves maintainability and makes the code more accessible to a broader audience.
Also, please don't change the name of the genrand_int32() function. It was a goal to change as little as possible from the official, standard version of the C code at http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html . For the most part, we just want to wrap that code for Python bindings, but not modify it.
$ ./python -m timeit -s 'import random''random.randbytes(10**6)'
200 loops, best of 5: 1.36 msec per loop
$ ./python -m timeit -s 'import random''random.getrandbits(10**6*8).to_bytes(10**6, "little")'
50 loops, best of 5: 6.31 msec per loop
The Python implementation is only 5 times slower than the C implementation. I am fine with implementing randbytes() in Python. This would automatically make it depending on the getrandbits() implementation.
Also, I don't want randbytes() in the C extension. We're tried to keep as much of the code as possible in pure Python and only have the MersenneTwister specific code in the C module. The improves maintainability and makes the code more accessible to a broader audience.
I don't see how 30 lines makes Python so harder to maintain. These lines make the function 4x to 5x faster. We are not talking about 5% or 10% faster. I think that such optimization is worth it. When did we decide to stop optimizing Python?
Raymond:
The randbytes() method needs to depend on genrandbits().
I created bpo-40346: "Redesign random.Random class inheritance" for a more generic fix, not just randbytes().
This code was already modified to replace "unsigned long" with "uint32_t" for example. I don't think that renaming genrand_int32() to genrand_uint32() makes the code impossible to maintain. Moreover, it seems like http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html was not updated for 13 years.
The randbytes() method needs to depend on genrandbits().
I created PR 19700 which allows to keep the optimization (C implementation in _randommodule.c) and Random subclasses implement randbytes() with getrandbits().
It removed the C implementation of randbytes(): it was the root issue which started discussions here and in bpo-40346. I rejected bpo-40346 (BaseRandom) and related PRs.
New changeset f01d1be by Raymond Hettinger in branch 'master': bpo-40286: Put methods in correct sections. Add security notice to use secrets for session tokens. (GH-19870) f01d1be
Activity
vstinner commentedon Apr 14, 2020
The random module lacks a getrandbytes() method which leads developers to be creative how to generate bytes:
https://stackoverflow.com/questions/5495492/random-byte-string-in-python
It's a common use request:
Python already has three functions to generate random bytes:
These 3 functions are based on system entropy and they block on Linux until the kernel collected enough entropy: PEP-524.
While many users are fine with these functions, there are also use cases for simulation where the security doesn't matter, and it's more about being able to get reproducible experience from a seed. That's what random.Random is about.
The numpy module provides numpy.random.bytes(length) function for such use case:
https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.random.bytes.html
One example can be to generate UUID4 with the ability to reproduce the random UUID from a seed for testing purpose, or to get reproducible behavior.
Attached PR implements the getrandbytes() method.
rhettinger commentedon Apr 14, 2020
If we have to have this, the method name should be differentiated from getrandbits() because the latter returns an integer. I suggest just random.bytes(n), the same as numpy.
Now, there will be four ;-)
vedgar commentedon Apr 15, 2020
The problem with this is that people who
from random import *(some schools insist on this, probably because most functions they need already start withrand) will shadow builtinbytes. Not that those schools do anything withbytes, but still, it might be inconvenient.(The metaproblem is of course that some functions already do the "poor man's namespacing" in C-style by starting with
rand, and some don't. I'm always for user control of namespacing, but I'm just saying that it doesn't correspond to how many beginners userandommodule.)rhettinger commentedon Apr 15, 2020
Do you have another name suggestion that doesn't have a parallelism problem with the existing name? The names getrandbytes() and getrandbits() suggest a parallelism that is incorrect.
vedgar commentedon Apr 15, 2020
I think that the "module owner";-P must decide whether the
randommodule should follow the C-namespacing or not. Of course, I'm in the "not" camp, so I believe those two "rand..." functions (randrange is completely redundant with random.choice(range)) should be supplemented with random.int and random.float. And then random.bytes will be completely natural. And people might be gently nudged into the right direction when using Python module namespaces.rhettinger commentedon Apr 15, 2020
I concur that bytes() isn't a good name, but am still concerned that the proposed name is a bad API decision.
serhiy-storchaka commentedon Apr 15, 2020
Maybe randbytes()?
vstinner commentedon Apr 15, 2020
I like "from random import randbytes" name. I concur that "from random import bytes" overrides bytes() builtin type and so can likely cause troubles.
vstinner commentedon Apr 15, 2020
I updated my PR to rename the method to randbytes().
[-]Add getrandbytes() method to random.Random[/-][+]Add randbytes() method to random.Random[/+][-]Add getrandbytes() method to random.Random[/-][+]Add randbytes() method to random.Random[/+]8 remaining items
rhettinger commentedon Apr 20, 2020
The randbytes() method needs to depend on genrandbits(). It is documented that custom generators can supply there own random() and genrandbits() methods and expect that the other downstream generators all follow. See the attached example which demonstrates that randbytes() bypasses this framework pattern.
Also, I don't want randbytes() in the C extension. We're tried to keep as much of the code as possible in pure Python and only have the MersenneTwister specific code in the C module. The improves maintainability and makes the code more accessible to a broader audience.
Also, please don't change the name of the genrand_int32() function. It was a goal to change as little as possible from the official, standard version of the C code at http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html . For the most part, we just want to wrap that code for Python bindings, but not modify it.
rhettinger commentedon Apr 20, 2020
Direct link to MT code that I would like to leave mostly unmodified: http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/CODES/mt19937ar.c
rhettinger commentedon Apr 20, 2020
When a new method gets added to a module, it should happen in a way that is in harmony with the module's design.
vstinner commentedon Apr 20, 2020
I created bpo-40346: "Redesign random.Random class inheritance".
serhiy-storchaka commentedon Apr 21, 2020
The Python implementation is only 5 times slower than the C implementation. I am fine with implementing randbytes() in Python. This would automatically make it depending on the getrandbits() implementation.
vstinner commentedon Apr 21, 2020
Raymond:
I don't see how 30 lines makes Python so harder to maintain. These lines make the function 4x to 5x faster. We are not talking about 5% or 10% faster. I think that such optimization is worth it. When did we decide to stop optimizing Python?
Raymond:
I created bpo-40346: "Redesign random.Random class inheritance" for a more generic fix, not just randbytes().
Raymond:
This code was already modified to replace "unsigned long" with "uint32_t" for example. I don't think that renaming genrand_int32() to genrand_uint32() makes the code impossible to maintain. Moreover, it seems like http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html was not updated for 13 years.
vstinner commentedon Apr 24, 2020
Raymond:
I created PR 19700 which allows to keep the optimization (C implementation in _randommodule.c) and Random subclasses implement randbytes() with getrandbits().
vstinner commentedon Apr 29, 2020
New changeset 2d87577 by Victor Stinner in branch 'master':
bpo-40286: Remove C implementation of Random.randbytes() (GH-19797)
2d87577
vstinner commentedon Apr 29, 2020
It removed the C implementation of randbytes(): it was the root issue which started discussions here and in bpo-40346. I rejected bpo-40346 (BaseRandom) and related PRs.
I close the issue.
rhettinger commentedon May 5, 2020
New changeset f01d1be by Raymond Hettinger in branch 'master':
bpo-40286: Put methods in correct sections. Add security notice to use secrets for session tokens. (GH-19870)
f01d1be