Skip to content

Add randbytes() method to random.Random #84466

Closed

Description

@vstinner
BPO 40286
Nosy @tim-one, @rhettinger, @mdickinson, @vstinner, @serhiy-storchaka, @vedgar
PRs
  • bpo-40286: Add randbytes() method to random.Random #19527
  • bpo-40286: Makes simpler the relation between randbytes() and getrandbits() #19574
  • bpo-40286: Use random.randbytes() in tests #19575
  • bpo-40286: Remove C implementation of Random.randbytes() #19797
  • bpo-40286: Put methods in correct sections. Add security notice #19870
  • Files
  • run_random.py: Custom PRNG where randbytes() doesn't fit
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2020-04-29.16:51:01.179>
    created_at = <Date 2020-04-14.22:30:18.004>
    labels = ['library', '3.9']
    title = 'Add randbytes() method to random.Random'
    updated_at = <Date 2020-05-05.05:52:17.085>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2020-05-05.05:52:17.085>
    actor = 'rhettinger'
    assignee = 'none'
    closed = True
    closed_date = <Date 2020-04-29.16:51:01.179>
    closer = 'vstinner'
    components = ['Library (Lib)']
    creation = <Date 2020-04-14.22:30:18.004>
    creator = 'vstinner'
    dependencies = []
    files = ['49078']
    hgrepos = []
    issue_num = 40286
    keywords = ['patch']
    message_count = 25.0
    messages = ['366454', '366457', '366496', '366498', '366500', '366502', '366505', '366507', '366512', '366514', '366524', '366665', '366666', '366676', '366677', '366860', '366861', '366869', '366894', '366911', '366919', '367180', '367676', '367677', '368104']
    nosy_count = 6.0
    nosy_names = ['tim.peters', 'rhettinger', 'mark.dickinson', 'vstinner', 'serhiy.storchaka', 'veky']
    pr_nums = ['19527', '19574', '19575', '19797', '19870']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue40286'
    versions = ['Python 3.9']

    Activity

    1. vstinner commented on Apr 14, 2020

      @vstinner
      MemberAuthor

      The random module lacks a getrandbytes() method which leads developers to be creative how to generate bytes:
      https://stackoverflow.com/questions/5495492/random-byte-string-in-python

      It's a common use request:

      Python already has three functions to generate random bytes:

      • os.getrandom(): specific to Linux, not portable
      • os.urandom()
      • secrets.token_bytes()

      These 3 functions are based on system entropy and they block on Linux until the kernel collected enough entropy: PEP-524.

      While many users are fine with these functions, there are also use cases for simulation where the security doesn't matter, and it's more about being able to get reproducible experience from a seed. That's what random.Random is about.

      The numpy module provides numpy.random.bytes(length) function for such use case:
      https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.random.bytes.html

      One example can be to generate UUID4 with the ability to reproduce the random UUID from a seed for testing purpose, or to get reproducible behavior.

      Attached PR implements the getrandbytes() method.

    2. added
      stdlibStandard Library Python modules in the Lib/ directory
      on Apr 14, 2020
    3. rhettinger commented on Apr 14, 2020

      @rhettinger
      Contributor

      If we have to have this, the method name should be differentiated from getrandbits() because the latter returns an integer. I suggest just random.bytes(n), the same as numpy.

      Python already has three functions to generate random bytes:

      Now, there will be four ;-)

    4. vedgar commented on Apr 15, 2020

      vedgarmannequin
      Mannequin

      I suggest just random.bytes(n), the same as numpy.

      The problem with this is that people who from random import * (some schools insist on this, probably because most functions they need already start with rand) will shadow builtin bytes. Not that those schools do anything with bytes, but still, it might be inconvenient.

      (The metaproblem is of course that some functions already do the "poor man's namespacing" in C-style by starting with rand, and some don't. I'm always for user control of namespacing, but I'm just saying that it doesn't correspond to how many beginners use random module.)

    5. rhettinger commented on Apr 15, 2020

      @rhettinger
      Contributor

      Do you have another name suggestion that doesn't have a parallelism problem with the existing name? The names getrandbytes() and getrandbits() suggest a parallelism that is incorrect.

    6. vedgar commented on Apr 15, 2020

      vedgarmannequin
      Mannequin

      I think that the "module owner";-P must decide whether the random module should follow the C-namespacing or not. Of course, I'm in the "not" camp, so I believe those two "rand..." functions (randrange is completely redundant with random.choice(range)) should be supplemented with random.int and random.float. And then random.bytes will be completely natural. And people might be gently nudged into the right direction when using Python module namespaces.

    7. rhettinger commented on Apr 15, 2020

      @rhettinger
      Contributor

      I concur that bytes() isn't a good name, but am still concerned that the proposed name is a bad API decision.

    8. serhiy-storchaka commented on Apr 15, 2020

      @serhiy-storchaka
      Member

      Maybe randbytes()?

    9. vstinner commented on Apr 15, 2020

      @vstinner
      MemberAuthor

      I like "from random import randbytes" name. I concur that "from random import bytes" overrides bytes() builtin type and so can likely cause troubles.

    10. vstinner commented on Apr 15, 2020

      @vstinner
      MemberAuthor

      I updated my PR to rename the method to randbytes().

    11. changed the title [-]Add getrandbytes() method to random.Random[/-] [+]Add randbytes() method to random.Random[/+] on Apr 15, 2020
    12. changed the title [-]Add getrandbytes() method to random.Random[/-] [+]Add randbytes() method to random.Random[/+] on Apr 15, 2020
    13. 8 remaining items

    14. rhettinger commented on Apr 20, 2020

      @rhettinger
      Contributor

      The randbytes() method needs to depend on genrandbits(). It is documented that custom generators can supply there own random() and genrandbits() methods and expect that the other downstream generators all follow. See the attached example which demonstrates that randbytes() bypasses this framework pattern.

      Also, I don't want randbytes() in the C extension. We're tried to keep as much of the code as possible in pure Python and only have the MersenneTwister specific code in the C module. The improves maintainability and makes the code more accessible to a broader audience.

      Also, please don't change the name of the genrand_int32() function. It was a goal to change as little as possible from the official, standard version of the C code at http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html . For the most part, we just want to wrap that code for Python bindings, but not modify it.

    15. rhettinger commented on Apr 20, 2020

      @rhettinger
      Contributor

      Direct link to MT code that I would like to leave mostly unmodified: http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/CODES/mt19937ar.c

    16. rhettinger commented on Apr 20, 2020

      @rhettinger
      Contributor

      When a new method gets added to a module, it should happen in a way that is in harmony with the module's design.

    17. vstinner commented on Apr 20, 2020

      @vstinner
      MemberAuthor

      I created bpo-40346: "Redesign random.Random class inheritance".

    18. serhiy-storchaka commented on Apr 21, 2020

      @serhiy-storchaka
      Member
      $ ./python -m timeit -s 'import random' 'random.randbytes(10**6)'
      200 loops, best of 5: 1.36 msec per loop
      
      $ ./python -m timeit -s 'import random' 'random.getrandbits(10**6*8).to_bytes(10**6, "little")'
      50 loops, best of 5: 6.31 msec per loop

      The Python implementation is only 5 times slower than the C implementation. I am fine with implementing randbytes() in Python. This would automatically make it depending on the getrandbits() implementation.

    19. vstinner commented on Apr 21, 2020

      @vstinner
      MemberAuthor

      Raymond:

      Also, I don't want randbytes() in the C extension. We're tried to keep as much of the code as possible in pure Python and only have the MersenneTwister specific code in the C module. The improves maintainability and makes the code more accessible to a broader audience.

      I don't see how 30 lines makes Python so harder to maintain. These lines make the function 4x to 5x faster. We are not talking about 5% or 10% faster. I think that such optimization is worth it. When did we decide to stop optimizing Python?

      Raymond:

      The randbytes() method needs to depend on genrandbits().

      I created bpo-40346: "Redesign random.Random class inheritance" for a more generic fix, not just randbytes().

      Raymond:

      Also, please don't change the name of the genrand_int32() function. It was a goal to change as little as possible from the official, standard version of the C code at http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html .

      This code was already modified to replace "unsigned long" with "uint32_t" for example. I don't think that renaming genrand_int32() to genrand_uint32() makes the code impossible to maintain. Moreover, it seems like http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html was not updated for 13 years.

    20. vstinner commented on Apr 24, 2020

      @vstinner
      MemberAuthor

      Raymond:

      The randbytes() method needs to depend on genrandbits().

      I created PR 19700 which allows to keep the optimization (C implementation in _randommodule.c) and Random subclasses implement randbytes() with getrandbits().

    21. vstinner commented on Apr 29, 2020

      @vstinner
      MemberAuthor

      New changeset 2d87577 by Victor Stinner in branch 'master':
      bpo-40286: Remove C implementation of Random.randbytes() (GH-19797)
      2d87577

    22. vstinner commented on Apr 29, 2020

      @vstinner
      MemberAuthor

      It removed the C implementation of randbytes(): it was the root issue which started discussions here and in bpo-40346. I rejected bpo-40346 (BaseRandom) and related PRs.

      I close the issue.

    23. rhettinger commented on May 5, 2020

      @rhettinger
      Contributor

      New changeset f01d1be by Raymond Hettinger in branch 'master':
      bpo-40286: Put methods in correct sections. Add security notice to use secrets for session tokens. (GH-19870)
      f01d1be

    24. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.9 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directory

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions