Skip to content

PEP 479: Change StopIteration handling inside generators #67095

Closed

Description

@Rosuav
BPO 22906
Nosy @gvanrossum, @rhettinger, @ncoghlan, @abalkin, @scoder, @vstinner, @bitdancer, @ethanfurman, @schlamar, @Rosuav, @berkerpeksag, @serhiy-storchaka, @1st1, @NeilGirdhar
Files
  • pep0479.patch: POC patch for PEP 479
  • itertoolsdoc.diff
  • pep0479.patch: Replacement POC patch
  • stopiter.py: Simple demo of new behaviour
  • pep0479.patch: Real patch, creates future directive and all
  • test_pep479.py
  • pep0479.patch
  • pep0479.patch
  • pep0479.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/1st1'
    closed_at = <Date 2015-05-09.18:45:52.986>
    created_at = <Date 2014-11-20.10:43:22.982>
    labels = ['interpreter-core', 'type-feature', 'library', 'release-blocker']
    title = 'PEP 479: Change StopIteration handling inside generators'
    updated_at = <Date 2015-10-07.01:43:49.577>
    user = 'https://github.com/Rosuav'

    bugs.python.org fields:

    activity = <Date 2015-10-07.01:43:49.577>
    actor = 'john.black.3k'
    assignee = 'yselivanov'
    closed = True
    closed_date = <Date 2015-05-09.18:45:52.986>
    closer = 'berker.peksag'
    components = ['Interpreter Core', 'Library (Lib)']
    creation = <Date 2014-11-20.10:43:22.982>
    creator = 'Rosuav'
    dependencies = []
    files = ['37233', '37297', '37640', '37641', '37646', '39079', '39316', '39317', '39323']
    hgrepos = []
    issue_num = 22906
    keywords = ['patch']
    message_count = 70.0
    messages = ['231422', '231447', '231478', '231481', '231482', '231483', '231485', '231486', '231487', '231675', '231779', '231826', '231828', '231852', '231853', '231855', '231856', '231857', '231858', '233620', '233621', '233622', '233629', '233640', '233645', '233665', '237223', '237247', '237277', '237278', '238449', '241255', '241290', '241999', '242115', '242630', '242632', '242633', '242698', '242709', '242732', '242733', '242734', '242736', '242740', '242741', '242743', '242746', '242752', '242773', '242785', '242795', '242814', '242815', '242817', '242819', '242820', '242821', '242822', '242823', '242824', '242825', '242828', '242849', '242863', '243578', '243830', '243832', '252429', '252443']
    nosy_count = 17.0
    nosy_names = ['gvanrossum', 'rhettinger', 'ncoghlan', 'belopolsky', 'scoder', 'vstinner', 'r.david.murray', 'ethan.furman', 'Yury.Selivanov', 'python-dev', 'schlamar', 'Rosuav', 'berker.peksag', 'serhiy.storchaka', 'yselivanov', 'NeilGirdhar', 'john.black.3k']
    pr_nums = []
    priority = 'release blocker'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue22906'
    versions = ['Python 3.5']

    Activity

    1. Rosuav commented on Nov 20, 2014

      @Rosuav
      ContributorAuthor

      See PEP for full details. Attached is POC patch: behaviour is altered globally (rather than respecting a __future__ directive), and minimal changes are made elsewhere to make the test suite mostly pass (test_generators does not - it'll need more comprehensive edits). Note that PEP-8 is deliberately not followed here; some of the edits ought to involve indenting slabs of code, but instead, "half-level" indentation is used, to keep the patch visibly minimal.

    2. added
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      stdlibStandard Library Python modules in the Lib/ directory
      on Nov 20, 2014
    3. python-dev commented on Nov 20, 2014

      python-devmannequin
      Mannequin

      New changeset dd19add74b21 by Guido van Rossum in branch 'default':
      PEP-479: Add link to bpo-22906.
      https://hg.python.org/peps/rev/dd19add74b21

    4. schlamar commented on Nov 21, 2014

      schlamarmannequin
      Mannequin

      AFAIS this would break all existing code for yield-based coroutine schedulers (Tornado, Twisted, Trollius, monocle, ...) when a coroutine is exited with raise StopIteration in client code.

      And this seems like a lot, a quick GitHub code search gives various examples, e.g.

    5. Rosuav commented on Nov 21, 2014

      @Rosuav
      ContributorAuthor

      Marc, are those all cases where the "raise StopIteration" is actually inside a generator? If so, it can be trivially replaced with "return". Yes, it'll break that way of spelling it, but it's a completely mechanical transformation, and making the change won't break your code for previous versions of Python. Personally, I would recommend using "return" there anyway, regardless of this proposal.

    6. schlamar commented on Nov 21, 2014

      schlamarmannequin
      Mannequin

      Yes, all yield-based coroutines are generators. I know that there is a backward compatible upgrade path, but this might have a huge impact on existing code.

      Interestingly, I didn't know before researching this PEP that you can actually use return without arguments in generators before Python 3.3 (even in 2.3) and I have worked a lot with coroutines/generators.

      So I'm not even against this proposal and using return instead of raise StopIteration seems the right way to exit a generator/coroutine, but there could be lots of affected users...

    7. Rosuav commented on Nov 21, 2014

      @Rosuav
      ContributorAuthor

      Yep, the question is whether any of the "raise StopIteration" lines are actually non-local flow control. If they're local, then it's easy: mechanical replacement with "return" and it becomes compatible with all versions (unless it has a value attached to it, as "return x" doesn't work in Py2). But if they're non-local, some refactoring will need to be done.

      In any case, my line of argument is: A generator function is not an iterator's __next__ method, ergo iterator protocol does not apply. Use of StopIteration is a hack that happens to work because of how generator functions are implemented (a thin wrapper around an iterator), but it's not part of the *concept* of a generator function.

    8. gvanrossum commented on Nov 21, 2014

      @gvanrossum
      Member

      If all examples were just using "raise StopIteration" instead of "return"
      in a generator I would be very happy. Such code can be trivially fixed by
      using "return" and the __future__ import will help the eventual transition.

      It's sad that apparently this use of return hasn't been better advertised
      -- it has existed since generators were first introduced.

      On Fri, Nov 21, 2014 at 9:44 AM, Chris Angelico <report@bugs.python.org>
      wrote:

      Chris Angelico added the comment:

      Yep, the question is whether any of the "raise StopIteration" lines are
      actually non-local flow control. If they're local, then it's easy:
      mechanical replacement with "return" and it becomes compatible with all
      versions (unless it has a value attached to it, as "return x" doesn't work
      in Py2). But if they're non-local, some refactoring will need to be done.

      In any case, my line of argument is: A generator function is not an
      iterator's __next__ method, ergo iterator protocol does not apply. Use of
      StopIteration is a hack that happens to work because of how generator
      functions are implemented (a thin wrapper around an iterator), but it's not
      part of the *concept* of a generator function.

      ----------


      Python tracker <report@bugs.python.org>
      <http://bugs.python.org/issue22906\>


    9. Rosuav commented on Nov 21, 2014

      @Rosuav
      ContributorAuthor

      Sadly, I don't know of a way to check if that's the case, other than by manually going through and eyeballing the code - if there's "raise StopIteration", see if there's also "yield" in the same function. The three cited examples are (I checked those straight away), but frankly, I am not volunteering to go manually through all of github in search of examples :|

    10. serhiy-storchaka commented on Nov 21, 2014

      @serhiy-storchaka
      Member

      I suggest to apply right now those changes which are compatible with current behavior and don't make code more cumbersome. E.g.

      •    while True:
        
      •        yield next(line_pair_iterator)
        

      + yield from line_pair_iterator

      and

      •    raise StopIteration
        

      + return

      To me these changes make code even better and are worth to be applied even if PEP-479 will be rejected.

    11. Rosuav commented on Nov 25, 2014

      @Rosuav
      ContributorAuthor

      Known issues with the current patch, if anyone feels like playing with this who better knows the code:

      1. Needs a __future__ directive to control behaviour
      2. test_generators needs to be heavily reworked
      3. The test of what exception was thrown needs to also handle StopIteration subclasses
      4. Check for refleaks and/or over-freeing
      5. Properly provide a traceback for the original StopIteration (not always happening)

      Any others?

    12. gvanrossum commented on Nov 27, 2014

      @gvanrossum
      Member

      Here's a doc patch for itertools.

    13. rhettinger commented on Nov 28, 2014

      @rhettinger
      Contributor

      FYI: I applied these two changes right after Guido pronounced on PEP-479:

      https://mail.python.org/pipermail/python-checkins/2014-November/133252.html

      https://mail.python.org/pipermail/python-checkins/2014-November/133253.html

      Also, I'm submitting a patch to fix the code in Django that will be broken by PEP-479.

    14. vstinner commented on Nov 28, 2014

      @vstinner
      Member

      FYI: I applied these two changes right after Guido pronounced on PEP-479:

      Extract of emails:

      changeset: 93542:9eb0d0eb0992
      parent: 93540:23f8a511050a
      user: Raymond Hettinger <python at rcn.com>
      date: Sat Nov 22 21:56:23 2014 -0800

      PEP-479: Don't let StopIteration bubble out of calls to next() inside a generator.

      changeset: 93543:e8b3083bb148
      user: Raymond Hettinger <python at rcn.com>
      date: Sat Nov 22 22:14:41 2014 -0800

      PEP-479: Use the return-keyword instead of raising StopIteration inside a generators.

    15. 49 remaining items

    16. Rosuav commented on May 9, 2015

      @Rosuav
      ContributorAuthor

      Weird. Tests ran fine on my machine too. Interestingly, that number is 0xdbdbdbdbdbdbdbda - does that mean anything? (It's negative 0x2424242424242426, for what that's worth.)

    17. YurySelivanov commented on May 9, 2015

      YurySelivanovmannequin
      Mannequin

      I think it crashes in debug mode or something. Somewhere we did too many decrefs.

    18. python-dev commented on May 9, 2015

      python-devmannequin
      Mannequin

      New changeset d15c26085591 by Yury Selivanov in branch 'default':
      bpo-22906: Add test file.
      https://hg.python.org/cpython/rev/d15c26085591

    19. python-dev commented on May 9, 2015

      python-devmannequin
      Mannequin

      New changeset 5d8bc813d270 by Yury Selivanov in branch 'default':
      bpo-22906: Increment refcount after PyException_SetContext
      https://hg.python.org/cpython/rev/5d8bc813d270

    20. 1st1 commented on May 9, 2015

      @1st1
      Member

      Berker, buildbots should be happy now.

    21. berkerpeksag commented on May 9, 2015

      @berkerpeksag
      Member

      Thanks!

    22. ncoghlan commented on May 10, 2015

      @ncoghlan
      Contributor

      Since it took me a moment to figure out why the extra incref was needed:

      • both PyException_SetCause and PyException_SetContext steal a reference to their second argument
      • hence we need the second incref, rather than relying solely on the reference received from PyErr_NormalizeException

      This does suggest the new incref is conceptually in the wrong spot - it should be before the call to PyException_SetCause, such that this block of code *always* possesses a valid reference while accessing "val". At the moment, we technically still don't have an active reference when passing "val" to PyException_SetContext.

    23. python-dev commented on May 10, 2015

      python-devmannequin
      Mannequin

      New changeset 787cc3d1d3af by Yury Selivanov in branch 'default':
      Issue bpo-22906: Do incref before SetCause/SetContext
      https://hg.python.org/cpython/rev/787cc3d1d3af

    24. ncoghlan commented on May 19, 2015

      @ncoghlan
      Contributor

      We missed the deprecation warning part of the PEP (for when the future import is *not* in effect, but the default behaviour will change in 3.7), but rather than reopening this one, I filed a new issue: bpo-24237

    25. python-dev commented on May 22, 2015

      python-devmannequin
      Mannequin

      New changeset 2771a0ac806b by Yury Selivanov in branch 'default':
      bpo-24237: Raise PendingDeprecationWarning per PEP-479
      https://hg.python.org/cpython/rev/2771a0ac806b

    26. python-dev commented on May 22, 2015

      python-devmannequin
      Mannequin

      New changeset c8a3e49f35e7 by Yury Selivanov in branch 'default':
      docs: Mention PEP-479 in whatsnew.
      https://hg.python.org/cpython/rev/c8a3e49f35e7

    27. johnblack3k commented on Oct 6, 2015

      johnblack3kmannequin
      Mannequin

      Consider the following generator function similar to the one that started this issue:

          from __future__ import generator_stop
      
          def myzip(*seqs):
              its = (iter(seq) for seq in seqs)
              while True:
                  try:        
                      yield tuple(next(it) for it in its)
                  except RuntimeError:
                      return
      
          g = myzip('abc', 'xyz')
          print([next(g) for i in range(5)]) # prints: [('a', 'x'), (), (), (), ()]

      A print(list(g)) would cause a hang.

    28. johnblack3k commented on Oct 7, 2015

      johnblack3kmannequin
      Mannequin

      Please ignore my previous comment - now I understood what is going on. The its generator is a one-shot iterator so is exhausted the first time the while loop is run. Therefore, on subsequent loops, only empty tuples are yielded.

      Still learning about generators... :)

    29. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    interpreter-core(Objects, Python, Grammar, and Parser dirs)release-blockerstdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions