Repository navigation
Closed
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)release-blockerstdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancementA feature request or enhancement
Description
| BPO | 22906 |
|---|---|
| Nosy | @gvanrossum, @rhettinger, @ncoghlan, @abalkin, @scoder, @vstinner, @bitdancer, @ethanfurman, @schlamar, @Rosuav, @berkerpeksag, @serhiy-storchaka, @1st1, @NeilGirdhar |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/1st1'
closed_at = <Date 2015-05-09.18:45:52.986>
created_at = <Date 2014-11-20.10:43:22.982>
labels = ['interpreter-core', 'type-feature', 'library', 'release-blocker']
title = 'PEP 479: Change StopIteration handling inside generators'
updated_at = <Date 2015-10-07.01:43:49.577>
user = 'https://github.com/Rosuav'bugs.python.org fields:
activity = <Date 2015-10-07.01:43:49.577>
actor = 'john.black.3k'
assignee = 'yselivanov'
closed = True
closed_date = <Date 2015-05-09.18:45:52.986>
closer = 'berker.peksag'
components = ['Interpreter Core', 'Library (Lib)']
creation = <Date 2014-11-20.10:43:22.982>
creator = 'Rosuav'
dependencies = []
files = ['37233', '37297', '37640', '37641', '37646', '39079', '39316', '39317', '39323']
hgrepos = []
issue_num = 22906
keywords = ['patch']
message_count = 70.0
messages = ['231422', '231447', '231478', '231481', '231482', '231483', '231485', '231486', '231487', '231675', '231779', '231826', '231828', '231852', '231853', '231855', '231856', '231857', '231858', '233620', '233621', '233622', '233629', '233640', '233645', '233665', '237223', '237247', '237277', '237278', '238449', '241255', '241290', '241999', '242115', '242630', '242632', '242633', '242698', '242709', '242732', '242733', '242734', '242736', '242740', '242741', '242743', '242746', '242752', '242773', '242785', '242795', '242814', '242815', '242817', '242819', '242820', '242821', '242822', '242823', '242824', '242825', '242828', '242849', '242863', '243578', '243830', '243832', '252429', '252443']
nosy_count = 17.0
nosy_names = ['gvanrossum', 'rhettinger', 'ncoghlan', 'belopolsky', 'scoder', 'vstinner', 'r.david.murray', 'ethan.furman', 'Yury.Selivanov', 'python-dev', 'schlamar', 'Rosuav', 'berker.peksag', 'serhiy.storchaka', 'yselivanov', 'NeilGirdhar', 'john.black.3k']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue22906'
versions = ['Python 3.5']Metadata
Metadata
Assignees
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)release-blockerstdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancementA feature request or enhancement
Activity
Rosuav commentedon Nov 20, 2014
See PEP for full details. Attached is POC patch: behaviour is altered globally (rather than respecting a __future__ directive), and minimal changes are made elsewhere to make the test suite mostly pass (test_generators does not - it'll need more comprehensive edits). Note that PEP-8 is deliberately not followed here; some of the edits ought to involve indenting slabs of code, but instead, "half-level" indentation is used, to keep the patch visibly minimal.
python-dev commentedon Nov 20, 2014
New changeset dd19add74b21 by Guido van Rossum in branch 'default':
PEP-479: Add link to bpo-22906.
https://hg.python.org/peps/rev/dd19add74b21
schlamar commentedon Nov 21, 2014
AFAIS this would break all existing code for yield-based coroutine schedulers (Tornado, Twisted, Trollius, monocle, ...) when a coroutine is exited with
raise StopIterationin client code.And this seems like a lot, a quick GitHub code search gives various examples, e.g.
Rosuav commentedon Nov 21, 2014
Marc, are those all cases where the "raise StopIteration" is actually inside a generator? If so, it can be trivially replaced with "return". Yes, it'll break that way of spelling it, but it's a completely mechanical transformation, and making the change won't break your code for previous versions of Python. Personally, I would recommend using "return" there anyway, regardless of this proposal.
schlamar commentedon Nov 21, 2014
Yes, all yield-based coroutines are generators. I know that there is a backward compatible upgrade path, but this might have a huge impact on existing code.
Interestingly, I didn't know before researching this PEP that you can actually use
returnwithout arguments in generators before Python 3.3 (even in 2.3) and I have worked a lot with coroutines/generators.So I'm not even against this proposal and using
returninstead ofraise StopIterationseems the right way to exit a generator/coroutine, but there could be lots of affected users...Rosuav commentedon Nov 21, 2014
Yep, the question is whether any of the "raise StopIteration" lines are actually non-local flow control. If they're local, then it's easy: mechanical replacement with "return" and it becomes compatible with all versions (unless it has a value attached to it, as "return x" doesn't work in Py2). But if they're non-local, some refactoring will need to be done.
In any case, my line of argument is: A generator function is not an iterator's __next__ method, ergo iterator protocol does not apply. Use of StopIteration is a hack that happens to work because of how generator functions are implemented (a thin wrapper around an iterator), but it's not part of the *concept* of a generator function.
gvanrossum commentedon Nov 21, 2014
If all examples were just using "raise StopIteration" instead of "return"
in a generator I would be very happy. Such code can be trivially fixed by
using "return" and the __future__ import will help the eventual transition.
It's sad that apparently this use of return hasn't been better advertised
-- it has existed since generators were first introduced.
On Fri, Nov 21, 2014 at 9:44 AM, Chris Angelico <report@bugs.python.org>
wrote:
Rosuav commentedon Nov 21, 2014
Sadly, I don't know of a way to check if that's the case, other than by manually going through and eyeballing the code - if there's "raise StopIteration", see if there's also "yield" in the same function. The three cited examples are (I checked those straight away), but frankly, I am not volunteering to go manually through all of github in search of examples :|
serhiy-storchaka commentedon Nov 21, 2014
I suggest to apply right now those changes which are compatible with current behavior and don't make code more cumbersome. E.g.
+ yield from line_pair_iterator
and
+ return
To me these changes make code even better and are worth to be applied even if PEP-479 will be rejected.
Rosuav commentedon Nov 25, 2014
Known issues with the current patch, if anyone feels like playing with this who better knows the code:
Any others?
gvanrossum commentedon Nov 27, 2014
Here's a doc patch for itertools.
rhettinger commentedon Nov 28, 2014
FYI: I applied these two changes right after Guido pronounced on PEP-479:
https://mail.python.org/pipermail/python-checkins/2014-November/133252.html
https://mail.python.org/pipermail/python-checkins/2014-November/133253.html
Also, I'm submitting a patch to fix the code in Django that will be broken by PEP-479.
vstinner commentedon Nov 28, 2014
Extract of emails:
changeset: 93542:9eb0d0eb0992
parent: 93540:23f8a511050a
user: Raymond Hettinger <python at rcn.com>
date: Sat Nov 22 21:56:23 2014 -0800
PEP-479: Don't let StopIteration bubble out of calls to next() inside a generator.
changeset: 93543:e8b3083bb148
user: Raymond Hettinger <python at rcn.com>
date: Sat Nov 22 22:14:41 2014 -0800
PEP-479: Use the return-keyword instead of raising StopIteration inside a generators.
49 remaining items
Rosuav commentedon May 9, 2015
Weird. Tests ran fine on my machine too. Interestingly, that number is 0xdbdbdbdbdbdbdbda - does that mean anything? (It's negative 0x2424242424242426, for what that's worth.)
YurySelivanov commentedon May 9, 2015
I think it crashes in debug mode or something. Somewhere we did too many decrefs.
python-dev commentedon May 9, 2015
New changeset d15c26085591 by Yury Selivanov in branch 'default':
bpo-22906: Add test file.
https://hg.python.org/cpython/rev/d15c26085591
python-dev commentedon May 9, 2015
New changeset 5d8bc813d270 by Yury Selivanov in branch 'default':
bpo-22906: Increment refcount after PyException_SetContext
https://hg.python.org/cpython/rev/5d8bc813d270
1st1 commentedon May 9, 2015
Berker, buildbots should be happy now.
berkerpeksag commentedon May 9, 2015
Thanks!
ncoghlan commentedon May 10, 2015
Since it took me a moment to figure out why the extra incref was needed:
This does suggest the new incref is conceptually in the wrong spot - it should be before the call to PyException_SetCause, such that this block of code *always* possesses a valid reference while accessing "val". At the moment, we technically still don't have an active reference when passing "val" to PyException_SetContext.
python-dev commentedon May 10, 2015
New changeset 787cc3d1d3af by Yury Selivanov in branch 'default':
Issue bpo-22906: Do incref before SetCause/SetContext
https://hg.python.org/cpython/rev/787cc3d1d3af
ncoghlan commentedon May 19, 2015
We missed the deprecation warning part of the PEP (for when the future import is *not* in effect, but the default behaviour will change in 3.7), but rather than reopening this one, I filed a new issue: bpo-24237
python-dev commentedon May 22, 2015
New changeset 2771a0ac806b by Yury Selivanov in branch 'default':
bpo-24237: Raise PendingDeprecationWarning per PEP-479
https://hg.python.org/cpython/rev/2771a0ac806b
python-dev commentedon May 22, 2015
New changeset c8a3e49f35e7 by Yury Selivanov in branch 'default':
docs: Mention PEP-479 in whatsnew.
https://hg.python.org/cpython/rev/c8a3e49f35e7
johnblack3k commentedon Oct 6, 2015
Consider the following generator function similar to the one that started this issue:
A print(list(g)) would cause a hang.
johnblack3k commentedon Oct 7, 2015
Please ignore my previous comment - now I understood what is going on. The
itsgenerator is a one-shot iterator so is exhausted the first time the while loop is run. Therefore, on subsequent loops, only empty tuples are yielded.Still learning about generators... :)