You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
subprocess.Popen.__exit__ doesn't wait for process end #56253
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee='https://github.com/gpshead'closed_at=<Date2011-05-12.05:23:31.318>created_at=<Date2011-05-09.19:24:14.200>labels= ['type-bug', 'library']
title="subprocess.Popen.__exit__ doesn't wait for process end"updated_at=<Date2011-07-05.20:36:58.179>user='https://github.com/pitrou'
I find it a bit strange that Popen.__exit__ closes all standard file descriptors leading to the child process, but doesn't wait for process end afterwards.
(context management support was added in bpo-10554)
I'm thinking there's not a lot we can do here, but also not a lot that we should do here. We don't want to wait, and we don't want to close, so maybe we should just document that the usage should be limited only to expecting the open handles to be closed?
I'm thinking there's not a lot we can do here, but also not a lot that
we should do here. We don't want to wait, and we don't want to close,
so maybe we should just document that the usage should be limited only
to expecting the open handles to be closed?
There's just one thing I'm concerned with.
People using context managers tend to expect the __exit__ method to
perform cleanup actions and release corresponding resources if
necessary, for example closing the underlying file or socket.
So my guess is that most people won't explicitly wait for the process
termination. You can already find such examples inside
test_subprocess.py, Lib/ctypes/util.py (to parse ldconfig's output),
and even in subprocess' documentation:
"""
with Popen(["ifconfig"], stdout=PIPE) as proc:
log.write(proc.stdout.read())
"""
The problem is that until a child process is wait()ed on or its parent
process terminates (so that the child gets re-parented to init), the
child remains a zombie/defunct process. So if you have long-running
process spawning many subprocesses, you could end up having many
zombie processes, potentially filling up your process table at some
point. And this really sucks.
So I think it could be worth mentioning this somewhere in the
documentation (I mean, if we can find find such leaks inside CPython
code base, then it's definitely something that should be documented).
I didn't initially like the idea of __exit__ blocking on another
process... but the zombie issue is real does make me think we should
reconsider this and have it wait().
It is a backwards incompatible change if anyone has started using the
Popen context manager to launch a long running subprocess that they
did not want to wait for. That should be exceedingly rare.
I say change the behavior to wait() in 3.3, 3.2.1 and 2.7.2. Keep the
note in the documentation and turn it into something that stands out
better like a note or a warning suggesting that people always call
wait() after the Popen context manager exits if they need to be
compatible with 2.7.1 or earlier.
neologix commented on May 11, 2011
neologixmannequin
Mannequin
More actions
I'm re-opening this issue, since Gregory agrees to change the current behaviour.
Patch attached (along with test and documentation update).
did my commits in the reverse order (default before 3.2), oops. this is fixed. this wasn't ever in 2.7 so no need for the documentation note. i'm not worried about adding a note about 3.2.0 vs 3.2.1 beyond the mention in Misc/NEWS as this was new in 3.2.0 and fixing this behavior is a pretty clear bug fix.
Activity
pitrou commentedon May 9, 2011
I find it a bit strange that Popen.__exit__ closes all standard file descriptors leading to the child process, but doesn't wait for process end afterwards.
(context management support was added in bpo-10554)
briancurtin commentedon May 9, 2011
Seems like it would be enough to add a wait() at the end?
pitrou commentedon May 9, 2011
Probably, perhaps with a try/finally?
I'm not entirely sure this is a good idea, by the way. May there be some
drawbacks?
briancurtin commentedon May 9, 2011
Actually, I don't think the wait() is a good idea. If you want to block and infinitely wait on the process to close, you should do so explicitly.
It's probably better that we try to use terminate() or kill() and raise if that fails.
pitrou commentedon May 9, 2011
Ok.
Uh, I think it's worse. Using a context manager shouldn't do something
potentially destructive like killing a process.
briancurtin commentedon May 9, 2011
Hm, yeah, not sure what I was thinking there.
I'm thinking there's not a lot we can do here, but also not a lot that we should do here. We don't want to wait, and we don't want to close, so maybe we should just document that the usage should be limited only to expecting the open handles to be closed?
pitrou commentedon May 9, 2011
Sounds good indeed :)
briancurtin commentedon May 10, 2011
Looks like we already mention that.
"""
Popen objects are supported as context managers via the with statement, closing any open file descriptors on exit.
"""
Antoine, do you think this should be more strongly worded?
pitrou commentedon May 10, 2011
Well, no, looks ok for me then :)
neologix commentedon May 10, 2011
There's just one thing I'm concerned with.
People using context managers tend to expect the __exit__ method to
perform cleanup actions and release corresponding resources if
necessary, for example closing the underlying file or socket.
So my guess is that most people won't explicitly wait for the process
termination. You can already find such examples inside
test_subprocess.py, Lib/ctypes/util.py (to parse ldconfig's output),
and even in subprocess' documentation:
"""
with Popen(["ifconfig"], stdout=PIPE) as proc:
log.write(proc.stdout.read())
"""
The problem is that until a child process is wait()ed on or its parent
process terminates (so that the child gets re-parented to init), the
child remains a zombie/defunct process. So if you have long-running
process spawning many subprocesses, you could end up having many
zombie processes, potentially filling up your process table at some
point. And this really sucks.
So I think it could be worth mentioning this somewhere in the
documentation (I mean, if we can find find such leaks inside CPython
code base, then it's definitely something that should be documented).
gpshead commentedon May 11, 2011
I didn't initially like the idea of __exit__ blocking on another
process... but the zombie issue is real does make me think we should
reconsider this and have it wait().
It is a backwards incompatible change if anyone has started using the
Popen context manager to launch a long running subprocess that they
did not want to wait for. That should be exceedingly rare.
I say change the behavior to wait() in 3.3, 3.2.1 and 2.7.2. Keep the
note in the documentation and turn it into something that stands out
better like a note or a warning suggesting that people always call
wait() after the Popen context manager exits if they need to be
compatible with 2.7.1 or earlier.
neologix commentedon May 11, 2011
I'm re-opening this issue, since Gregory agrees to change the current behaviour.
Patch attached (along with test and documentation update).
python-dev commentedon May 12, 2011
New changeset 7a3f3ad83676 by Gregory P. Smith in branch 'default':
http://hg.python.org/cpython/rev/7a3f3ad83676
python-dev commentedon May 12, 2011
New changeset b00a64a5cb93 by Gregory P. Smith in branch '3.2':
merge - 7a3f3ad83676 Fixes Issue bpo-12044.
http://hg.python.org/cpython/rev/b00a64a5cb93
gpshead commentedon May 12, 2011
did my commits in the reverse order (default before 3.2), oops. this is fixed. this wasn't ever in 2.7 so no need for the documentation note. i'm not worried about adding a note about 3.2.0 vs 3.2.1 beyond the mention in Misc/NEWS as this was new in 3.2.0 and fixing this behavior is a pretty clear bug fix.
python-dev commentedon May 12, 2011
New changeset 361f87c8f36a by Łukasz Langa in branch 'default':
Cleaned up a backward merge after fixes issue bpo-12044.
http://hg.python.org/cpython/rev/361f87c8f36a
vstinner commentedon Jul 5, 2011
See also issue bpo-12494: "subprocess: check_output() doesn't close pipes on error".