Skip to content

subprocess.Popen.__exit__ doesn't wait for process end #56253

Closed

Description

@pitrou
BPO 12044
Nosy @gpshead, @pitrou, @vstinner, @briancurtin
Files
  • subprocess_zombie.diff
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/gpshead'
    closed_at = <Date 2011-05-12.05:23:31.318>
    created_at = <Date 2011-05-09.19:24:14.200>
    labels = ['type-bug', 'library']
    title = "subprocess.Popen.__exit__ doesn't wait for process end"
    updated_at = <Date 2011-07-05.20:36:58.179>
    user = 'https://github.com/pitrou'

    bugs.python.org fields:

    activity = <Date 2011-07-05.20:36:58.179>
    actor = 'vstinner'
    assignee = 'gregory.p.smith'
    closed = True
    closed_date = <Date 2011-05-12.05:23:31.318>
    closer = 'gregory.p.smith'
    components = ['Library (Lib)']
    creation = <Date 2011-05-09.19:24:14.200>
    creator = 'pitrou'
    dependencies = []
    files = ['21969']
    hgrepos = []
    issue_num = 12044
    keywords = ['patch']
    message_count = 17.0
    messages = ['135635', '135641', '135642', '135648', '135649', '135652', '135658', '135675', '135690', '135715', '135765', '135783', '135814', '135815', '135816', '135841', '139904']
    nosy_count = 6.0
    nosy_names = ['gregory.p.smith', 'pitrou', 'vstinner', 'brian.curtin', 'neologix', 'python-dev']
    pr_nums = []
    priority = 'normal'
    resolution = 'accepted'
    stage = None
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue12044'
    versions = ['Python 3.2', 'Python 3.3']

    Activity

    1. pitrou commented on May 9, 2011

      @pitrou
      MemberAuthor

      I find it a bit strange that Popen.__exit__ closes all standard file descriptors leading to the child process, but doesn't wait for process end afterwards.
      (context management support was added in bpo-10554)

    2. added
      type-bugAn unexpected behavior, bug, or error
      on May 9, 2011
    3. briancurtin commented on May 9, 2011

      @briancurtin
      Member

      Seems like it would be enough to add a wait() at the end?

      diff -r 9e473917cbfb Lib/subprocess.py
      --- a/Lib/subprocess.py Mon May 09 21:17:02 2011 +0200
      +++ b/Lib/subprocess.py Mon May 09 15:30:02 2011 -0500
      @@ -796,6 +796,7 @@
                   self.stderr.close()
               if self.stdin:
                   self.stdin.close()
      +        self.wait()
           def __del__(self, _maxsize=sys.maxsize, _active=_active):
               if not self._child_created:
    4. pitrou commented on May 9, 2011

      @pitrou
      MemberAuthor

      Seems like it would be enough to add a wait() at the end?

      Probably, perhaps with a try/finally?

      I'm not entirely sure this is a good idea, by the way. May there be some
      drawbacks?

    5. briancurtin commented on May 9, 2011

      @briancurtin
      Member

      Actually, I don't think the wait() is a good idea. If you want to block and infinitely wait on the process to close, you should do so explicitly.

      It's probably better that we try to use terminate() or kill() and raise if that fails.

    6. pitrou commented on May 9, 2011

      @pitrou
      MemberAuthor

      Actually, I don't think the wait() is a good idea. If you want to
      block and infinitely wait on the process to close, you should do so
      explicitly.

      Ok.

      It's probably better that we try to use terminate() or kill() and raise if that fails.

      Uh, I think it's worse. Using a context manager shouldn't do something
      potentially destructive like killing a process.

    7. briancurtin commented on May 9, 2011

      @briancurtin
      Member

      Hm, yeah, not sure what I was thinking there.

      I'm thinking there's not a lot we can do here, but also not a lot that we should do here. We don't want to wait, and we don't want to close, so maybe we should just document that the usage should be limited only to expecting the open handles to be closed?

    8. pitrou commented on May 9, 2011

      @pitrou
      MemberAuthor

      I'm thinking there's not a lot we can do here, but also not a lot that
      we should do here. We don't want to wait, and we don't want to close,
      so maybe we should just document that the usage should be limited only
      to expecting the open handles to be closed?

      Sounds good indeed :)

    9. briancurtin commented on May 10, 2011

      @briancurtin
      Member

      Looks like we already mention that.

      """
      Popen objects are supported as context managers via the with statement, closing any open file descriptors on exit.
      """

      Antoine, do you think this should be more strongly worded?

    10. pitrou commented on May 10, 2011

      @pitrou
      MemberAuthor

      Well, no, looks ok for me then :)

    11. neologix commented on May 10, 2011

      neologixmannequin
      Mannequin

      There's just one thing I'm concerned with.
      People using context managers tend to expect the __exit__ method to
      perform cleanup actions and release corresponding resources if
      necessary, for example closing the underlying file or socket.
      So my guess is that most people won't explicitly wait for the process
      termination. You can already find such examples inside
      test_subprocess.py, Lib/ctypes/util.py (to parse ldconfig's output),
      and even in subprocess' documentation:

      """
      with Popen(["ifconfig"], stdout=PIPE) as proc:
      log.write(proc.stdout.read())
      """

      The problem is that until a child process is wait()ed on or its parent
      process terminates (so that the child gets re-parented to init), the
      child remains a zombie/defunct process. So if you have long-running
      process spawning many subprocesses, you could end up having many
      zombie processes, potentially filling up your process table at some
      point. And this really sucks.
      So I think it could be worth mentioning this somewhere in the
      documentation (I mean, if we can find find such leaks inside CPython
      code base, then it's definitely something that should be documented).

    12. gpshead commented on May 11, 2011

      @gpshead
      Member

      I didn't initially like the idea of __exit__ blocking on another
      process... but the zombie issue is real does make me think we should
      reconsider this and have it wait().

      It is a backwards incompatible change if anyone has started using the
      Popen context manager to launch a long running subprocess that they
      did not want to wait for. That should be exceedingly rare.

      I say change the behavior to wait() in 3.3, 3.2.1 and 2.7.2. Keep the
      note in the documentation and turn it into something that stands out
      better like a note or a warning suggesting that people always call
      wait() after the Popen context manager exits if they need to be
      compatible with 2.7.1 or earlier.

    13. neologix commented on May 11, 2011

      neologixmannequin
      Mannequin

      I'm re-opening this issue, since Gregory agrees to change the current behaviour.
      Patch attached (along with test and documentation update).

    14. added
      stdlibStandard Library Python modules in the Lib/ directory
      on May 11, 2011
    15. self-assigned this
      on May 11, 2011
    16. python-dev commented on May 12, 2011

      python-devmannequin
      Mannequin

      New changeset 7a3f3ad83676 by Gregory P. Smith in branch 'default':

    17. python-dev commented on May 12, 2011

      python-devmannequin
      Mannequin

      New changeset b00a64a5cb93 by Gregory P. Smith in branch '3.2':
      merge - 7a3f3ad83676 Fixes Issue bpo-12044.
      http://hg.python.org/cpython/rev/b00a64a5cb93

    18. gpshead commented on May 12, 2011

      @gpshead
      Member

      did my commits in the reverse order (default before 3.2), oops. this is fixed. this wasn't ever in 2.7 so no need for the documentation note. i'm not worried about adding a note about 3.2.0 vs 3.2.1 beyond the mention in Misc/NEWS as this was new in 3.2.0 and fixing this behavior is a pretty clear bug fix.

    19. python-dev commented on May 12, 2011

      python-devmannequin
      Mannequin

      New changeset 361f87c8f36a by Łukasz Langa in branch 'default':
      Cleaned up a backward merge after fixes issue bpo-12044.
      http://hg.python.org/cpython/rev/361f87c8f36a

    20. vstinner commented on Jul 5, 2011

      @vstinner
      Member

      See also issue bpo-12494: "subprocess: check_output() doesn't close pipes on error".

    21. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions