Skip to content

Subprocess: Popen'ed children hang due to open pipes #48362

Closed

Description

@boye
mannequin
BPO 4112
Nosy @giampaolo, @bitdancer

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-04-05.16:18:31.480>
created_at = <Date 2008-10-13.10:36:18.589>
labels = ['type-bug', 'library']
title = "Subprocess: Popen'ed children hang due to open pipes"
updated_at = <Date 2011-04-05.16:18:31.479>
user = 'https://bugs.python.org/boye'

bugs.python.org fields:

activity = <Date 2011-04-05.16:18:31.479>
actor = 'rosslagerwall'
assignee = 'none'
closed = True
closed_date = <Date 2011-04-05.16:18:31.480>
closer = 'rosslagerwall'
components = ['Library (Lib)']
creation = <Date 2008-10-13.10:36:18.589>
creator = 'boye'
dependencies = []
files = []
hgrepos = []
issue_num = 4112
keywords = []
message_count = 5.0
messages = ['74680', '109592', '110209', '125987', '133045']
nosy_count = 5.0
nosy_names = ['giampaolo.rodola', 'boye', 'r.david.murray', 'BreamoreBoy', 'rosslagerwall']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'test needed'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue4112'
versions = ['Python 3.1', 'Python 2.7']

Activity

  1. boye commented on Oct 13, 2008

    boyemannequin
    MannequinAuthor

    subprocess.Popen.wait() hangs if you have spawned multiple child
    processes that do not terminate until their standard input is closed
    ('cat' is example of such a process). This happens on POSIX platforms.
    I'm using Python 2.5.1, but I believe the issue is present in the SVN
    trunk version of subprocess as well.

    Here is a test program:
    --------------------------

    import subprocess, sys
    
    p1 = subprocess.Popen("cat", bufsize=0, stdin=subprocess.PIPE)
    p2 = subprocess.Popen("cat", bufsize=0, stdin=subprocess.PIPE)
    
    p1.stdin.close()
    ret = p1.wait()
    print >>sys.stderr, "Child 1 wait completed with ret", ret
    
    p2.stdin.close()
    ret = p2.wait()
    print >>sys.stderr, "Child 2 wait completed with ret", ret, "Bye bye"

    The call to p1.wait() will never return. If p2.wait is called first,
    the program terminates cleanly.

    p1 never terminates because p1.stdin is duplicated in the (second) child
    process when the parent process forks as part of the call to p2 =
    subprocess.Popen(). The line p1.stdin.close() thus has no effect.

    I am not sure whether this is a bug or deliberate design, but the
    behavior above seems a bit weird to me, and it took me quite some time
    to figure out what happened. However, the proposed fix below of course
    has side effects that may be undesirable.

    wait() will not hang if the "close on exec" flag is set on the
    subprocess pipes. Conveniently, the method _set_cloexec_flag already
    exists to do this, so the fix amounts to calling this from
    Popen._get_handles:

    elif stdin == PIPE:
    p2cread, p2cwrite = os.pipe()
    self._set_cloexec_flag(p2cwrite)

    The last line is the added one, similar lines must be added for stdout
    and stderr.

    Alternatively, the test program above will terminate cleanly if
    "p1._set_cloexec_flag(p1.stdin)" is added before the creation of p2.

  2. added
    stdlibStandard Library Python modules in the Lib/ directory
    type-bugAn unexpected behavior, bug, or error
    on Oct 13, 2008
  3. BreamoreBoy commented on Jul 8, 2010

    BreamoreBoymannequin
    Mannequin

    Would someone with knowledge of subprocess please comment on this.

  4. bitdancer commented on Jul 13, 2010

    @bitdancer
    Member

    The suggestion looks reasonable to me, and the current behavior does not look like an intentional design.

    Boye, would you be interested in proposing a patch with unit tests?

  5. rosslagerwall commented on Jan 11, 2011

    rosslagerwallmannequin
    Mannequin

    This issue has been fixed on 3.2.

  6. rosslagerwall commented on Apr 5, 2011

    rosslagerwallmannequin
    Mannequin

    This has been fixed with all the subprocess improvements in between 3.1 and 3.2 but the decision has been taken (msg125910) not to backport the fix to 3.1 and 2.7 which would involve a C extension.

    However, a workaround on 2.7 and 3.1 is to set close_fds=True.

    Closing as "wont fix".

  7. transferred this issue fromon Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions