Skip to content

PyFile_FromString leaks file descriptors in python 2.7 #58710

Closed

Description

@doko42
BPO 14505
Nosy @doko42, @pitrou, @vstinner

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-04-05.12:16:26.487>
created_at = <Date 2012-04-05.09:01:22.822>
labels = ['interpreter-core', 'performance']
title = 'PyFile_FromString leaks file descriptors in python 2.7'
updated_at = <Date 2012-04-05.17:25:05.928>
user = 'https://github.com/doko42'

bugs.python.org fields:

activity = <Date 2012-04-05.17:25:05.928>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2012-04-05.12:16:26.487>
closer = 'pitrou'
components = ['Interpreter Core']
creation = <Date 2012-04-05.09:01:22.822>
creator = 'doko'
dependencies = []
files = []
hgrepos = []
issue_num = 14505
keywords = []
message_count = 8.0
messages = ['157555', '157559', '157560', '157561', '157563', '157574', '157575', '157607']
nosy_count = 4.0
nosy_names = ['doko', 'pitrou', 'vstinner', 'python-dev']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue14505'
versions = ['Python 2.7']

Activity

  1. doko42 commented on Apr 5, 2012

    @doko42
    MemberAuthor

    [forwarded from http://bugs.debian.org/664529]

    seen with 2.7.3 rc2

    File descriptors opened by PyFile_FromString don't get closed when the
    reference count is decreased.

    Here's my test program, pythony.c:

    #include <Python.h>
    
    int main()
    {
      int i = 0;
      PyObject *obj;
    
      Py_Initialize();
      while (i++ < 5) {
        obj = PyFile_FromString("hello.py", "r");
        assert(obj);
        Py_DECREF(obj);
      }
      Py_Finalize();
    }

    hello.py is 'print("hello world")'.

    I'm compiling it with both Python 2.6 and 2.7.
    $ gcc pythony.c -lpython2.6 -L/usr/lib/python2.6/config -I/usr/include/python2.6/ -o pythony-2.6
    $ gcc pythony.c -lpython2.7 -L/usr/lib/python2.7/config -I/usr/include/python2.7/ -o pythony-2.7

    $ strace ./pythony-2.6 2>&1 | tail -n 20
    ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
    ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fffb1d097b0) = -1 EINVAL (Invalid argument)
    ioctl(2, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fffb1d097b0) = -1 EINVAL (Invalid argument)
    open("hello.py", O_RDONLY)              = 3
    fstat(3, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
    close(3)                                = 0
    open("hello.py", O_RDONLY)              = 3
    fstat(3, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
    close(3)                                = 0
    open("hello.py", O_RDONLY)              = 3
    fstat(3, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
    close(3)                                = 0
    open("hello.py", O_RDONLY)              = 3
    fstat(3, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
    close(3)                                = 0
    open("hello.py", O_RDONLY)              = 3
    fstat(3, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
    close(3)                                = 0
    rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f1e1a0224f0}, {0x7f1e1a49a160, [], SA_RESTORER, 0x7f1e1a0224f0}, 8) = 0
    exit_group(0)                           = ?
    
    
    $ strace ./pythony-2.7 2>&1 | tail -n 20
    fstat(4, {st_mode=S_IFREG|0644, st_size=1950, ...}) = 0
    read(4, "", 4096)                       = 0
    close(4)                                = 0
    munmap(0x7fa41f10f000, 4096)            = 0
    close(3)                                = 0
    ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
    ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7ffff7bd33f0) = -1 EINVAL (Invalid argument)
    ioctl(2, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7ffff7bd33f0) = -1 EINVAL (Invalid argument)
    open("hello.py", O_RDONLY)              = 3
    fstat(3, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
    open("hello.py", O_RDONLY)              = 4
    fstat(4, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
    open("hello.py", O_RDONLY)              = 5
    fstat(5, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
    open("hello.py", O_RDONLY)              = 6
    fstat(6, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
    open("hello.py", O_RDONLY)              = 7
    fstat(7, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
    rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fa4206e24f0}, {0x7fa420b8dd50, [], SA_RESTORER, 0x7fa4206e24f0}, 8) = 0
    exit_group(0)                           = ?

    The Python 2.7 version never calls close, not even at Py_Finalize().

    On #d-d, jwilk suspected that this change might be the cause:
    http://hg.python.org/cpython/rev/0f5b64630fda/#l4.46

  2. vstinner commented on Apr 5, 2012

    @vstinner
    Member

    File descriptors opened by PyFile_FromString don't get closed
    when the reference count is decreased.

    Correct, PyFile_FromString() doesn't close the file descriptor, even if you call its close() method. You have to call manually fclose(file->f_fp).

    Or you can use PyFile_FromFile:

    fp = fopen(name, mode);
    if (fp == NULL) ...
    obj = PyFile_FromFile(fp, name, mode, fclose);
    if (obj == NULL) {
       /* no need to call fclose(fp) here, it's done by PyFile_FromFile() */
       ...
    }
    ...
    Py_DECREF(obj);

    Would you like to write a patch for the documentation?

  3. pitrou commented on Apr 5, 2012

    @pitrou
    Member

    Victor, that sounds like a strange behaviour to me. PyFile_FromString is a public API and maybe it shouldn't have changed between 2.6 and 2.7.

  4. vstinner commented on Apr 5, 2012

    @vstinner
    Member

    Victor, that sounds like a strange behaviour to me.

    We may change PyFile_FromString() to call fclose() when the file is
    closed explicitly (call its close() method), but it may break backward
    compatibility. I prefer to document the behaviour instead.

    PyFile_FromString is a public API and maybe it shouldn't
    have changed between 2.6 and 2.7.

    PyFile_FromString() didn't change in Python 2.7.

    I changed PyFile_FromFile() in Python 2.7 to fix the issue bpo-7732.

    changeset: 72456:0f5b64630fda
    branch: 2.7
    parent: 72450:c02e790c4535
    user: Victor Stinner <victor.stinner@haypocalc.com>
    date: Fri Sep 23 19:37:03 2011 +0200
    files: Doc/c-api/file.rst Lib/test/test_import.py Misc/NEWS
    Objects/fileobject.c Python/import.c
    description:
    Issue bpo-7732: Fix a crash on importing a module if a directory has the same name
    than a Python module (e.g. "__init__.py"): don't close the file twice.

    PyFile_FromFile() does also close the file if PyString_FromString() failed. It
    did already close the file on fill_file_fields() error (e.g. if the file is a
    directory).

  5. pitrou commented on Apr 5, 2012

    @pitrou
    Member

    Victor, what exactly are you talking about? http://hg.python.org/cpython/rev/0f5b64630fda/ *does* change PyFile_FromString.
    And Matthias mentioned that the problem didn't occur with 2.6.

  6. python-dev commented on Apr 5, 2012

    python-devmannequin
    Mannequin

    New changeset 8258e5fa4a19 by Antoine Pitrou in branch '2.7':
    Issue bpo-14505: Fix file descriptor leak when deallocating file objects created with PyFile_FromString().
    http://hg.python.org/cpython/rev/8258e5fa4a19

  7. pitrou commented on Apr 5, 2012

    @pitrou
    Member

    Matthias, this should be fixed now.

  8. vstinner commented on Apr 5, 2012

    @vstinner
    Member

    Victor, what exactly are you talking about? http://hg.python.org/cpython/rev/0f5b64630fda/ *does* change PyFile_FromString.

    Oh. I don't know (remember) why I did this change!? I suppose that I
    changed it to test something and then forget to remove the temporary
    hack :-/

  9. transferred this issue fromon Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    interpreter-core(Objects, Python, Grammar, and Parser dirs)performancePerformance or resource usage

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions