Skip to content

weak dict iterators are fragile because of unpredictable GC runs #51354

Closed

Description

@pitrou
BPO 7105
Nosy @mdickinson, @pitrou, @kristjanvalur, @devdanzin, @jparise, @benjaminp, @asqui
PRs
  • bpo-40895: Update weakref documentation to remove old warnings #20687
  • Files
  • weakiter3.patch
  • issue7105.patch
  • weakref.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2014-04-29.10:06:43.518>
    created_at = <Date 2009-10-11.17:29:21.257>
    labels = ['type-bug', 'library']
    title = 'weak dict iterators are fragile because of unpredictable GC runs'
    updated_at = <Date 2020-06-08.05:00:46.316>
    user = 'https://github.com/pitrou'

    bugs.python.org fields:

    activity = <Date 2020-06-08.05:00:46.316>
    actor = 'gvanrossum'
    assignee = 'none'
    closed = True
    closed_date = <Date 2014-04-29.10:06:43.518>
    closer = 'kristjan.jonsson'
    components = ['Library (Lib)']
    creation = <Date 2009-10-11.17:29:21.257>
    creator = 'pitrou'
    dependencies = []
    files = ['15114', '32689', '32932']
    hgrepos = []
    issue_num = 7105
    keywords = ['patch']
    message_count = 30.0
    messages = ['93863', '93865', '93912', '93934', '97392', '97425', '111259', '142392', '203250', '203286', '203318', '203345', '203349', '203354', '203355', '203356', '204975', '204980', '205013', '205014', '205019', '205090', '205092', '205093', '205124', '205215', '205224', '205285', '217444', '266818']
    nosy_count = 14.0
    nosy_names = ['dcjim', 'tseaver', 'mark.dickinson', 'pitrou', 'kristjan.jonsson', 'ajaksu2', 'jon', 'benjamin.peterson', 'vdupras', 'elachuni', 'neologix', 'python-dev', 'qelan', 'dfortunov']
    pr_nums = ['20687']
    priority = 'normal'
    resolution = 'fixed'
    stage = None
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue7105'
    versions = ['Python 2.6', 'Python 2.7']

    Activity

    1. pitrou commented on Oct 11, 2009

      @pitrou
      MemberAuthor

      As mentioned in bpo-7060, weak dict iterators are easily broken by
      cyclic garbage collection changing the size of the underlying dict storage:

      File "/home/rdmurray/python/py3k/Lib/weakref.py", line 121, in items
      for wr in self.data.values():
      RuntimeError: dictionary changed size during iteration

      One possible solution is to delay all removals until all iterators are
      done. Here is a context manager-based solution.

    2. added
      stdlibStandard Library Python modules in the Lib/ directory
      type-bugAn unexpected behavior, bug, or error
      on Oct 11, 2009
    3. gvanrossum commented on Oct 11, 2009

      @gvanrossum
      Member

      delay all removals until all iterators are done

      +1

    4. pitrou commented on Oct 12, 2009

      @pitrou
      MemberAuthor

      This new patch makes it possible to mutate the dict without messing with
      the delayed removal when an iterator exists.

    5. pitrou commented on Oct 13, 2009

      @pitrou
      MemberAuthor

      It occurs weaksets have the same problem, here is a new patch fixing
      them as well.

    6. benjaminp commented on Jan 8, 2010

      @benjaminp
      Contributor

      LGTM

    7. pitrou commented on Jan 8, 2010

      @pitrou
      MemberAuthor

      Committed in r77365 (py3k) and r77366 (3.1). Thank you.

    8. mdickinson commented on Jul 23, 2010

      @mdickinson
      Member

      The issue still exists in 2.6 and 2.7.

      Closing bpo-839159 as a duplicate of this one.

    9. qelan commented on Aug 18, 2011

      qelanmannequin
      Mannequin

      Any plans on actually patching this in 2.7 any time soon? This is affecting our software and hanging it on random occasions.

    10. vstinner commented on Nov 18, 2013

      @vstinner
      Member

      What is the status of this issue? Does anyone still want to backport the fix to Python 2.7?

      (I found this issue while searching for test_multiprocessing failures, and I found bpo-7060 which refers this issue.)

    11. kristjanvalur commented on Nov 18, 2013

      kristjanvalurmannequin
      Mannequin

      Attached is a version for 2.7

      Btw, I think a cleaner way to deal with unpredictable GC runs is to be able to temporarily disable GC with a context manager.

    12. pitrou commented on Nov 18, 2013

      @pitrou
      MemberAuthor

      Btw, I think a cleaner way to deal with unpredictable GC runs is to be able to temporarily disable GC with a context manager

      Disabling the GC in a library function sounds very ugly to me.

    13. 6 remaining items

    14. gvanrossum commented on Dec 2, 2013

      @gvanrossum
      Member

      I appreciate the simplicity, but I don't think it is acceptable -- if the dict is large, materializing the entire list of keys/values/items might allocate a prohibitive amount of memory. It's worse if you have code that is expected to break out of the loop.

    15. kristjanvalur commented on Dec 2, 2013

      kristjanvalurmannequin
      Mannequin

      Yes, the old memory argument.
      But is it valid? Is there a conceivable application where a dict of weak references would be storing a large chunk of the application memory?
      Remember, all of the data must be referred to from elsewhere, or else, the weak refs would not exist. An extra list of pointers is unlikely to make a difference.
      I think the chief reason to use iterators has to do with performance by avoiding the creation of temporary objects, not saving memory per-se.

      Before the invention of "iteritems()" and friends, all such iteration was by lists (and hence, memory usage). We should try to remain nimble enough so that we can undo an optimization previously done, if the requirements merit us doing so.

      As a completely unrelated example of such nimbleness: Faced with stricter regulations in the 70s, american car makers had to sell their muscle cars with increasingly less powerful engines, efectively rolling back previous optimizations :)

      Anyway, it's not for me to decide. We have currently three options:
      a) my first patch, which is a duplication of the 3.x work but is non-trivial and could bring stability issues
      b) my second patch, which will increase memory use, but to no more than previous versions of python used while iterating
      c) do nothing and have iterations over weak dicts randomly break when an underlying cycle is unraveled during iteration.

      Cheers!

    16. pitrou commented on Dec 2, 2013

      @pitrou
      MemberAuthor

      Anyway, it's not for me to decide. We have currently three options:
      a) my first patch, which is a duplication of the 3.x work but is
      non-trivial and could bring stability issues
      b) my second patch, which will increase memory use, but to no more
      than previous versions of python used while iterating
      c) do nothing and have iterations over weak dicts randomly break when
      an underlying cycle is unraveled during iteration.

      Either a) or c), for me. We shouldn't change semantics in bugfix
      releases.

    17. gvanrossum commented on Dec 2, 2013

      @gvanrossum
      Member

      I'm with Antoine. Have we heard of any problems with the 3.x version of the patch? How different is it?

    18. kristjanvalur commented on Dec 3, 2013

      kristjanvalurmannequin
      Mannequin

      Strictly speaking b) is not a semantic change. Depending on your semantic definition of semantics. At any rate it is even less so than a) since the temporary list is hidden from view and the only side effect is additional memory usage.

    19. kristjanvalur commented on Dec 3, 2013

      kristjanvalurmannequin
      Mannequin

      d), We could also simply issue a (documentation) warning, that the "iterator" methods of these dictionares are known to be fragile, and recommend that people use the keys(), values() and items() instead.

    20. pitrou commented on Dec 3, 2013

      @pitrou
      MemberAuthor

      d) sounds like a good enough resolution at this point.

    21. gvanrossum commented on Dec 3, 2013

      @gvanrossum
      Member

      I'm not sure I understand the hesitation about backporting the Python 3 solution. We're acknowledging it's a bug, so the fix is not a feature. The Python 3 solution is the future. So why not fix it?

    22. kristjanvalur commented on Dec 4, 2013

      kristjanvalurmannequin
      Mannequin

      That's the spirit, Guido :)
      I just think people are being extra careful after the "regression" introduced in 2.7.5.
      However, IMHO we must never let the odd mistake scare us from making necessary moves.
      Unless Antoine explicitly objects, I think I'll submit my patch from november and we'll just watch what happens.

    23. pitrou commented on Dec 4, 2013

      @pitrou
      MemberAuthor

      I'm ok with the backport.

    24. python-dev commented on Dec 5, 2013

      python-devmannequin
      Mannequin

      New changeset 03fcc12282fc by Kristján Valur Jónsson in branch '2.7':
      Issue bpo-7105: weak dict iterators are fragile because of unpredictable GC runs
      http://hg.python.org/cpython/rev/03fcc12282fc

    25. pitrou commented on Apr 28, 2014

      @pitrou
      MemberAuthor

      Since this is backported, shouldn't it be closed?

    26. neologix commented on Jun 1, 2016

      neologixmannequin
      Mannequin

      Shouldn't the documentation be updated?

      https://docs.python.org/3.6/library/weakref.html#weakref.WeakKeyDictionary

      Note Caution: Because a WeakKeyDictionary is built on top of a Python dictionary, it must not change size when iterating over it. This can be difficult to ensure for a WeakKeyDictionary because actions performed by the program during iteration may cause items in the dictionary to vanish “by magic” (as a side effect of garbage collection).

    27. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions