You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
weak dict iterators are fragile because of unpredictable GC runs #51354
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee=Noneclosed_at=<Date2014-04-29.10:06:43.518>created_at=<Date2009-10-11.17:29:21.257>labels= ['type-bug', 'library']
title='weak dict iterators are fragile because of unpredictable GC runs'updated_at=<Date2020-06-08.05:00:46.316>user='https://github.com/pitrou'
As mentioned in bpo-7060, weak dict iterators are easily broken by
cyclic garbage collection changing the size of the underlying dict storage:
File "/home/rdmurray/python/py3k/Lib/weakref.py", line 121, in items
for wr in self.data.values():
RuntimeError: dictionary changed size during iteration
One possible solution is to delay all removals until all iterators are
done. Here is a context manager-based solution.
I appreciate the simplicity, but I don't think it is acceptable -- if the dict is large, materializing the entire list of keys/values/items might allocate a prohibitive amount of memory. It's worse if you have code that is expected to break out of the loop.
kristjanvalur commented on Dec 2, 2013
kristjanvalurmannequin
Mannequin
More actions
Yes, the old memory argument.
But is it valid? Is there a conceivable application where a dict of weak references would be storing a large chunk of the application memory?
Remember, all of the data must be referred to from elsewhere, or else, the weak refs would not exist. An extra list of pointers is unlikely to make a difference.
I think the chief reason to use iterators has to do with performance by avoiding the creation of temporary objects, not saving memory per-se.
Before the invention of "iteritems()" and friends, all such iteration was by lists (and hence, memory usage). We should try to remain nimble enough so that we can undo an optimization previously done, if the requirements merit us doing so.
As a completely unrelated example of such nimbleness: Faced with stricter regulations in the 70s, american car makers had to sell their muscle cars with increasingly less powerful engines, efectively rolling back previous optimizations :)
Anyway, it's not for me to decide. We have currently three options:
a) my first patch, which is a duplication of the 3.x work but is non-trivial and could bring stability issues
b) my second patch, which will increase memory use, but to no more than previous versions of python used while iterating
c) do nothing and have iterations over weak dicts randomly break when an underlying cycle is unraveled during iteration.
Anyway, it's not for me to decide. We have currently three options:
a) my first patch, which is a duplication of the 3.x work but is
non-trivial and could bring stability issues
b) my second patch, which will increase memory use, but to no more
than previous versions of python used while iterating
c) do nothing and have iterations over weak dicts randomly break when
an underlying cycle is unraveled during iteration.
Either a) or c), for me. We shouldn't change semantics in bugfix
releases.
I'm with Antoine. Have we heard of any problems with the 3.x version of the patch? How different is it?
kristjanvalur commented on Dec 3, 2013
kristjanvalurmannequin
Mannequin
More actions
Strictly speaking b) is not a semantic change. Depending on your semantic definition of semantics. At any rate it is even less so than a) since the temporary list is hidden from view and the only side effect is additional memory usage.
kristjanvalur commented on Dec 3, 2013
kristjanvalurmannequin
Mannequin
More actions
d), We could also simply issue a (documentation) warning, that the "iterator" methods of these dictionares are known to be fragile, and recommend that people use the keys(), values() and items() instead.
I'm not sure I understand the hesitation about backporting the Python 3 solution. We're acknowledging it's a bug, so the fix is not a feature. The Python 3 solution is the future. So why not fix it?
kristjanvalur commented on Dec 4, 2013
kristjanvalurmannequin
Mannequin
More actions
That's the spirit, Guido :)
I just think people are being extra careful after the "regression" introduced in 2.7.5.
However, IMHO we must never let the odd mistake scare us from making necessary moves.
Unless Antoine explicitly objects, I think I'll submit my patch from november and we'll just watch what happens.
Note Caution: Because a WeakKeyDictionary is built on top of a Python dictionary, it must not change size when iterating over it. This can be difficult to ensure for a WeakKeyDictionary because actions performed by the program during iteration may cause items in the dictionary to vanish “by magic” (as a side effect of garbage collection).
Activity
pitrou commentedon Oct 11, 2009
As mentioned in bpo-7060, weak dict iterators are easily broken by
cyclic garbage collection changing the size of the underlying dict storage:
File "/home/rdmurray/python/py3k/Lib/weakref.py", line 121, in items
for wr in self.data.values():
RuntimeError: dictionary changed size during iteration
One possible solution is to delay all removals until all iterators are
done. Here is a context manager-based solution.
gvanrossum commentedon Oct 11, 2009
+1
pitrou commentedon Oct 12, 2009
This new patch makes it possible to mutate the dict without messing with
the delayed removal when an iterator exists.
pitrou commentedon Oct 13, 2009
It occurs weaksets have the same problem, here is a new patch fixing
them as well.
benjaminp commentedon Jan 8, 2010
LGTM
pitrou commentedon Jan 8, 2010
Committed in r77365 (py3k) and r77366 (3.1). Thank you.
mdickinson commentedon Jul 23, 2010
The issue still exists in 2.6 and 2.7.
Closing bpo-839159 as a duplicate of this one.
qelan commentedon Aug 18, 2011
Any plans on actually patching this in 2.7 any time soon? This is affecting our software and hanging it on random occasions.
vstinner commentedon Nov 18, 2013
What is the status of this issue? Does anyone still want to backport the fix to Python 2.7?
(I found this issue while searching for test_multiprocessing failures, and I found bpo-7060 which refers this issue.)
kristjanvalur commentedon Nov 18, 2013
Attached is a version for 2.7
Btw, I think a cleaner way to deal with unpredictable GC runs is to be able to temporarily disable GC with a context manager.
pitrou commentedon Nov 18, 2013
Disabling the GC in a library function sounds very ugly to me.
6 remaining items
gvanrossum commentedon Dec 2, 2013
I appreciate the simplicity, but I don't think it is acceptable -- if the dict is large, materializing the entire list of keys/values/items might allocate a prohibitive amount of memory. It's worse if you have code that is expected to break out of the loop.
kristjanvalur commentedon Dec 2, 2013
Yes, the old memory argument.
But is it valid? Is there a conceivable application where a dict of weak references would be storing a large chunk of the application memory?
Remember, all of the data must be referred to from elsewhere, or else, the weak refs would not exist. An extra list of pointers is unlikely to make a difference.
I think the chief reason to use iterators has to do with performance by avoiding the creation of temporary objects, not saving memory per-se.
Before the invention of "iteritems()" and friends, all such iteration was by lists (and hence, memory usage). We should try to remain nimble enough so that we can undo an optimization previously done, if the requirements merit us doing so.
As a completely unrelated example of such nimbleness: Faced with stricter regulations in the 70s, american car makers had to sell their muscle cars with increasingly less powerful engines, efectively rolling back previous optimizations :)
Anyway, it's not for me to decide. We have currently three options:
a) my first patch, which is a duplication of the 3.x work but is non-trivial and could bring stability issues
b) my second patch, which will increase memory use, but to no more than previous versions of python used while iterating
c) do nothing and have iterations over weak dicts randomly break when an underlying cycle is unraveled during iteration.
Cheers!
pitrou commentedon Dec 2, 2013
Either a) or c), for me. We shouldn't change semantics in bugfix
releases.
gvanrossum commentedon Dec 2, 2013
I'm with Antoine. Have we heard of any problems with the 3.x version of the patch? How different is it?
kristjanvalur commentedon Dec 3, 2013
Strictly speaking b) is not a semantic change. Depending on your semantic definition of semantics. At any rate it is even less so than a) since the temporary list is hidden from view and the only side effect is additional memory usage.
kristjanvalur commentedon Dec 3, 2013
d), We could also simply issue a (documentation) warning, that the "iterator" methods of these dictionares are known to be fragile, and recommend that people use the keys(), values() and items() instead.
pitrou commentedon Dec 3, 2013
d) sounds like a good enough resolution at this point.
gvanrossum commentedon Dec 3, 2013
I'm not sure I understand the hesitation about backporting the Python 3 solution. We're acknowledging it's a bug, so the fix is not a feature. The Python 3 solution is the future. So why not fix it?
kristjanvalur commentedon Dec 4, 2013
That's the spirit, Guido :)
I just think people are being extra careful after the "regression" introduced in 2.7.5.
However, IMHO we must never let the odd mistake scare us from making necessary moves.
Unless Antoine explicitly objects, I think I'll submit my patch from november and we'll just watch what happens.
pitrou commentedon Dec 4, 2013
I'm ok with the backport.
python-dev commentedon Dec 5, 2013
New changeset 03fcc12282fc by Kristján Valur Jónsson in branch '2.7':
Issue bpo-7105: weak dict iterators are fragile because of unpredictable GC runs
http://hg.python.org/cpython/rev/03fcc12282fc
pitrou commentedon Apr 28, 2014
Since this is backported, shouldn't it be closed?
neologix commentedon Jun 1, 2016
Shouldn't the documentation be updated?
https://docs.python.org/3.6/library/weakref.html#weakref.WeakKeyDictionary
Note Caution: Because a WeakKeyDictionary is built on top of a Python dictionary, it must not change size when iterating over it. This can be difficult to ensure for a WeakKeyDictionary because actions performed by the program during iteration may cause items in the dictionary to vanish “by magic” (as a side effect of garbage collection).