Repository navigation
Closed
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)release-blockertype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/larryhastings'
closed_at = <Date 2015-09-06.07:53:19.607>
created_at = <Date 2015-08-21.22:02:50.644>
labels = ['interpreter-core', 'type-bug', 'release-blocker']
title = 'The type of cached objects is mutable'
updated_at = <Date 2021-04-30.11:27:56.656>
user = 'https://github.com/serhiy-storchaka'bugs.python.org fields:
activity = <Date 2021-04-30.11:27:56.656>
actor = 'erlendaasland'
assignee = 'larry'
closed = True
closed_date = <Date 2015-09-06.07:53:19.607>
closer = 'larry'
components = ['Interpreter Core']
creation = <Date 2015-08-21.22:02:50.644>
creator = 'serhiy.storchaka'
dependencies = []
files = ['40310', '40312']
hgrepos = []
issue_num = 24912
keywords = ['patch']
message_count = 69.0
messages = ['248981', '248982', '248983', '249019', '249180', '249189', '249198', '249345', '249350', '249351', '249354', '249357', '249360', '249363', '249369', '249388', '249392', '249393', '249396', '249397', '249398', '249438', '249446', '249450', '249455', '249456', '249457', '249467', '249468', '249470', '249471', '249473', '249483', '249484', '249489', '249495', '249499', '249500', '249502', '249504', '249505', '249506', '249509', '249510', '249511', '249604', '249607', '249618', '249623', '249685', '249854', '249869', '249871', '249888', '249889', '249890', '249891', '249893', '249930', '249931', '249934', '249938', '249940', '249943', '249944', '249948', '249954', '249979', '392411']
nosy_count = 18.0
nosy_names = ['lemburg', 'gvanrossum', 'brett.cannon', 'rhettinger', 'ncoghlan', 'pitrou', 'vstinner', 'larry', 'benjamin.peterson', 'ned.deily', 'Arfrever', 'njs', 'Mark.Shannon', 'python-dev', 'eltoder', 'eric.snow', 'serhiy.storchaka', 'erlendaasland']
pr_nums = ['25714']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24912'
versions = ['Python 3.5', 'Python 3.6']Metadata
Metadata
Assignees
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)release-blockertype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Activity
serhiy-storchaka commentedon Aug 21, 2015
The type of non-heap types can be changed in 3.5. This means that the type of cached immutable objects such as small ints, empty or 1-character strings, etc can be changed.
pitrou commentedon Aug 21, 2015
Isn't there already an issue open for this?
serhiy-storchaka commentedon Aug 21, 2015
I don't remember. I noticed this once in comments on Rietveld, but don't remember the issue.
markshannon commentedon Aug 23, 2015
If there is another issue for this, then it doesn't seem to be a release blocker. I think it should be.
larryhastings commentedon Aug 26, 2015
As Python 3.5 Release Manager, my official statement is: Eek!
markshannon commentedon Aug 26, 2015
Eek! indeed :)
I intend to track down the cause of this on Monday, and hopefully come up with a fix, unless anyone beats me to it.
Does anyone know if there is another issue for this?
larryhastings commentedon Aug 26, 2015
This exciting new feature was added in checkin c0d25de5919e addressing issue bpo-22986. Perhaps the core devs who added it would like to chime in.
njsmith commentedon Aug 29, 2015
Well, yeah, that indeed sucks.
Not sure what the best solution is. Some options:
"Don't do that then"
Explicitly add a "__class__" property to every immutable type, that unconditionally errors out on assignment.
Add a hack to typeobject.c checking for the important immutable types
Something cleverer...? A new type flag?
The immutable types are: int, float, str, tuple, bool, frozenset, complex, bytes, and... anything else?
serhiy-storchaka commentedon Aug 30, 2015
Later I had got a crash.
Current thread 0xb7583700 (most recent call first):
Aborted (core dumped)
The stdlib is full of implicit caches. Virtually any hashable object can be cached and shared. Why __class__ assignment is allowed at all? There are only two uses of __class__ assignment in the stdlib besides tests (in Lib/importlib/util.py and in Lib/xml/sax/saxutils.py), and in both cases it looks as optimization trick.
benjaminp commentedon Aug 30, 2015
Probably the patch on that bug should be reverted.
njsmith commentedon Aug 30, 2015
Python goes to great lengths to make __class__ assignment work in general (I was surprised too). Historically the exception has been that instances of statically allocated C extension type objects could not have their __class__ reassigned. Semantically, this is totally arbitrary, and Guido even suggested that fixing this would be a good idea in this thread:
https://mail.python.org/pipermail/python-dev/2014-December/137430.html
The actual rule that's causing problems here is that *immutable* objects should not allow __class__ reassignment. (And specifically, objects which are assumed to be immutable by the interpreter. Most semantically immutable types in Python are defined by users and their immutability falls under the "consenting adults" rule; it's not enforced by the interpreter. Also this is very different from "hashable" -- even immutability isn't a requirement for being hashable, e.g., all user-defined types are mutable and hashable by default!)
By accident this category of "immutable-by-interpreter-invariant" has tended to be a subset of "defined in C using the old class definition API", so fixing the one issue uncovered the other.
This goal that motivated this patch was getting __class__ assignment to work on modules, which are mutable and uncacheable and totally safe. With this patch it becomes possible to e.g. issue deprecation warnings on module attribute access, which lets us finally deprecate some horrible global constants in numpy that have been confusing users for a decade now:
http://mail.scipy.org/pipermail/numpy-discussion/2015-July/073205.html
https://pypi.python.org/pypi/metamodule
I'd *really* prefer not to revert this totally, given the above. (Also, if you read that python-dev message above, you'll notice that the patch also caught and fixed a different really obscure interpreter-crashing bug.)
I think the Correct solution would be to disallow __class__ assignment specifically for the classes where it violates invariants.
If this is considered to be release critical -- and I'm not entirely sure it is, given that similar tricks have long been possible and are even referenced in the official docs?
https://www.reddit.com/r/Python/comments/2441cv/can_you_change_the_value_of_1/ch3dwxt
https://docs.python.org/3/c-api/long.html#c.PyLong_FromLong
-- but if it is considered to be release critical, and it's considered too short notice to accomplish a proper fix, then a simple hack would be to add something like
to typeobject.c:object_set_class. As compared to reverting the whole patch, this would preserve the most important case, which is one that we know is safe, and we could then progressively relax the check further in the future...
markshannon commentedon Aug 30, 2015
Please revert c0d25de5919e.
Breaking the interpreter in order to facilitate some obscure use case is unacceptable.
If you want to do fancy stuff with modules, fine. Take a look at the source of the six module
https://bitbucket.org/gutworth/six/src/cd1e81d33eaf3d6944f859c2aa7d5d3f515013c8/six.py?at=default for some tips.
I think immutability is a fundamental property of an object. The "consenting" adults ideas is fine for accessibility. However, making previously immutable object mutable forces developers to use lots of defensive copying and causes obscure bugs (like this one).
I do not regard the immutable of numbers as an historical accident, but as vitally important for any sort of numerical reasoning.
Just take a look at a language with mutable strings to see the problems that causes.
49 remaining items
markshannon commentedon Sep 5, 2015
Why has the change allowing the __class__ attribute of modules been merged?
It is not necessary (as I have stated repeatedly) and not known to be safe.
larryhastings commentedon Sep 5, 2015
Because the BDFL asked that it be so.
markshannon commentedon Sep 5, 2015
Well, there are important reasons why not to make the __class__ attribute of modules mutable.
First of all, there is no valid rationale.
How do know for sure that modifying the class of the sys or builtins module is not going to cause much the same problems the changing the class of ints did?
markshannon commentedon Sep 5, 2015
Oh, and has anyone considered the potential impact this might have on Jython or PyPy? Modules are quite fundamental to the way that Python works.
python-dev commentedon Sep 5, 2015
New changeset 27cc5cce0292 by Guido van Rossum in branch '3.5':
Issue bpo-24912: Prevent __class__ assignment to immutable built-in objects.
https://hg.python.org/cpython/rev/27cc5cce0292
New changeset 1c55f169f4ee by Guido van Rossum in branch '3.5':
Issue bpo-24912: Prevent __class__ assignment to immutable built-in objects. (Merge 3.5.0 -> 3.5)
https://hg.python.org/cpython/rev/1c55f169f4ee
New changeset b045465e5dba by Guido van Rossum in branch 'default':
Issue bpo-24912: Prevent __class__ assignment to immutable built-in objects. (Merge 3.5 -> 3.6)
https://hg.python.org/cpython/rev/b045465e5dba
gvanrossum commentedon Sep 5, 2015
Mark, please calm down. Modules are incredibly simple objects compared to classes -- and yet you can change the __class__ of a class. You can also directly modify the __dict__ of a module. You can shoot yourself in the foot phenomenally this way, but that's not the point. The point is not to violate *internal* constraints of the interpreter. Changing immutable objects was violating such a constraint. Changing a module's __class__ is not.
markshannon commentedon Sep 5, 2015
Guido,
I just think this change is misguided. The original rationale for the change just doesn't hold water.
If you want the change anyway, then fine, but this seems an odd way to introduce it.
njsmith commentedon Sep 5, 2015
I didn't want to get further into the weeds of debating basic design points on the tracker while blocking rc3, but, for the record -- the proposal in msg249504 that one could keep the namespace of an old and new module object effectively in sync by defining __getattr__ on the new module doesn't work AFAICT. (I assume this is the basis of Mark's assertion that the change is unmotivated.) Going through __getattr__ creates an unacceptable performance hit on attribute access. You can avoid this by copying the namespace entries over (b/c __getattr__ is only called as a fallback when __dict__ lookup fails), but then you have the same problem as before of needing to keep things in sync. (And remember that module globals can change at any time -- the __globals__ pointer for any functions defined in that module will continue to point to the old namespace.)
Re: PyPy: last I heard their main complaint about __class__ assignment was that it was hard to restrict it to *only* handle the cases that CPython does. (They do have a concept of "heap types"; AFAICT it's just a flag that says "the CPython version of this type has some quirks that we need to check for and emulate".) Not sure about Jython, but I doubt they have a natural heap/non-heap type distinction either.
Thanks Guido for handling the patches!
ericsnowcurrently commentedon Sep 6, 2015
While I recognize the practicality/purity argument here, I somewhat agree with Mark. Assigning to module.__class__ makes sense for the use case, but it does open up a number of negative possible side effects (e.g. changing sys.__class__). Ideally it would be restricted to just the currently running module, but doing that through __class__ assignment is not a great idea. Having a dedicated builtin function or even syntax would make more sense.
In some ways I'm reminded of __metaclass__ in Python 2 class definitions. Something like that has come up before but didn't get a lot of momentum at the time. Ultimately I'd like to see a more explicit mechanism to meet the need that motivated the __class__ change here (as well as the replace-this-module-in-sys-modules technique). [1] While you can already accomplish this via a custom finder, that's not the most convenient tool for this sort of situation.
FWIW, if assigning to sys.modules[name].__class__ is the blessed way then we should make it official in the language reference. The same goes for modules replacing themselves in sys.modules. If we have reservations about making either official (which I for one do [2]) then we should work on a more appropriate official solution.
[1] It all boils down to customizing the current module's type from within the module rather than from outside (using a custom loader).
[2] Neither spelling is very friendly.
larryhastings commentedon Sep 6, 2015
Marking as closed for now. If we decide it's a problem we can reopen later.
vstinner commentedon Apr 30, 2021
Update: a new Py_TPFLAGS_IMMUTABLETYPE flag was added in bpo-43908, and it's now checked rather than relying on Py_TPFLAGS_HEAPTYPE to decide if a type is "mutable" or not.