Skip to content

The type of cached objects is mutable #69100

Closed

Description

@serhiy-storchaka
BPO 24912
Nosy @malemburg, @gvanrossum, @brettcannon, @rhettinger, @ncoghlan, @pitrou, @vstinner, @larryhastings, @benjaminp, @ned-deily, @njsmith, @markshannon, @eltoder, @ericsnowcurrently, @serhiy-storchaka, @erlend-aasland
PRs
  • bpo-43973: Use Py_TPFLAGS_IMMUTABLETYPE to check for class assignments #25714
  • Files
  • issue24912.patch
  • issue24912-v2.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/larryhastings'
    closed_at = <Date 2015-09-06.07:53:19.607>
    created_at = <Date 2015-08-21.22:02:50.644>
    labels = ['interpreter-core', 'type-bug', 'release-blocker']
    title = 'The type of cached objects is mutable'
    updated_at = <Date 2021-04-30.11:27:56.656>
    user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2021-04-30.11:27:56.656>
    actor = 'erlendaasland'
    assignee = 'larry'
    closed = True
    closed_date = <Date 2015-09-06.07:53:19.607>
    closer = 'larry'
    components = ['Interpreter Core']
    creation = <Date 2015-08-21.22:02:50.644>
    creator = 'serhiy.storchaka'
    dependencies = []
    files = ['40310', '40312']
    hgrepos = []
    issue_num = 24912
    keywords = ['patch']
    message_count = 69.0
    messages = ['248981', '248982', '248983', '249019', '249180', '249189', '249198', '249345', '249350', '249351', '249354', '249357', '249360', '249363', '249369', '249388', '249392', '249393', '249396', '249397', '249398', '249438', '249446', '249450', '249455', '249456', '249457', '249467', '249468', '249470', '249471', '249473', '249483', '249484', '249489', '249495', '249499', '249500', '249502', '249504', '249505', '249506', '249509', '249510', '249511', '249604', '249607', '249618', '249623', '249685', '249854', '249869', '249871', '249888', '249889', '249890', '249891', '249893', '249930', '249931', '249934', '249938', '249940', '249943', '249944', '249948', '249954', '249979', '392411']
    nosy_count = 18.0
    nosy_names = ['lemburg', 'gvanrossum', 'brett.cannon', 'rhettinger', 'ncoghlan', 'pitrou', 'vstinner', 'larry', 'benjamin.peterson', 'ned.deily', 'Arfrever', 'njs', 'Mark.Shannon', 'python-dev', 'eltoder', 'eric.snow', 'serhiy.storchaka', 'erlendaasland']
    pr_nums = ['25714']
    priority = 'release blocker'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue24912'
    versions = ['Python 3.5', 'Python 3.6']

    Activity

    1. serhiy-storchaka commented on Aug 21, 2015

      @serhiy-storchaka
      MemberAuthor

      The type of non-heap types can be changed in 3.5. This means that the type of cached immutable objects such as small ints, empty or 1-character strings, etc can be changed.

      >>> class I(int):
      ...     __slots__ = ()
      ...     def __repr__(self):
      ...         return 'Answer to The Ultimate Question of Life, the Universe, and Everything'
      ...     def __add__(self, other):
      ...         return self * other
      ... 
      >>> (42).__class__ = I
      >>> ord('*')
      Answer to The Ultimate Question of Life, the Universe, and Everything
      >>> x = 42; x + 2
      84
      >>> class S(str):
      ...     __slots__ = ()
      ...     def __len__(self):
      ...         return 123
      ... 
      >>> 'a'.__class__ = S
      >>> i = 1; len('abc'[:i])
      123
    2. pitrou commented on Aug 21, 2015

      @pitrou
      Member

      Isn't there already an issue open for this?

    3. serhiy-storchaka commented on Aug 21, 2015

      @serhiy-storchaka
      MemberAuthor

      I don't remember. I noticed this once in comments on Rietveld, but don't remember the issue.

    4. markshannon commented on Aug 23, 2015

      @markshannon
      Member

      If there is another issue for this, then it doesn't seem to be a release blocker. I think it should be.

    5. larryhastings commented on Aug 26, 2015

      @larryhastings
      Contributor

      As Python 3.5 Release Manager, my official statement is: Eek!

    6. markshannon commented on Aug 26, 2015

      @markshannon
      Member

      Eek! indeed :)

      I intend to track down the cause of this on Monday, and hopefully come up with a fix, unless anyone beats me to it.

      Does anyone know if there is another issue for this?

    7. larryhastings commented on Aug 26, 2015

      @larryhastings
      Contributor

      This exciting new feature was added in checkin c0d25de5919e addressing issue bpo-22986. Perhaps the core devs who added it would like to chime in.

    8. njsmith commented on Aug 29, 2015

      @njsmith
      Contributor

      Well, yeah, that indeed sucks.

      Not sure what the best solution is. Some options:

      1. "Don't do that then"

      2. Explicitly add a "__class__" property to every immutable type, that unconditionally errors out on assignment.

      3. Add a hack to typeobject.c checking for the important immutable types

      4. Something cleverer...? A new type flag?

      The immutable types are: int, float, str, tuple, bool, frozenset, complex, bytes, and... anything else?

    9. serhiy-storchaka commented on Aug 30, 2015

      @serhiy-storchaka
      MemberAuthor

      Later I had got a crash.

      >>> class S(str): __slots__ = ()
      ... 
      >>> 'a'.__class__ = S
      >>> 
      >>> def f(a): pass
      ... 
      Fatal Python error: non-string found in code slot

      Current thread 0xb7583700 (most recent call first):
      Aborted (core dumped)

      The stdlib is full of implicit caches. Virtually any hashable object can be cached and shared. Why __class__ assignment is allowed at all? There are only two uses of __class__ assignment in the stdlib besides tests (in Lib/importlib/util.py and in Lib/xml/sax/saxutils.py), and in both cases it looks as optimization trick.

    10. benjaminp commented on Aug 30, 2015

      @benjaminp
      Contributor

      Probably the patch on that bug should be reverted.

    11. njsmith commented on Aug 30, 2015

      @njsmith
      Contributor

      Python goes to great lengths to make __class__ assignment work in general (I was surprised too). Historically the exception has been that instances of statically allocated C extension type objects could not have their __class__ reassigned. Semantically, this is totally arbitrary, and Guido even suggested that fixing this would be a good idea in this thread:
      https://mail.python.org/pipermail/python-dev/2014-December/137430.html
      The actual rule that's causing problems here is that *immutable* objects should not allow __class__ reassignment. (And specifically, objects which are assumed to be immutable by the interpreter. Most semantically immutable types in Python are defined by users and their immutability falls under the "consenting adults" rule; it's not enforced by the interpreter. Also this is very different from "hashable" -- even immutability isn't a requirement for being hashable, e.g., all user-defined types are mutable and hashable by default!)

      By accident this category of "immutable-by-interpreter-invariant" has tended to be a subset of "defined in C using the old class definition API", so fixing the one issue uncovered the other.

      This goal that motivated this patch was getting __class__ assignment to work on modules, which are mutable and uncacheable and totally safe. With this patch it becomes possible to e.g. issue deprecation warnings on module attribute access, which lets us finally deprecate some horrible global constants in numpy that have been confusing users for a decade now:
      http://mail.scipy.org/pipermail/numpy-discussion/2015-July/073205.html
      https://pypi.python.org/pypi/metamodule

      I'd *really* prefer not to revert this totally, given the above. (Also, if you read that python-dev message above, you'll notice that the patch also caught and fixed a different really obscure interpreter-crashing bug.)

      I think the Correct solution would be to disallow __class__ assignment specifically for the classes where it violates invariants.

      If this is considered to be release critical -- and I'm not entirely sure it is, given that similar tricks have long been possible and are even referenced in the official docs?
      https://www.reddit.com/r/Python/comments/2441cv/can_you_change_the_value_of_1/ch3dwxt
      https://docs.python.org/3/c-api/long.html#c.PyLong_FromLong
      -- but if it is considered to be release critical, and it's considered too short notice to accomplish a proper fix, then a simple hack would be to add something like

      if (!(oldto->tp_flags & Py_TPFLAGS_HEAPTYPE) && oldto != &PyModule_Type) {
          PyErr_Format(PyExc_TypeError, ...);
          return -1;
      }

      to typeobject.c:object_set_class. As compared to reverting the whole patch, this would preserve the most important case, which is one that we know is safe, and we could then progressively relax the check further in the future...

    12. markshannon commented on Aug 30, 2015

      @markshannon
      Member

      Please revert c0d25de5919e.
      Breaking the interpreter in order to facilitate some obscure use case is unacceptable.

      If you want to do fancy stuff with modules, fine. Take a look at the source of the six module
      https://bitbucket.org/gutworth/six/src/cd1e81d33eaf3d6944f859c2aa7d5d3f515013c8/six.py?at=default for some tips.

      I think immutability is a fundamental property of an object. The "consenting" adults ideas is fine for accessibility. However, making previously immutable object mutable forces developers to use lots of defensive copying and causes obscure bugs (like this one).

      I do not regard the immutable of numbers as an historical accident, but as vitally important for any sort of numerical reasoning.
      Just take a look at a language with mutable strings to see the problems that causes.

    13. 49 remaining items

    14. markshannon commented on Sep 5, 2015

      @markshannon
      Member

      Why has the change allowing the __class__ attribute of modules been merged?

      It is not necessary (as I have stated repeatedly) and not known to be safe.

    15. larryhastings commented on Sep 5, 2015

      @larryhastings
      Contributor

      Because the BDFL asked that it be so.

    16. markshannon commented on Sep 5, 2015

      @markshannon
      Member

      Well, there are important reasons why not to make the __class__ attribute of modules mutable.

      First of all, there is no valid rationale.

      How do know for sure that modifying the class of the sys or builtins module is not going to cause much the same problems the changing the class of ints did?

    17. markshannon commented on Sep 5, 2015

      @markshannon
      Member

      Oh, and has anyone considered the potential impact this might have on Jython or PyPy? Modules are quite fundamental to the way that Python works.

    18. python-dev commented on Sep 5, 2015

      python-devmannequin
      Mannequin

      New changeset 27cc5cce0292 by Guido van Rossum in branch '3.5':
      Issue bpo-24912: Prevent __class__ assignment to immutable built-in objects.
      https://hg.python.org/cpython/rev/27cc5cce0292

      New changeset 1c55f169f4ee by Guido van Rossum in branch '3.5':
      Issue bpo-24912: Prevent __class__ assignment to immutable built-in objects. (Merge 3.5.0 -> 3.5)
      https://hg.python.org/cpython/rev/1c55f169f4ee

      New changeset b045465e5dba by Guido van Rossum in branch 'default':
      Issue bpo-24912: Prevent __class__ assignment to immutable built-in objects. (Merge 3.5 -> 3.6)
      https://hg.python.org/cpython/rev/b045465e5dba

    19. gvanrossum commented on Sep 5, 2015

      @gvanrossum
      Member

      Mark, please calm down. Modules are incredibly simple objects compared to classes -- and yet you can change the __class__ of a class. You can also directly modify the __dict__ of a module. You can shoot yourself in the foot phenomenally this way, but that's not the point. The point is not to violate *internal* constraints of the interpreter. Changing immutable objects was violating such a constraint. Changing a module's __class__ is not.

    20. markshannon commented on Sep 5, 2015

      @markshannon
      Member

      Guido,
      I just think this change is misguided. The original rationale for the change just doesn't hold water.

      If you want the change anyway, then fine, but this seems an odd way to introduce it.

    21. njsmith commented on Sep 5, 2015

      @njsmith
      Contributor

      I didn't want to get further into the weeds of debating basic design points on the tracker while blocking rc3, but, for the record -- the proposal in msg249504 that one could keep the namespace of an old and new module object effectively in sync by defining __getattr__ on the new module doesn't work AFAICT. (I assume this is the basis of Mark's assertion that the change is unmotivated.) Going through __getattr__ creates an unacceptable performance hit on attribute access. You can avoid this by copying the namespace entries over (b/c __getattr__ is only called as a fallback when __dict__ lookup fails), but then you have the same problem as before of needing to keep things in sync. (And remember that module globals can change at any time -- the __globals__ pointer for any functions defined in that module will continue to point to the old namespace.)

      Re: PyPy: last I heard their main complaint about __class__ assignment was that it was hard to restrict it to *only* handle the cases that CPython does. (They do have a concept of "heap types"; AFAICT it's just a flag that says "the CPython version of this type has some quirks that we need to check for and emulate".) Not sure about Jython, but I doubt they have a natural heap/non-heap type distinction either.

      Thanks Guido for handling the patches!

    22. ericsnowcurrently commented on Sep 6, 2015

      @ericsnowcurrently
      Member

      While I recognize the practicality/purity argument here, I somewhat agree with Mark. Assigning to module.__class__ makes sense for the use case, but it does open up a number of negative possible side effects (e.g. changing sys.__class__). Ideally it would be restricted to just the currently running module, but doing that through __class__ assignment is not a great idea. Having a dedicated builtin function or even syntax would make more sense.

      In some ways I'm reminded of __metaclass__ in Python 2 class definitions. Something like that has come up before but didn't get a lot of momentum at the time. Ultimately I'd like to see a more explicit mechanism to meet the need that motivated the __class__ change here (as well as the replace-this-module-in-sys-modules technique). [1] While you can already accomplish this via a custom finder, that's not the most convenient tool for this sort of situation.

      FWIW, if assigning to sys.modules[name].__class__ is the blessed way then we should make it official in the language reference. The same goes for modules replacing themselves in sys.modules. If we have reservations about making either official (which I for one do [2]) then we should work on a more appropriate official solution.

      [1] It all boils down to customizing the current module's type from within the module rather than from outside (using a custom loader).
      [2] Neither spelling is very friendly.

    23. larryhastings commented on Sep 6, 2015

      @larryhastings
      Contributor

      Marking as closed for now. If we decide it's a problem we can reopen later.

    24. vstinner commented on Apr 30, 2021

      @vstinner
      Member

      Update: a new Py_TPFLAGS_IMMUTABLETYPE flag was added in bpo-43908, and it's now checked rather than relying on Py_TPFLAGS_HEAPTYPE to decide if a type is "mutable" or not.

    25. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    interpreter-core(Objects, Python, Grammar, and Parser dirs)release-blockertype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions