Skip to content

invalid exception context #47861

Closed
Closed

Description

@vstinner
BPO 3611
Nosy @amauryfa, @pitrou, @vstinner, @benjaminp
Files
  • lostcontext.py
  • 3611.patch
  • lostcontext2.py
  • pyerr_setobject_reentrant.patch: Fix PyErr_SetObject() to be re-entrant
  • pyerr_setobject_reentrant-v2.patch: New version of my previous patch: fix PyErr_SetObject() to be re-entrant
  • pyerr_seterror_protect.py
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2008-08-29.07:14:24.613>
    created_at = <Date 2008-08-20.02:33:18.833>
    labels = ['interpreter-core', 'type-crash', 'release-blocker']
    title = 'invalid exception context'
    updated_at = <Date 2008-08-29.07:14:24.595>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2008-08-29.07:14:24.595>
    actor = 'amaury.forgeotdarc'
    assignee = 'none'
    closed = True
    closed_date = <Date 2008-08-29.07:14:24.613>
    closer = 'amaury.forgeotdarc'
    components = ['Interpreter Core']
    creation = <Date 2008-08-20.02:33:18.833>
    creator = 'vstinner'
    dependencies = []
    files = ['11180', '11182', '11196', '11201', '11203', '11207']
    hgrepos = []
    issue_num = 3611
    keywords = ['patch', 'needs review']
    message_count = 41.0
    messages = ['71503', '71505', '71506', '71522', '71548', '71551', '71552', '71553', '71557', '71558', '71569', '71570', '71574', '71575', '71579', '71587', '71589', '71591', '71592', '71593', '71594', '71595', '71596', '71657', '71658', '71659', '71661', '71663', '71664', '71665', '71698', '71704', '71711', '71739', '71740', '71741', '71793', '71794', '72041', '72043', '72120']
    nosy_count = 4.0
    nosy_names = ['amaury.forgeotdarc', 'pitrou', 'vstinner', 'benjamin.peterson']
    pr_nums = []
    priority = 'release blocker'
    resolution = 'fixed'
    stage = None
    status = 'closed'
    superseder = None
    type = 'crash'
    url = 'https://bugs.python.org/issue3611'
    versions = ['Python 3.0']

    Activity

    1. vstinner commented on Aug 20, 2008

      @vstinner
      MemberAuthor

      After few seconds (30 sec to 5 minutes), my program (Fusil) crashs at:

      PyEval_EvalFrameEx (f=0x85b4324, throwflag=0) at Python/ceval.c:2459
          Py_CLEAR(tstate->exc_traceback);

      It crashs because tstate->exc_traceback points to a zombi object:
      {_ob_next = 0xdbdbdbdb, _ob_prev = 0xdbdbdbdb, ob_refcnt = -606348326,
      ob_type = 0xdbdbdbdb}

      (refcnt is 0xdbdbdbdb-1)

      I'm using py3k rev 65882 compiled with CFLAGS "-O0 -ggdb"
      and --with-pydebug. Sorry, I don't have more informations yet and I
      can't explain how to reproduce the bug :-/

    2. added
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      type-crashA hard crash of the interpreter, possibly with a core dump
      on Aug 20, 2008
    3. vstinner commented on Aug 20, 2008

      @vstinner
      MemberAuthor

      The crash is on ProcError exception raising (line 21):

      20      except IOError as err:
      21          raise ProcError("Unable to open %r: %s" % (filename, 
      

      ---

      (gdb) where
      #0 0x0805bcc7 in PyObject_Hash (v=0x8691034) at Objects/object.c:796
      #1 0x08062125 in set_contains_key (so=0x856ec6c, key=0x8691034) at
      Objects/setobject.c:682
      #2 0x0806659f in PySet_Contains (anyset=0x856ec6c, key=0x8691034) at
      Objects/setobject.c:2263
      #3 0x080def3a in print_exception_recursive (f=0xb7bea654,
      value=0x8656814, seen=0x856ec6c) at Python/pythonrun.c:1432
      #4 0x080df0ea in PyErr_Display (exception=0x84542ac, value=0x8656814,
      tb=0x865bc34) at Python/pythonrun.c:1470
      #5 0x080e68f6 in sys_excepthook (self=0xb7e013b4, args=0x865b334) at
      Python/sysmodule.c:119
      #6 0x08161d29 in PyCFunction_Call (func=0xb7e016c4, arg=0x865b334,
      kw=0x0) at Objects/methodobject.c:81
      #7 0x08118cc5 in PyObject_Call (func=0xb7e016c4, arg=0x865b334,
      kw=0x0) at Objects/abstract.c:2181
      #8 0x080b2ea8 in PyEval_CallObjectWithKeywords (func=0xb7e016c4,
      arg=0x865b334, kw=0x0) at Python/ceval.c:3283
      #9 0x080de463 in PyErr_PrintEx (set_sys_last_vars=1) at
      Python/pythonrun.c:1258
      #10 0x080de06f in PyErr_Print () at Python/pythonrun.c:1150
      #11 0x080e0007 in Py_FatalError (msg=0xbf9a1498 "Python/ceval.c:2459
      object at 0x865b6b4 has negative ref count -606348326")
      at Python/pythonrun.c:1863
      #12 0x0805a821 in _Py_NegativeRefcount
      (fname=0x8197098 "Python/ceval.c", lineno=2459, op=0x865b6b4) at
      Objects/object.c:194
      #13 0x080b004f in PyEval_EvalFrameEx (f=0x85b4324, throwflag=0) at
      Python/ceval.c:2459
      (...)

      (gdb) frame 13
      #13 0x080b004f in PyEval_EvalFrameEx (f=0x85c9794, throwflag=0) at 
      Python/ceval.c:2459
      2459                                            
      Py_CLEAR(tstate->exc_traceback);

      (gdb) pystack
      /home/haypo/ptrace/ptrace/linux_proc.py (21): openProc
      /home/haypo/ptrace/ptrace/linux_proc.py (24): readProc
      /home/haypo/ptrace/ptrace/linux_proc.py (31): readProcessProc
      /home/haypo/ptrace/ptrace/linux_proc.py (74): readProcessStat
      (...)

      ---

      linux_proc.py code near line 21:
      (...)
      7 class ProcError(Exception):
      8 pass
      (...)
      16 def openProc(path):
      17 try:
      18 filename = procFilename(path)
      19 return open(filename)
      20 except IOError as err:
      21 raise ProcError("Unable to open %r: %s" % (filename,
      err))
      (...)
      29 def readProcessProc(pid, key):
      30 try:
      31 return readProc(path_join(str(pid), str(key)))
      32 except ProcError as error:
      33 raise ProcError("Process %s doesn't exist: %s" % (
      34 pid, error))
      (...)
      73 def readProcessStat(pid):
      74 stat = readProcessProc(pid, 'stat')
      75 return ProcessState(stat)
      76

    4. vstinner commented on Aug 20, 2008

      @vstinner
      MemberAuthor

      I also noticed a crash in PyErr_SetObject(): in block <line 68..86>,
      tstate->exc_value value may changes and so it's possible that
      tstate->exc_value becomes NULL. I added a test to avoid this crash:

      Index: Python/errors.c
      ===================================================================

      --- Python/errors.c     (révision 65899)
      +++ Python/errors.c     (copie de travail)
      @@ -88,7 +88,7 @@
                         This is O(chain length) but context chains are
                         usually very short. Sensitive readers may try
                         to inline the call to PyException_GetContext. */
      -               if (tstate->exc_value != value) {
      +               if (tstate->exc_value != value && tstate->exc_value != 
      NULL) {
                              PyObject *o = tstate->exc_value, *context;
                              while ((context = PyException_GetContext(o))) 
      {
                                      Py_DECREF(context);
    5. pitrou commented on Aug 20, 2008

      @pitrou
      Member

      I also noticed a crash in PyErr_SetObject(): in block <line 68..86>,
      tstate->exc_value value may changes and so it's possible that
      tstate->exc_value becomes NULL.

      How can tstate->exc_value become NULL at that point? Could you investigate?

    6. vstinner commented on Aug 20, 2008

      @vstinner
      MemberAuthor

      How can tstate->exc_value become NULL at that point? [error.c:86]
      Could you investigate?

      PyEval_CallObject(exception, args) may calls PyErr_SetObject(). Since
      the same thread state is the same, tstate->exc_value also changes
      during this call.

    7. pitrou commented on Aug 20, 2008

      @pitrou
      Member

      Selon STINNER Victor <report@bugs.python.org>:

      > How can tstate->exc_value become NULL at that point? [error.c:86]
      > Could you investigate?

      PyEval_CallObject(exception, args) may calls PyErr_SetObject(). Since
      the same thread state is the same, tstate->exc_value also changes
      during this call.

      Hmm, indeed.
      However, I don't see why your Python code triggers that. Does instantiating the
      ProcError object somehow fail, and for what reason? Or did you witness it in
      other circumstances?

      As for the original problem ("tstate->exc_traceback points to a zombi object")
      it would be nice to have a small snippet of Python code to reproduce it. Is it
      possible for you to do that?

    8. pitrou commented on Aug 20, 2008

      @pitrou
      Member

      I retract what I said above. PyErr_SetObject() and friends only change
      tstate->curexc_* variables, not tstate->exc_*.

      The former (tstate->curexc_) contain the current, pending (uncaught)
      exception state of the thread. The latter (tstate->exc_
      ) contain the
      exception currently handled by an "except" statement in Python code.

      So, theoretically, there is no way calling PyEval_CallObject() can
      change tstate->exc_value:

      • if it raises an exception or calls code that raises an exception
        without catching it, the exception ends up in tstate->curexc_, not in
        tstate->exc_
      • if it calls code that raises an exception /and catches it/, exception
        stacking means the exception should be discarded after the end of the
        "except" block and the last tstate->exc_* values restored.

      Of course, there may be bugs in the implementation of the above. The
      best way to find out would be to have a small snippet of Python code
      reproducing the problem. :-)

    9. vstinner commented on Aug 20, 2008

      @vstinner
      MemberAuthor

      About the PyEval_CallObject() call in errors.c, here is an example:
      Call exception<0x81dcee0>(args<0x8751dc4>)
      with exception=
      object : <class 'AttributeError'>
      type : type
      refcount: 6
      address : 0x81dcee0
      and args=
      object : ("type object 'ProcError' has no
      attribute '__subclasscheck__'",)
      type : tuple
      refcount: 1
      address : 0x8751dc4

      This exception may comes from PyObject_IsSubclass() which calls
      PyObject_GetAttr(cls, "__subclasscheck__").

    10. pitrou commented on Aug 20, 2008

      @pitrou
      Member

      [...]

      This exception may comes from PyObject_IsSubclass() which calls
      PyObject_GetAttr(cls, "__subclasscheck__").

      If you look at the code for PyObject_IsSubclass(), there is a pair of
      PyErr_Fetch / PyErr_Restore calls around PyObject_GetAttr(cls,
      "__subclasscheck__"). So this exception should vanish immediately rather than
      stay around.

    11. vstinner commented on Aug 20, 2008

      @vstinner
      MemberAuthor

      I'm unable to write a short Python script to reproduce the bug. So
      download the full program:
      http://neudorf.hachoir.org/tmp/fusil3000.tar.gz

      Run in with:
      $ gdb python3.0
      (gdb) run fusil-python --fast

    12. amauryfa commented on Aug 20, 2008

      @amauryfa
      Contributor

      The problem comes when PyErr_SetObject triggers garbage collection which
      runs python code (finalizers...).

      I could reproduce the crash several times, and each time garbage
      collection was triggered by the normalization of the exception (the call
      to PyEval_CallObject(exception, args)).

      An obvious fix is to save exc_value near the "/* Implicit exception
      chaining */" comment.

      However, I fear that there are other places where exc_value is reset,
      and "exception chaining" may break in subtle ways.

    13. pitrou commented on Aug 20, 2008

      @pitrou
      Member

      The problem comes when PyErr_SetObject triggers garbage collection which
      runs python code (finalizers...).

      Mmmh, normally this shouldn't change the value of tstate->exc_value once
      that Python code returns. That's what exception stacking is for.

      Having a snippet deterministically reproducing the problem would really
      help in any case.

      An obvious fix is to save exc_value near the "/* Implicit exception
      chaining */" comment.

      Well, it may be a fix for the crash but I'm not sure it makes the
      semantics correct. If tstate->exc_value was really changed, there is a
      reason and I'm not sure it should be ignored.

    14. amauryfa commented on Aug 20, 2008

      @amauryfa
      Contributor

      Having a snippet deterministically reproducing the problem
      would really help in any case.
      Here you are. Don't ask how I found this.

      The attached script, when run, prints

      (KeyError(), ValueError())
      (KeyError(), None)

      The current exception context (tstate->exc_value) is lost when a
      generator is deleted, if it was paused inside a "try" block.

      This is the cause of the crash: the gc runs inside PyErr_SetObject and
      collects such a generator.

    15. 18 remaining items

    16. amauryfa commented on Aug 21, 2008

      @amauryfa
      Contributor

      Antoine, your script hangs at the end due to the io.py deadlock (see bpo-3618)

      • at the end of the script, flush_io() is called
      • this enters code in io.py
      • here, garbage collection occurs (thanks to low thresholds)
      • the Bug() instance is collected
      • the exception is handled by a PyErr_WriteUnraisable
      • which tries to print
    17. amauryfa commented on Aug 21, 2008

      @amauryfa
      Contributor

      I found another way to loose the current exception context; see
      lostcontext2.py

      Completely removing the block starting with
             if (why == WHY_EXCEPTION && !throwflag) {
      corrects the script;

      but according to Benjamin: """removing those lines causes a RuntimeError
      about exceeding the recursion limit for about 1/3 of the tests."""

      So a better fix must be found. At least we have the unit test :-).

    18. vstinner commented on Aug 21, 2008

      @vstinner
      MemberAuthor

      It's now known that PyErr_SetObject() have to be re-entrant because of
      the garbage collector interaction. As I wrote in my comments, tstate
      may be changed during PyEval_CallObject() call. The problem is to
      known which values have to be protected during PyErr_SetObject()
      re-entrant call... I tried to save/restore tstate->exc_value if it's
      value changes, but I'm not sure that it's enough.

    19. vstinner commented on Aug 21, 2008

      @vstinner
      MemberAuthor

      Ok, it was not enough: exc_{type,value,traceback} have to be
      saved/restored. So here is a new patch. I doesn't change Python
      behaviour for previous pitrou snippet (msg71658): it doesn't crash
      here and display two errors.

      I think that this issue mix differents bugs:
      (1) lostcontext.py fixed by « if (why == WHY_EXCEPTION
      && !throwflag) »
      (2) pitrou snippet
      (3) PyErr_SetObject() is no re-entrant: not fixed yet (please review
      my new patch attached to this comment ;-))

    20. vstinner commented on Aug 22, 2008

      @vstinner
      MemberAuthor

      With my last patch (pyerr_setobject_reentrant-v2.patch) my program
      works fine! I known that PyErr_SetObject() makes re-entrant calls
      every ~30 seconds, so the code is tested ;-)

    21. amauryfa commented on Aug 22, 2008

      @amauryfa
      Contributor

      Victor, your patch addresses the symptom of the problem, not the cause.

      The cause is that in some cases, the exception context chain is lost. Of
      course what could be a minor annoyance becomes dramatic when this
      precisely happens in code that makes use of this context.

      IMO, PyErr_SetObject should not restore the previous exc_* members; it's
      not its job to correct others' mistakes.
      I join a patch that only protects it from crashing in this case: it just
      saves exc_value in a local variable.

      The bug shown by lostcontext2.py is still unresolved, but at least it
      won't crash the interpreter.

    22. pitrou commented on Aug 22, 2008

      @pitrou
      Member

      Agreed with Amaury, it's not PyErr_SetObject's job to try to
      save/restore the tstate->exc_* variables. We'll probably have to live
      with the small context-losing glitches in 3.0.

      For 3.1, a radical solution would be to drop the "exception
      normalization" misfeature (which is a source of complications and
      potential bugs) and always instantiate exception objects as soon as they
      are raised rather than lazily. We need to make sure the performance loss
      is reasonable though.

    23. pitrou commented on Aug 22, 2008

      @pitrou
      Member

      Hmm, answering to myself, I don't think dropping exception normalization
      would solve the problem, it would just let it occur in a different place...

    24. amauryfa commented on Aug 22, 2008

      @amauryfa
      Contributor

      Benjamin,

      I am testing the patch proposed earlier by Victor in
      http://bugs.python.org/msg71579 (who might be the correct one, after all),
      and I don't see the problems you mentioned.
      I run the test suite without problem, on winXP & Linux Debian x86.

      Can you please try again?

    25. benjaminp commented on Aug 22, 2008

      @benjaminp
      Contributor

      I can't reproduce it either. I suspect that I removed the wrong lines
      initially and caused other problems. :) Sorry for the false alarm.

    26. amauryfa commented on Aug 27, 2008

      @amauryfa
      Contributor

      Some progress:
      The lines suppressed by the patch at http://bugs.python.org/msg71579
      either do nothing (because e.g exc_type is already NULL or None),
      or happen to be in a case similar to the script "lostcontext2.py"
      (Most of the time, the flush() function in io.py).

      So again, I think these lines should be suppressed.
      The test suite still pass, and all the "-R::" I ran did return identical
      result.
      Can this patch go in?

    27. pitrou commented on Aug 27, 2008

      @pitrou
      Member

      Can this patch go in?

      I'm ok for it.

    28. amauryfa commented on Aug 29, 2008

      @amauryfa
      Contributor

      Committed as r66056.

    29. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      interpreter-core(Objects, Python, Grammar, and Parser dirs)release-blockertype-crashA hard crash of the interpreter, possibly with a core dump

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions