You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Fatal error on thread creation in low memory condition #51793
Using my fuzzer (Fusil) on Python trunk, I got sometimes errors on
multiprocessing.Pool():
Fatal Python error: PyEval_AcquireThread: NULL new thread state
I'm sorry but I don't have example script to reproduce the bug. I
suppose that the error depends on the system load. I have the error on
an Intel Core2 Quad Core Q9300 (CPU able to run 4 processes/threads at
the time same).
How can I get more information on the bug? How can I reproduce it?
haypo> Using my fuzzer (Fusil) on Python trunk, I got sometimes
haypo> errors on multiprocessing.Pool():
haypo>
haypo> Fatal Python error: PyEval_AcquireThread: NULL new thread state
I read the source code of the thread module. This error means that
PyThreadState_New() returns NULL which occurs if malloc() failed. I hit this
error using my fuzzer because the fuzzer limits the total memory to something
around 100 MB using setrlimit().
Said differently: in low memory condition, creating a new thread may exit the
whole Python process if a memory allocation fail.
--
Sometimes, I get another error, similar to the "NULL new thread state" error:
changed the title [-]multiprocessing.Pool(): Fatal Python error: PyEval_AcquireThread: NULL new thread state[/-][+]Fatal error on thread creation in low memory condition[/+]on Dec 23, 2009
nirai commented on Dec 27, 2009
niraimannequin
Mannequin
More actions
Memory can be pre-allocated by thread_PyThread_start_new_thread() before
thread is spawned.
Here I come with a patch! nirai idea was the good one: prealloc PyThreadState before creating the thread. Raise a MemoryError if the allocation fail, instead of raising a *fatal* Python error.
Patch is quite simple and allow better error handling.
The patch looks good; the line
tstate->thread_id = PyThread_get_thread_ident();
is needed because the tstate is created in the main thread, but used in another thread.
I'm not sure about autoTLSkey value. PyThreadState_New() calls _PyGILState_NoteThreadState() which checks that autoTLSkey is not zero, but I don't know if autoTLSkey have the right value with my preallocation patch.
There is another problem in _testcapi: test_thread_state() calls directly PyThread_start_new_thread() and the thread function calls PyGILState_Ensure(). PyGILState_Ensure() would then require to create a new thread state (call PyThreadState_New()) because it was not done by PyThread_start_new_thread(). On low memory condition, we hit the same bug here.
Only thread and _testcapi modules calls directly PyThread_start_new_thread(). _*test*capi is reserved to tests, so I consider that we don't care about bugs under low memory condition in this module.
Traceback (most recent call last):
File "/home/antoine/cpython/debug/Lib/test/test_threading.py", line 476, in test_3_join_in_forked_from_threadself._run_and_join(script)
File "/home/antoine/cpython/debug/Lib/test/test_threading.py", line 412, in _run_and_joinself.assertEqual(data, "end of main\nend of thread\n")
AssertionError: 'end of main\n' != 'end of main\nend of thread\n'
The problem is here: PyThreadState_New() -> _PyGILState_NoteThreadState() -> PyThread_set_key_value() -> find_key() and find_key() calls PyThread_get_thread_ident(), but the ident is not the right ident :-/
PyThreadState_New() should not call _PyGILState_NoteThreadState(), it should be done _in_ the thread.
I wrote a new patch fixing the unit test. PyThreadState_New() is part of the public API, so I can't change its prototype. And _PyGILState_NoteThreadState() is a private function (use the "static" keyword). Because of that, I introduced two new functions:
PyThreadState_Prealloc(): like PyThreadState_New() but for thread preallocation, can be called outside the new state (from another thread)
PyThreadState_Init(): have to be called in the new thread to finish the thread state initialization, only required for state created by PyThreadState_Prealloc() (not for PyThreadState_New())
I tried all Lib/test/test_thread*.py tests in debug mode and all test now pass ;-)
Another problem with my patch! If initsite() calls PyGILState_Ensure(): assert(autoInterpreterState) fails because autoInterpreterState is NULL.
_PyGILState_Init() have to be called before initsite(). I don't know where it should be called exactly. Why not just after the PyThreadState_New() (in Py_InitializeEx())?
Note: I was using gdb to track a bug on a debug build (--with-pydebug). I used "pyo" macro which calls _PyObject_Dump(), and _PyObject_Dump() calls PyGILState_Ensure() => assertion error.
PyThreadState_Prealloc and PyThreadState_Init should probably be prefixed with an underscore, because there's no use for them outside of the interpreter
_PyThreadState_New should be static. Besides, static functions usually don't get the "Py" in their name and are all lowercased, so something like "new_threadstate"
the last change ("Py_InitializeEx() calls _PyGILState_Init() before initsite()") should be part of a separate patch (and issue?)
PyThreadState_Prealloc and PyThreadState_Init should (...) be prefixed with an underscore (...)
done
_PyThreadState_New should be static (...) so something like "new_threadstate"
done
the last change ("Py_InitializeEx() calls _PyGILState_Init() before initsite()") should be part of a separate patch (and issue?)
I'm unable to reproduce the bug related to this patch. I keep the patch somewhere in my disk, and I will open a new issue if I'm able to reproduce it :-)
you can use the -R option to regrtest to find out if there are any reference leaks (e.g. "-R 3:2:")
I tried all test*thread*py. test_threadsignals leaks a reference... even without my patch, so I opened a new issue: bpo-7825.
Activity
vstinner commentedon Dec 19, 2009
Using my fuzzer (Fusil) on Python trunk, I got sometimes errors on
multiprocessing.Pool():
Fatal Python error: PyEval_AcquireThread: NULL new thread state
I'm sorry but I don't have example script to reproduce the bug. I
suppose that the error depends on the system load. I have the error on
an Intel Core2 Quad Core Q9300 (CPU able to run 4 processes/threads at
the time same).
How can I get more information on the bug? How can I reproduce it?
pitrou commentedon Dec 19, 2009
Enable core dumps (
ulimit -c unlimited) and re-run your tests. Thefatal Python error will leave a core file so that you can find out the
context.
You can also try a debug build of Python which may give you more
information, assuming it can reproduce the bug.
(as for how to reproduce the bug, it's your task to find it out :-))
vstinner commentedon Dec 20, 2009
Use a core dump: good idea!
haypo> Using my fuzzer (Fusil) on Python trunk, I got sometimes
haypo> errors on multiprocessing.Pool():
haypo>
haypo> Fatal Python error: PyEval_AcquireThread: NULL new thread state
I read the source code of the thread module. This error means that
PyThreadState_New() returns NULL which occurs if malloc() failed. I hit this
error using my fuzzer because the fuzzer limits the total memory to something
around 100 MB using setrlimit().
Said differently: in low memory condition, creating a new thread may exit the
whole Python process if a memory allocation fail.
--
Sometimes, I get another error, similar to the "NULL new thread state" error:
Fatal Python error: Couldn't create autoTLSkey mapping
I guess that the reason is the same: memory allocation failed. It should be
the malloc() in find_key() (Python/thread.c).
pitrou commentedon Dec 21, 2009
We just had this error on one of the buildbots:
[...]
test_threadsignals
sem_wait: Unknown error 512
test_docxmlrpc
Fatal Python error: Invalid thread state for this thread
Fatal Python error: PyEval_AcquireThread: non-NULL old thread state
make: *** [buildbottest] Aborted
(see
http://www.python.org/dev/buildbot/all/builders/x86%20Ubuntu%20trunk/builds/161/steps/test/logs/stdio
)
I don't know it's related.
vstinner commentedon Dec 21, 2009
No, I don't think that this issue is related, because it starts with
another message: "Invalid thread state for this thread".
[-]multiprocessing.Pool(): Fatal Python error: PyEval_AcquireThread: NULL new thread state[/-][+]Fatal error on thread creation in low memory condition[/+]nirai commentedon Dec 27, 2009
Memory can be pre-allocated by thread_PyThread_start_new_thread() before
thread is spawned.
vstinner commentedon Jan 9, 2010
Here I come with a patch! nirai idea was the good one: prealloc PyThreadState before creating the thread. Raise a MemoryError if the allocation fail, instead of raising a *fatal* Python error.
Patch is quite simple and allow better error handling.
amauryfa commentedon Jan 11, 2010
The patch looks good; the line
tstate->thread_id = PyThread_get_thread_ident();
is needed because the tstate is created in the main thread, but used in another thread.
vstinner commentedon Jan 11, 2010
I'm not sure about autoTLSkey value. PyThreadState_New() calls _PyGILState_NoteThreadState() which checks that autoTLSkey is not zero, but I don't know if autoTLSkey have the right value with my preallocation patch.
vstinner commentedon Jan 11, 2010
After some tests (printf fun), it looks ok.
--
There is another problem in _testcapi: test_thread_state() calls directly PyThread_start_new_thread() and the thread function calls PyGILState_Ensure(). PyGILState_Ensure() would then require to create a new thread state (call PyThreadState_New()) because it was not done by PyThread_start_new_thread(). On low memory condition, we hit the same bug here.
Only thread and _testcapi modules calls directly PyThread_start_new_thread(). _*test*capi is reserved to tests, so I consider that we don't care about bugs under low memory condition in this module.
pitrou commentedon Jan 12, 2010
Running the tests in debug mode gives the following error:
test_3_join_in_forked_from_thread (test.test_threading.ThreadJoinOnShutdown) ... Fatal Python error: Invalid thread state for this thread
[21851 refs]
FAIL
[snip]
======================================================================
FAIL: test_3_join_in_forked_from_thread (test.test_threading.ThreadJoinOnShutdown)
----------------------------------------------------------------------
Ran 78 tests in 3.739s
vstinner commentedon Jan 13, 2010
I tried all Lib/test/test_thread*py, but not in debug mode :-/
The problem is here: PyThreadState_New() -> _PyGILState_NoteThreadState() -> PyThread_set_key_value() -> find_key() and find_key() calls PyThread_get_thread_ident(), but the ident is not the right ident :-/
PyThreadState_New() should not call _PyGILState_NoteThreadState(), it should be done _in_ the thread.
I wrote a new patch fixing the unit test. PyThreadState_New() is part of the public API, so I can't change its prototype. And _PyGILState_NoteThreadState() is a private function (use the "static" keyword). Because of that, I introduced two new functions:
PyThreadState_Prealloc(): like PyThreadState_New() but for thread preallocation, can be called outside the new state (from another thread)
PyThreadState_Init(): have to be called in the new thread to finish the thread state initialization, only required for state created by PyThreadState_Prealloc() (not for PyThreadState_New())
I tried all Lib/test/test_thread*.py tests in debug mode and all test now pass ;-)
vstinner commentedon Jan 13, 2010
Another problem with my patch! If initsite() calls PyGILState_Ensure(): assert(autoInterpreterState) fails because autoInterpreterState is NULL.
_PyGILState_Init() have to be called before initsite(). I don't know where it should be called exactly. Why not just after the PyThreadState_New() (in Py_InitializeEx())?
vstinner commentedon Jan 13, 2010
Note: I was using gdb to track a bug on a debug build (--with-pydebug). I used "pyo" macro which calls _PyObject_Dump(), and _PyObject_Dump() calls PyGILState_Ensure() => assertion error.
vstinner commentedon Jan 21, 2010
Sum up of my patch:
pitrou commentedon Jan 30, 2010
I didn't test the patch but some comments:
pitrou commentedon Jan 30, 2010
Oh, and besides, you can use the -R option to regrtest to find out if there are any reference leaks (e.g. "-R 3:2:").
vstinner commentedon Jan 31, 2010
done
done
I'm unable to reproduce the bug related to this patch. I keep the patch somewhere in my disk, and I will open a new issue if I'm able to reproduce it :-)
I tried all test*thread*py. test_threadsignals leaks a reference... even without my patch, so I opened a new issue: bpo-7825.
vstinner commentedon Mar 3, 2010
Commited: r78638 (trunk), r78639 (py3k), r78640 (3.1).
Keep the issue open to remember me that I have to backport it to 2.6 (the branch is frozen waiting for the 2.6.5 final version).
vstinner commentedon Mar 21, 2010
Backport done: r79199 (2.6).