You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
trip_signal() gets NULL tstate on Windows on CTRL+C #84263
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee=Noneclosed_at=<Date2020-04-08.21:36:15.466>created_at=<Date2020-03-27.06:07:12.027>labels= ['interpreter-core', '3.9', 'type-crash']
title='trip_signal() gets NULL tstate on Windows on CTRL+C'updated_at=<Date2020-04-08.21:36:15.465>user='https://github.com/ariccio'
While trying to make sense of some static analysis warnings for the Windows console IO module, I Ctrl+C'd in the middle of an intentionally absurd __repr__ output, and on proceeding in the debugger (which treated it as an exception), I immediately hit the assertion right here:
/* Get the Python thread state using PyGILState API, since _PyThreadState_GET() returns NULL if the GIL is released. For example, signal.raise_signal() releases the GIL. */PyThreadState*tstate=PyGILState_GetThisThreadState();
assert(tstate!=NULL);
...With the stacktrace:
ucrtbased.dll!issue_debug_notification(const wchar_t * const message) Line 28 C++
ucrtbased.dll!__acrt_report_runtime_error(const wchar_t * message) Line 154 C++
ucrtbased.dll!abort() Line 51 C++
ucrtbased.dll!common_assert_to_stderr_direct(const wchar_t * const expression, const wchar_t * const file_name, const unsigned int line_number) Line 161 C++
ucrtbased.dll!common_assert_to_stderr<wchar_t>(const wchar_t * const expression, const wchar_t * const file_name, const unsigned int line_number) Line 175 C++
ucrtbased.dll!common_assert<wchar_t>(const wchar_t * const expression, const wchar_t * const file_name, const unsigned int line_number, void * const return_address) Line 420 C++
ucrtbased.dll!_wassert(const wchar_t * expression, const wchar_t * file_name, unsigned int line_number) Line 443 C++
python39_d.dll!trip_signal(int sig_num) Line 266 C
python39_d.dll!signal_handler(int sig_num) Line 342 C
ucrtbased.dll!ctrlevent_capture(const unsigned long ctrl_type) Line 206 C++
KernelBase.dll!_CtrlRoutine@4�() Unknown
kernel32.dll!@BaseThreadInitThunk@12�() Unknown
ntdll.dll!__RtlUserThreadStart() Unknown
ntdll.dll!__RtlUserThreadStart@8�() Unknown
...I'm not entirely sure why this happened, but I can tell a few things. _PyRuntime.gilstate.autoInterpreterState is NOT null, in fact the gilstate object is as displayed in my watch window:
This looks to me like there's some kind of race condition in the thread local storage. Later, when TlsGetValue is called in PyThread_tss_get with the value of 5, the error code is lost, so I don't even know the exact reported error, and apparently 0 (nullptr) is a valid return anyways *shrug*. If I'm correctly decoding the address of the TLS slot in the TEB (I think *(unsigned int*)(void*)((@fs+0xe10)+ (20)) is the correct address for the 5th item? 4 bytes, key=5?), then there is actually a null tstate there. Not sure why.
This is in a relatively recent (<1wk old) branch of the code, with some non-behavioral tweaks to add annotations in while I'm bug hunting, so it shouldn't matter. I'm not sure if I can reproduce this, but it happened, so there's a bug somewhere.
What else? The signal number that tripped this was 2, sig interrupt, which makes sense.
There are other threads active, so maybe that's why? The TLS was never initialized for that thread? Here's the dump from the visual studio threads window, 22316 is the active thread.
Not Flagged 10360 0 Main Thread Main Thread python39_d.dll!_PyOS_WindowsConsoleReadline
ntdll.dll!_NtDeviceIoControlFile@40�()
KernelBase.dll!ConsoleCallServerGeneric()
KernelBase.dll!_ReadConsoleInternal@24�()
KernelBase.dll!_ReadConsoleW@20�()
python39_d.dll!_PyOS_WindowsConsoleReadline(void * hStdIn) Line 120
python39_d.dll!PyOS_StdioReadline(_iobuf * sys_stdin, _iobuf * sys_stdout, const char * prompt) Line 253
python39_d.dll!PyOS_Readline(_iobuf * sys_stdin, _iobuf * sys_stdout, const char * prompt) Line 358
python39_d.dll!tok_nextc(tok_state * tok) Line 856
python39_d.dll!tok_get(tok_state * tok, const char * * p_start, const char * * p_end) Line 1166
python39_d.dll!PyTokenizer_Get(tok_state * tok, const char * * p_start, const char * * p_end) Line 1813
python39_d.dll!parsetok(tok_state * tok, grammar * g, int start, perrdetail * err_ret, int * flags) Line 253
python39_d.dll!PyParser_ParseFileObject(_iobuf * fp, _object * filename, const char * enc, grammar * g, int start, const char * ps1, const char * ps2, perrdetail * err_ret, int * flags) Line 188
python39_d.dll!PyParser_ASTFromFileObject(_iobuf * fp, _object * filename, const char * enc, int start, const char * ps1, const char * ps2, PyCompilerFlags * flags, int * errcode, _arena * arena) Line 1388
python39_d.dll!PyRun_InteractiveOneObjectEx(_iobuf * fp, _object * filename, PyCompilerFlags * flags) Line 240
python39_d.dll!PyRun_InteractiveLoopFlags(_iobuf * fp, const char * filename_str, PyCompilerFlags * flags) Line 122
python39_d.dll!PyRun_AnyFileExFlags(_iobuf * fp, const char * filename, int closeit, PyCompilerFlags * flags) Line 81
python39_d.dll!pymain_run_stdin(PyConfig * config, PyCompilerFlags * cf) Line 467
python39_d.dll!pymain_run_python(int * exitcode) Line 556
python39_d.dll!Py_RunMain() Line 632
python39_d.dll!pymain_main(_PyArgv * args) Line 663
python39_d.dll!Py_Main(int argc, wchar_t * * argv) Line 674
python_d.exe!wmain(int argc, wchar_t * * argv) Line 10
python_d.exe!invoke_main() Line 90
python_d.exe!__scrt_common_main_seh() Line 288
python_d.exe!__scrt_common_main() Line 331
python_d.exe!wmainCRTStartup() Line 17
kernel32.dll!@BaseThreadInitThunk@12�()
ntdll.dll!__RtlUserThreadStart()
ntdll.dll!__RtlUserThreadStart@8�()
Lmao the name mangling comes up as a mailto. That's interesting.
ariccio commented on Mar 27, 2020
aricciomannequin
MannequinAuthor
More actions
Hmmm, happens every time I interrupt while attached. Is there some obvious gotcha in the docs that I'm missing?
neonene commented on Apr 6, 2020
neonenemannequin
Mannequin
More actions
On Windows, PyGILState_GetThisThreadState() returns NULL when ^C-interrupt occurs. It is from TlsGetValue() winAPI and I don't think the os's behevior is wrong.
In trip_signal(), crash can be avoided by skipping PyEval_SignalReceived() if tstate is invalid. But I'm not sure the skip itself is ok.
Oh oh. This issue is quite annoying for my work on subinterpreters.
I introduced this bug when I moved pending calls from _PyRuntimeState to PyInterpreterState in bpo-39984. _PyEval_AddPendingCall() now requires tstate to add a function to pending calls of the proper interpreter.
The problem on Windows is that each CTRL+c is executed in a different thread. Here is a modified Python 3.8 which dumps the thread identifier ("tid") at startup and when trip_signal() is triggered by CTRL+C:
vstinner@WIN C:\vstinner\python\3.8>python
Running Release|x64 interpreter...
pymain_main: tid=1788
Python 3.8.1+ (heads/3.8-dirty:19be85c765, Apr 8 2020, 19:35:20) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> ^C
trip_signal: tid=6996 tstate=0000000000000000
KeyboardInterrupt
>>> ^C
trip_signal: tid=2384 tstate=0000000000000000
KeyboardInterrupt
>>> ^C
trip_signal: tid=32 tstate=0000000000000000
KeyboardInterrupt
When trip_signal() is called, PyGILState_GetThisThreadState() returns NULL.
Activity
ariccio commentedon Mar 27, 2020
While trying to make sense of some static analysis warnings for the Windows console IO module, I Ctrl+C'd in the middle of an intentionally absurd __repr__ output, and on proceeding in the debugger (which treated it as an exception), I immediately hit the assertion right here:
...With the stacktrace:
ucrtbased.dll!issue_debug_notification(const wchar_t * const message) Line 28 C++
ucrtbased.dll!__acrt_report_runtime_error(const wchar_t * message) Line 154 C++
ucrtbased.dll!abort() Line 51 C++
ucrtbased.dll!common_assert_to_stderr_direct(const wchar_t * const expression, const wchar_t * const file_name, const unsigned int line_number) Line 161 C++
ucrtbased.dll!common_assert_to_stderr<wchar_t>(const wchar_t * const expression, const wchar_t * const file_name, const unsigned int line_number) Line 175 C++
ucrtbased.dll!common_assert<wchar_t>(const wchar_t * const expression, const wchar_t * const file_name, const unsigned int line_number, void * const return_address) Line 420 C++
ucrtbased.dll!_wassert(const wchar_t * expression, const wchar_t * file_name, unsigned int line_number) Line 443 C++
...I'm not entirely sure why this happened, but I can tell a few things. _PyRuntime.gilstate.autoInterpreterState is NOT null, in fact the gilstate object is as displayed in my watch window:
This looks to me like there's some kind of race condition in the thread local storage. Later, when TlsGetValue is called in PyThread_tss_get with the value of 5, the error code is lost, so I don't even know the exact reported error, and apparently 0 (nullptr) is a valid return anyways *shrug*. If I'm correctly decoding the address of the TLS slot in the TEB (I think *(unsigned int*)(void*)((@fs+0xe10)+ (20)) is the correct address for the 5th item? 4 bytes, key=5?), then there is actually a null tstate there. Not sure why.
This is in a relatively recent (<1wk old) branch of the code, with some non-behavioral tweaks to add annotations in while I'm bug hunting, so it shouldn't matter. I'm not sure if I can reproduce this, but it happened, so there's a bug somewhere.
What else? The signal number that tripped this was 2, sig interrupt, which makes sense.
There are other threads active, so maybe that's why? The TLS was never initialized for that thread? Here's the dump from the visual studio threads window, 22316 is the active thread.
Not Flagged 10360 0 Main Thread Main Thread python39_d.dll!_PyOS_WindowsConsoleReadline
ntdll.dll!_NtDeviceIoControlFile@40�()
KernelBase.dll!ConsoleCallServerGeneric()
KernelBase.dll!_ReadConsoleInternal@24�()
KernelBase.dll!_ReadConsoleW@20�()
python39_d.dll!_PyOS_WindowsConsoleReadline(void * hStdIn) Line 120
python39_d.dll!PyOS_StdioReadline(_iobuf * sys_stdin, _iobuf * sys_stdout, const char * prompt) Line 253
python39_d.dll!PyOS_Readline(_iobuf * sys_stdin, _iobuf * sys_stdout, const char * prompt) Line 358
python39_d.dll!tok_nextc(tok_state * tok) Line 856
python39_d.dll!tok_get(tok_state * tok, const char * * p_start, const char * * p_end) Line 1166
python39_d.dll!PyTokenizer_Get(tok_state * tok, const char * * p_start, const char * * p_end) Line 1813
python39_d.dll!parsetok(tok_state * tok, grammar * g, int start, perrdetail * err_ret, int * flags) Line 253
python39_d.dll!PyParser_ParseFileObject(_iobuf * fp, _object * filename, const char * enc, grammar * g, int start, const char * ps1, const char * ps2, perrdetail * err_ret, int * flags) Line 188
python39_d.dll!PyParser_ASTFromFileObject(_iobuf * fp, _object * filename, const char * enc, int start, const char * ps1, const char * ps2, PyCompilerFlags * flags, int * errcode, _arena * arena) Line 1388
python39_d.dll!PyRun_InteractiveOneObjectEx(_iobuf * fp, _object * filename, PyCompilerFlags * flags) Line 240
python39_d.dll!PyRun_InteractiveLoopFlags(_iobuf * fp, const char * filename_str, PyCompilerFlags * flags) Line 122
python39_d.dll!PyRun_AnyFileExFlags(_iobuf * fp, const char * filename, int closeit, PyCompilerFlags * flags) Line 81
python39_d.dll!pymain_run_stdin(PyConfig * config, PyCompilerFlags * cf) Line 467
python39_d.dll!pymain_run_python(int * exitcode) Line 556
python39_d.dll!Py_RunMain() Line 632
python39_d.dll!pymain_main(_PyArgv * args) Line 663
python39_d.dll!Py_Main(int argc, wchar_t * * argv) Line 674
python_d.exe!wmain(int argc, wchar_t * * argv) Line 10
python_d.exe!invoke_main() Line 90
python_d.exe!__scrt_common_main_seh() Line 288
python_d.exe!__scrt_common_main() Line 331
python_d.exe!wmainCRTStartup() Line 17
kernel32.dll!@BaseThreadInitThunk@12�()
ntdll.dll!__RtlUserThreadStart()
ntdll.dll!__RtlUserThreadStart@8�()
Not Flagged 14944 0 Worker Thread ntdll.dll!_TppWorkerThread@4�() ntdll.dll!_NtWaitForWorkViaWorkerFactory@20
ntdll.dll!_NtWaitForWorkViaWorkerFactory@20�()
ntdll.dll!_TppWorkerThread@4�()
kernel32.dll!@BaseThreadInitThunk@12�()
ntdll.dll!__RtlUserThreadStart()
ntdll.dll!__RtlUserThreadStart@8�()
Not Flagged > 22316 0 Worker Thread KernelBase.dll!_CtrlRoutine@4�() ucrtbased.dll!issue_debug_notification
ucrtbased.dll!issue_debug_notification(const wchar_t * const message) Line 28
ucrtbased.dll!__acrt_report_runtime_error(const wchar_t * message) Line 154
ucrtbased.dll!abort() Line 51
ucrtbased.dll!common_assert_to_stderr_direct(const wchar_t * const expression, const wchar_t * const file_name, const unsigned int line_number) Line 161
ucrtbased.dll!common_assert_to_stderr<wchar_t>(const wchar_t * const expression, const wchar_t * const file_name, const unsigned int line_number) Line 175
ucrtbased.dll!common_assert<wchar_t>(const wchar_t * const expression, const wchar_t * const file_name, const unsigned int line_number, void * const return_address) Line 420
ucrtbased.dll!_wassert(const wchar_t * expression, const wchar_t * file_name, unsigned int line_number) Line 443
python39_d.dll!trip_signal(int sig_num) Line 266
python39_d.dll!signal_handler(int sig_num) Line 342
ucrtbased.dll!ctrlevent_capture(const unsigned long ctrl_type) Line 206
KernelBase.dll!_CtrlRoutine@4�()
kernel32.dll!@BaseThreadInitThunk@12�()
ntdll.dll!__RtlUserThreadStart()
ntdll.dll!__RtlUserThreadStart@8�()
Not Flagged 10180 0 Worker Thread ntdll.dll!_TppWorkerThread@4�() ntdll.dll!_NtWaitForWorkViaWorkerFactory@20
ntdll.dll!_NtWaitForWorkViaWorkerFactory@20�()
ntdll.dll!_TppWorkerThread@4�()
kernel32.dll!@BaseThreadInitThunk@12�()
ntdll.dll!__RtlUserThreadStart()
ntdll.dll!__RtlUserThreadStart@8�()
Not Flagged 28940 0 Worker Thread ntdll.dll!_TppWorkerThread@4�() ntdll.dll!_NtWaitForWorkViaWorkerFactory@20
ntdll.dll!_NtWaitForWorkViaWorkerFactory@20�()
ntdll.dll!_TppWorkerThread@4�()
kernel32.dll!@BaseThreadInitThunk@12�()
ntdll.dll!__RtlUserThreadStart()
ntdll.dll!__RtlUserThreadStart@8�()
Not Flagged 9396 0 Worker Thread ntdll.dll!_TppWorkerThread@4�() ntdll.dll!_NtWaitForWorkViaWorkerFactory@20
ntdll.dll!_NtWaitForWorkViaWorkerFactory@20�()
ntdll.dll!_TppWorkerThread@4�()
kernel32.dll!@BaseThreadInitThunk@12�()
ntdll.dll!__RtlUserThreadStart()
ntdll.dll!__RtlUserThreadStart@8�()
Not Flagged 28960 0 Worker Thread ntdll.dll!_TppWorkerThread@4�() ntdll.dll!_NtWaitForWorkViaWorkerFactory@20
ntdll.dll!_NtWaitForWorkViaWorkerFactory@20�()
ntdll.dll!_TppWorkerThread@4�()
kernel32.dll!@BaseThreadInitThunk@12�()
ntdll.dll!__RtlUserThreadStart()
ntdll.dll!__RtlUserThreadStart@8�()
Anyways, I hope that's a useful report for an obscure bug.
ariccio commentedon Mar 27, 2020
Lmao the name mangling comes up as a mailto. That's interesting.
ariccio commentedon Mar 27, 2020
Hmmm, happens every time I interrupt while attached. Is there some obvious gotcha in the docs that I'm missing?
neonene commentedon Apr 6, 2020
On Windows, PyGILState_GetThisThreadState() returns NULL when ^C-interrupt occurs. It is from TlsGetValue() winAPI and I don't think the os's behevior is wrong.
In trip_signal(), crash can be avoided by skipping PyEval_SignalReceived() if tstate is invalid. But I'm not sure the skip itself is ok.
vstinner commentedon Apr 8, 2020
Oh oh. This issue is quite annoying for my work on subinterpreters.
I introduced this bug when I moved pending calls from _PyRuntimeState to PyInterpreterState in bpo-39984. _PyEval_AddPendingCall() now requires tstate to add a function to pending calls of the proper interpreter.
The problem on Windows is that each CTRL+c is executed in a different thread. Here is a modified Python 3.8 which dumps the thread identifier ("tid") at startup and when trip_signal() is triggered by CTRL+C:
When trip_signal() is called, PyGILState_GetThisThreadState() returns NULL.
[-]Assertion failure in trip_signal[/-][+]trip_signal() gets NULL tstate on Windows on CTRL+C[/+][-]Assertion failure in trip_signal[/-][+]trip_signal() gets NULL tstate on Windows on CTRL+C[/+]vstinner commentedon Apr 8, 2020
New changeset b54a99d by Victor Stinner in branch 'master':
bpo-40082: trip_signal() uses the main interpreter (GH-19441)
b54a99d
vstinner commentedon Apr 8, 2020
Thanks for the bug report Alexander Riccio. I fixed bug in master. Python 3.8 is not affected.