You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
atexit callbacks should be run at subinterpreter shutdown #76082
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee=Noneclosed_at=<Date2017-12-20.10:18:52.338>created_at=<Date2017-10-30.15:44:43.120>labels= ['interpreter-core', 'type-bug', '3.7']
title='atexit callbacks should be run at subinterpreter shutdown'updated_at=<Date2017-12-20.10:18:52.337>user='https://github.com/Dormouse759'
Hmm, I don't think calling Py_Finalize in any interpreter other than the main one is actually defined at all, so I'm inclined to just make it an error, rather than trying to figure out how it should work. It would then be up to the embedding application to make sure it switched back to the main interpreter before calling Py_Finalize.
(We don't have this concern with Py_Initialize, since that *creates* the main interpreter)
For the main interpreter, Py_Finalize should be destroying all other subinterpreters as one of the first things it does, so if we're *not* currently doing that, it would make sense to fix it as part of Eric's subinterpreters PEP.
I guess we allow an unhandled SystemExit in a child thread to propagate to (and hence terminate) the main thread, so allowing a Py_Finalize call in a subinterpreter to terminate the main interpreter would be comparable to that.
My main rationale for *requiring* that the main interpreter be active (or be made active) when shutting down is to reduce the number of scenarios we need to test (right now we only test Py_Initialize/Py_Finalize cycles with a single interpreter, and officially allowing finalization from arbitrary interpreters expands that test matrix a fair bit).
I'm not sure where the concept of "main subinterpreter" comes in, with respect to this issue.
I thnik the issue of atexit callbacks could be solved by something like keeping info about each callback's subinterpreter, and switching subinterpreters before running each one. I can see the locking needed for that being quite hairy, but independent of which thread calls all this.
When you destroy a subinterpreter before Py_Finalize is called, Python can't start calling its atexit callbacks – they no longer have a subinterpreter to run in.
Therefore I think callbacks for a particular subinterpreter should be called when (and only when) that subinterpreter is destroyed. Regardless of whether it's the main one or not.
grahamd commented on Nov 8, 2017
grahamdmannequin
Mannequin
More actions
FWIW, that atexit callbacks were not called for sub interpreters ever was a pain point for mod_wsgi.
What mod_wsgi does is override the destruction sequence so that it will first go through each sub interpreter when and shutdown threading explicitly, then call atexit handlers. When that is done, only then will it destroy the sub interpreter and the main interpreter.
I have noted this previously in discussion associated with:
I've retitled the issue to make it clearer that the problem is where the responsibility for calling atexit functions lives - currently it's in Py_Finalize, but really it should be inside the interpreter object's own teardown code.
changed the title [-]atexit callbacks only called for current subinterpreter[/-][+]atexit callbacks should be run at subinterpreter shutdown[/+]on Nov 9, 2017
This makes Py_EndInterpreter() call atexit callbacks for the subinterpreter it is destroying.
It doesn't make Py_Finalize() end all subinterpreters, as the current implementation of subinterpreters makes it hard to do so. This is the same as the current behaviour: you need to end all subinterpreters before calling Py_Finalize().
New changeset 776407f by Antoine Pitrou (Marcel Plch) in branch 'master': bpo-31901: atexit callbacks should be run at subinterpreter shutdown (bpo-4611) 776407f
Activity
Dormouse759 commentedon Oct 30, 2017
Py_FinalizeEx()only executes callbacks from the subinterpreter from which Py_FinalizeEx is called.What is the expected behavior here?
Should the callbacks be specific to individual subinterpreters?
ncoghlan commentedon Oct 31, 2017
Hmm, I don't think calling Py_Finalize in any interpreter other than the main one is actually defined at all, so I'm inclined to just make it an error, rather than trying to figure out how it should work. It would then be up to the embedding application to make sure it switched back to the main interpreter before calling Py_Finalize.
(We don't have this concern with Py_Initialize, since that *creates* the main interpreter)
ncoghlan commentedon Oct 31, 2017
For the main interpreter, Py_Finalize should be destroying all other subinterpreters as one of the first things it does, so if we're *not* currently doing that, it would make sense to fix it as part of Eric's subinterpreters PEP.
encukou commentedon Oct 31, 2017
I don't have a reason to think Py_Finalize is not *destroying* the other subinterpreters. But it's not calling their atexit callbacks.
(Is there any reason Py_Finalize couldn't switch to the main interpreter itself? Assuming it's even necessary.)
ncoghlan commentedon Oct 31, 2017
I guess we allow an unhandled SystemExit in a child thread to propagate to (and hence terminate) the main thread, so allowing a Py_Finalize call in a subinterpreter to terminate the main interpreter would be comparable to that.
My main rationale for *requiring* that the main interpreter be active (or be made active) when shutting down is to reduce the number of scenarios we need to test (right now we only test Py_Initialize/Py_Finalize cycles with a single interpreter, and officially allowing finalization from arbitrary interpreters expands that test matrix a fair bit).
encukou commentedon Oct 31, 2017
I'm not sure where the concept of "main subinterpreter" comes in, with respect to this issue.
I thnik the issue of atexit callbacks could be solved by something like keeping info about each callback's subinterpreter, and switching subinterpreters before running each one. I can see the locking needed for that being quite hairy, but independent of which thread calls all this.
encukou commentedon Nov 8, 2017
When you destroy a subinterpreter before Py_Finalize is called, Python can't start calling its atexit callbacks – they no longer have a subinterpreter to run in.
Therefore I think callbacks for a particular subinterpreter should be called when (and only when) that subinterpreter is destroyed. Regardless of whether it's the main one or not.
grahamd commentedon Nov 8, 2017
FWIW, that atexit callbacks were not called for sub interpreters ever was a pain point for mod_wsgi.
What mod_wsgi does is override the destruction sequence so that it will first go through each sub interpreter when and shutdown threading explicitly, then call atexit handlers. When that is done, only then will it destroy the sub interpreter and the main interpreter.
I have noted this previously in discussion associated with:
https://bugs.python.org/issue6531
ncoghlan commentedon Nov 9, 2017
Ah, I see - yeah, that makes a lot of sense.
I've retitled the issue to make it clearer that the problem is where the responsibility for calling atexit functions lives - currently it's in Py_Finalize, but really it should be inside the interpreter object's own teardown code.
[-]atexit callbacks only called for current subinterpreter[/-][+]atexit callbacks should be run at subinterpreter shutdown[/+]Dormouse759 commentedon Dec 4, 2017
I created a PR with fix on this issue - #4611
This makes Py_EndInterpreter() call atexit callbacks for the subinterpreter it is destroying.
It doesn't make Py_Finalize() end all subinterpreters, as the current implementation of subinterpreters makes it hard to do so. This is the same as the current behaviour: you need to end all subinterpreters before calling Py_Finalize().
pitrou commentedon Dec 20, 2017
New changeset 776407f by Antoine Pitrou (Marcel Plch) in branch 'master':
bpo-31901: atexit callbacks should be run at subinterpreter shutdown (bpo-4611)
776407f