You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Kristjan wrote improved locking primitives in bpo-15038 that use the new (in Vista) SRWLock and Condition Variable APIs. SRWLocks (used in exclusive mode only) replace Critical Sections, which is slower than SRWLock and provides no features we use that might justify it. Condition Variables replace Semaphores, where the former is user mode (cheap) and the latter kernel mode (expensive).
These changes remain disabled by default.
Given that CPython dropped support for pre-Vista OSes in 3.5, I propose enabling the faster locking primitives by default. The PR I'll submit leaves the condition variable emulation code in the source so it's available to people who might try to build XP/WS03 compatible code, it just tweaks the define so it defaults to using the Vista+ APIs.
Based on the numbers from bpo-15038, we should expect to see a significant improvement in speed.
Note: Beyond turning on the new primitives by default, I also made a change to PyCOND_TIMEDWAIT. The original code looked wrong, in that:
It assumed that when SleepConditionVariableSRW returned non-zero, you didn't know if the wait had timed out or not, so it returned 2 to indicate "Might have timed out, act as if it timed out"
It assumed that when SleepConditionVariableSRW returned zero, it had suffered a critical failure, and returned -1 to indicate that.
Neither of these things is true AFAICT. SleepConditionVariableSRW returns non-zero when it's alerted prior to timing out. Otherwise, it returns 0, and you have to check GetLastError to distinguish timeout from critical failures. Documentation is here: https://msdn.microsoft.com/en-us/library/windows/desktop/ms686304(v=vs.85).aspx
It's subject to "spurious" wake-ups, but so is pthread_cond_timedwait and the pthreads code doesn't use an "all timedwaits time out".
MojoVampire commented on Mar 22, 2017
MojoVampiremannequin
MannequinAuthor
More actions
Hmm... I was only running the threading related tests originally, and they passed, but it looks like this causes problems with multiprocessing (test_multiprocessing_spawn and test_concurrent_futures both stall out forever with this change, with or without my fix to the PyCOND_TIMEDWAIT function). I'll investigate further, don't bother to review yet.
kristjanvalur commented on Mar 22, 2017
kristjanvalurmannequin
Mannequin
More actions
Hi there.
Looking at the API docs today
(https://msdn.microsoft.com/en-us/library/windows/desktop/ms686304(v=vs.85).aspx)
it appears that the timeout case is documented. I'm fairly sure that it wasn't when I implemented it. There was a good reason for the "2" return code. The idea was: Either there was an error (-1) or we woke up. Since there are spurious wakeups and stolen wakeups, the predicate must be tested again anyway. The '2' return code would mean that the timeout condition should be tested by looking at some external clock.
Now, the api documentation is bad. Return value is BOOL.
Nonzero means "success" (whatever that means)
Failure means ´zero´
Timeout means FALSE and GetLastError() == ERROR_TIMEOUT
If memory serves, FALSE == 0 on windows....
Anyway, I've been out of this part of the code for sufficient time for details to be blurry.
My advise:
Check that the API of the function is indeed correct.
If there is no bulletproof way of distinguishing timeout from normal return, just consider all returns normal (remember, non-error return means that we woke up, not that _we_ were signaled)
Verify that the code that is failing can indeed support spurious wakeups/stolen wakeups. It used to be that the python condition variables didn't have this property, because of the way they were implemented. This may have made people lazy.
Unfortunately, there appear to be a number of regressions due to this. I'm not going to have time right now to work through them myself, but if someone else can it would be great to resolve these and get the change in.
On bpo-35562 Jeff posted a deeper analysis of the issue in TIMEDWAIT. That will need fixing along with the other regressions before we can enable these.
MojoVampire commented on Jan 15, 2019
MojoVampiremannequin
MannequinAuthor
More actions
I assume you meant bpo-35662 (based on the superseder note in the history).
Activity
MojoVampire commentedon Mar 22, 2017
Kristjan wrote improved locking primitives in bpo-15038 that use the new (in Vista) SRWLock and Condition Variable APIs. SRWLocks (used in exclusive mode only) replace Critical Sections, which is slower than SRWLock and provides no features we use that might justify it. Condition Variables replace Semaphores, where the former is user mode (cheap) and the latter kernel mode (expensive).
These changes remain disabled by default.
Given that CPython dropped support for pre-Vista OSes in 3.5, I propose enabling the faster locking primitives by default. The PR I'll submit leaves the condition variable emulation code in the source so it's available to people who might try to build XP/WS03 compatible code, it just tweaks the define so it defaults to using the Vista+ APIs.
Based on the numbers from bpo-15038, we should expect to see a significant improvement in speed.
MojoVampire commentedon Mar 22, 2017
Note: Beyond turning on the new primitives by default, I also made a change to PyCOND_TIMEDWAIT. The original code looked wrong, in that:
Neither of these things is true AFAICT. SleepConditionVariableSRW returns non-zero when it's alerted prior to timing out. Otherwise, it returns 0, and you have to check GetLastError to distinguish timeout from critical failures. Documentation is here: https://msdn.microsoft.com/en-us/library/windows/desktop/ms686304(v=vs.85).aspx
It's subject to "spurious" wake-ups, but so is pthread_cond_timedwait and the pthreads code doesn't use an "all timedwaits time out".
MojoVampire commentedon Mar 22, 2017
Hmm... I was only running the threading related tests originally, and they passed, but it looks like this causes problems with multiprocessing (test_multiprocessing_spawn and test_concurrent_futures both stall out forever with this change, with or without my fix to the PyCOND_TIMEDWAIT function). I'll investigate further, don't bother to review yet.
kristjanvalur commentedon Mar 22, 2017
Hi there.
Looking at the API docs today
(https://msdn.microsoft.com/en-us/library/windows/desktop/ms686304(v=vs.85).aspx)
it appears that the timeout case is documented. I'm fairly sure that it wasn't when I implemented it. There was a good reason for the "2" return code. The idea was: Either there was an error (-1) or we woke up. Since there are spurious wakeups and stolen wakeups, the predicate must be tested again anyway. The '2' return code would mean that the timeout condition should be tested by looking at some external clock.
Now, the api documentation is bad. Return value is BOOL.
Nonzero means "success" (whatever that means)
Failure means ´zero´
Timeout means FALSE and GetLastError() == ERROR_TIMEOUT
If memory serves, FALSE == 0 on windows....
Anyway, I've been out of this part of the code for sufficient time for details to be blurry.
My advise:
K
zooba commentedon May 24, 2017
I updated the PR to be mergeable and let the AppVeyor run work - https://ci.appveyor.com/project/python/cpython/build/3.7.0a0.2452
Unfortunately, there appear to be a number of regressions due to this. I'm not going to have time right now to work through them myself, but if someone else can it would be great to resolve these and get the change in.
vstinner commentedon Jan 18, 2018
FYI I proposed a PR which removes the emulation of condition variable:
#5231
This PR removes code specific to Windows Vista: see bpo-32592.
zooba commentedon Jan 15, 2019
On bpo-35562 Jeff posted a deeper analysis of the issue in TIMEDWAIT. That will need fixing along with the other regressions before we can enable these.
MojoVampire commentedon Jan 15, 2019
I assume you meant bpo-35662 (based on the superseder note in the history).
zooba commentedon Feb 21, 2019
Yes indeed. I am apparently massively dyslexic when it comes to copying issue numbers into the bpo comment field :)
Meanwhile, over on bpo-35662, Jeff has a fix for at least one of the regressions.