Repository navigation
Closed
Labels
3.7 (EOL)end of lifeend of life3.8 (EOL)end of lifeend of life3.9 (EOL)end of lifeend of lifetestsTests in the Lib/test dirTests in the Lib/test dirtype-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2019-10-09.15:06:10.652>
created_at = <Date 2019-07-09.19:34:22.877>
labels = ['3.8', '3.7', 'tests', '3.9', 'type-crash']
title = 'Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS'
updated_at = <Date 2020-04-14.23:05:24.157>
user = 'https://github.com/nanjekyejoannah'bugs.python.org fields:
activity = <Date 2020-04-14.23:05:24.157>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-10-09.15:06:10.652>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2019-07-09.19:34:22.877>
creator = 'nanjekyejoannah'
dependencies = []
files = ['48555', '48556']
hgrepos = []
issue_num = 37531
keywords = ['patch']
message_count = 51.0
messages = ['347576', '349689', '349703', '349704', '349708', '350009', '350012', '350014', '350017', '350020', '350023', '350044', '350057', '350058', '350060', '350061', '350132', '350181', '350183', '350191', '350233', '350246', '350272', '350329', '350578', '351399', '351517', '351713', '351714', '351716', '351815', '352273', '352619', '352632', '352633', '352674', '352679', '352726', '352730', '352796', '352804', '352806', '354078', '354088', '354223', '354270', '354272', '354277', '354817', '366410', '366459']
nosy_count = 7.0
nosy_names = ['vstinner', 'jkloth', 'jeremy.kloth', 'koobs', 'eryksun', 'miss-islington', 'nanjekyejoannah']
pr_nums = ['14679', '15072', '15279', '15280', '15345', '15419', '15871', '15874', '16220', '16223', '16224', '16247', '16282', '16285', '16289', '16659', '16676', '16677', '16827']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue37531'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']Metadata
Metadata
Assignees
Labels
3.7 (EOL)end of lifeend of life3.8 (EOL)end of lifeend of life3.9 (EOL)end of lifeend of lifetestsTests in the Lib/test dirTests in the Lib/test dirtype-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump
Activity
nanjekyejoannah commentedon Jul 9, 2019
Reported by Victor Stinner.
regrtest has a --timeout parameter which calls
faulthandler.dump_traceback_later(timeout), but sometimes it's not
enough to kill a test. regrtest should kill a worker process if it
runs longer than --timeout seconds.
25 hours whereas regrtest was run using --timeout 900
https://mail.python.org/pipermail/python-buildbots/2019-June/000285.html
vstinner commentedon Aug 14, 2019
New changeset b0c8369 by Victor Stinner in branch 'master':
bpo-37531: Fix regrtest timeout for subprocesses (GH-15072)
b0c8369
vstinner commentedon Aug 14, 2019
New changeset 93bee6a by Victor Stinner in branch '3.7':
[3.7] bpo-37531: Fix regrtest timeout for subprocesses (GH-15072) (GH-15280)
93bee6a
vstinner commentedon Aug 14, 2019
New changeset d85c567 by Victor Stinner in branch '3.8':
[3.8] bpo-37531: Fix regrtest timeout for subprocesses (GH-15072) (GH-15279)
d85c567
vstinner commentedon Aug 14, 2019
Thanks Joannah! That's a nice enhancement which should help to stabilize buildbots even more.
The next step is bpo-37711: "regrtest: re-run failed tests in subprocesses".
jkloth commentedon Aug 20, 2019
It seems that an issue still exists.
https://buildbot.python.org/all/#/builders/130/builds/1050
has been running for nearly 9 hours at this point. I can leave it "stuck" if there is some diagnostics that would be beneficial. But I would prefer not to have this impeding other testing for too long.
vstinner commentedon Aug 20, 2019
I just tested the 3.7 branch manually: regrtest works as expected.
I'm using this local changes:
Then I ran:
vstinner@apu$ ./python -m test -j2 test_os -m test_access --timeout=3
Run tests in parallel using 2 child processes
0:00:04 load avg: 0.59 [1/1/1] test_os timed out (4 sec 512 ms)
== Tests result: FAILURE ==
1 test failed:
test_os
Total duration: 4 sec 545 ms
Tests result: FAILURE
As you can see, test_os is killed after 4.5 seconds, as expected (it uses timeout x 1.5, since faulthandler is supposed to be first watchdog with timeout seconds).
--
Maybe something is wrong with test_concurrent_futures on Windows. Maybe subprocess.Popen.kill() is unable to kill the process immediately for an unknown reason?
Or maybe subprocess.Popen.communicate(timeout=self.timeout) didn't fail with TimeoutExpired?
Sadly, libregrtest is quite when it kills a worker process because of timeout, since the following output should be enough to understand what happened. But here the worker was not killed, or failed to be killed, or something else...
vstinner commentedon Aug 20, 2019
Can you check the process hierarchy? I would like to know how many worker processes are still running under the main regrtest process. I expect to see exactly one. I don't know how to investigate more on such issue on Windows.
jeremykloth commentedon Aug 20, 2019
There are no child processes of the main test process. There are 5
orphaned processes from multiprocessing. Their parent process is gone
(assuming the parent_pid argument is correct). There is also an
orphaned typeperf.exe process, probably also from the killed process.
30 remaining items
vstinner commentedon Sep 18, 2019
It seems like the new regrtest design doesn't work as I expected.
Calling popen.kill() + popen.stdout.close() + popen.stderr.close() from a thread B does not always interrupt popen.communicate() in thread A.
See https://bugs.python.org/issue38207#msg352729 for an example on Linux where communicate() still blocks until all child processes complete.
I see different options:
Revert changes to restore regrtest old design which didn't have these new issues
Find a way to fix bpo-38207 on all platforms: be able to interrupt communicate() as soon as the process is killed and/or when all pipes are closed.
Call communicate() with shorter timeout to workaround the blocking communicate() issue (bpo-38207).
Maybe experiment asyncio which supports asynchronous subprocess.
asyncio subprocess uses overlapped operations which can be cancelled. So maybe it isn't affected by bpo-38207.
vstinner commentedon Sep 19, 2019
New changeset b2dd2dd by Victor Stinner in branch 'master':
bpo-37531: Skip test_regrtest.test_multiprocessing_timeout() on all platforms (GH-16282)
b2dd2dd
vstinner commentedon Sep 19, 2019
New changeset fb7746d by Victor Stinner in branch '3.8':
bpo-37531: sync regrtest with master branch (GH-16285)
fb7746d
vstinner commentedon Sep 19, 2019
New changeset 5e1400a by Victor Stinner in branch '3.7':
bpo-37531: sync regrtest with master branch (GH-16285) (GH-16289)
5e1400a
vstinner commentedon Oct 7, 2019
Oh, regrtest main process was killed by faulthandler watchdog: it was blocked in _get_result() for longer than 1 minute.
Extract of _get_result():
self.ns.timeout is supposed to be 1500.
I don't understand why regrtest got a timeout of 60 seconds?
https://buildbot.python.org/all/#/builders/168/builds/1583
LD_LIBRARY_PATH=/usr/home/buildbot/python/3.x.koobs-freebsd-current/build ./python ./Tools/scripts/run_tests.py -j 1 -u all -W --slowest --fail-env-changed --timeout=1500 -j2 -j4
== CPython 3.9.0a0 (heads/master:ed8efd8e2c, Oct 7 2019, 21:38:27) [Clang 8.0.1 (tags/RELEASE_801/final 366581)]
== FreeBSD-13.0-CURRENT-amd64-64bit-ELF little-endian
== cwd: /usr/home/buildbot/python/3.x.koobs-freebsd-current/build/build/test_python_67537
== CPU count: 4
== encodings: locale=UTF-8, FS=utf-8
Using random seed 1307598
0:00:00 load avg: 3.41 Run tests in parallel using 4 child processes
(...)
0:00:11 load avg: 3.55 [ 12/419] test_extcall passed
0:00:13 load avg: 3.55 [ 13/419] test_pydoc passed
0:00:14 load avg: 3.55 [ 14/419] test_devpoll skipped
test_devpoll skipped -- test works only on Solaris OS family
0:00:15 load avg: 3.55 [ 15/419] test_class passed
0:00:17 load avg: 3.50 [ 16/419] test_ntpath passed
Timeout (0:01:00)!
Thread 0x0000000802d30c00 (most recent call first):
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/selectors.py", line 415 in select
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/subprocess.py", line 1964 in _communicate
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/subprocess.py", line 1113 in communicate
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 199 in _run_process
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 228 in _runtest
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 265 in run
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/threading.py", line 944 in _bootstrap_inner
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/threading.py", line 902 in _bootstrap
Thread 0x0000000802d2fd00 (most recent call first):
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/selectors.py", line 415 in select
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/subprocess.py", line 1964 in _communicate
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/subprocess.py", line 1113 in communicate
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 199 in _run_process
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 228 in _runtest
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 265 in run
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/threading.py", line 944 in _bootstrap_inner
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/threading.py", line 902 in _bootstrap
Thread 0x0000000802d2f800 (most recent call first):
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/selectors.py", line 415 in select
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/subprocess.py", line 1964 in _communicate
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/subprocess.py", line 1113 in communicate
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 199 in _run_process
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 228 in _runtest
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 265 in run
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/threading.py", line 944 in _bootstrap_inner
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/threading.py", line 902 in _bootstrap
Thread 0x0000000802d30200 (most recent call first):
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/selectors.py", line 415 in select
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/subprocess.py", line 1964 in _communicate
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/subprocess.py", line 1113 in communicate
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 199 in _run_process
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 228 in _runtest
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 265 in run
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/threading.py", line 944 in _bootstrap_inner
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/threading.py", line 902 in _bootstrap
Thread 0x0000000800acc000 (most recent call first):
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/threading.py", line 307 in wait
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/queue.py", line 179 in get
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 362 in _get_result
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 413 in run_tests
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 434 in run_tests_multiprocess
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/main.py", line 513 in run_tests
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/main.py", line 687 in _main
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/main.py", line 634 in main
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/main.py", line 712 in main
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/main.py", line 2 in <module>
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/runpy.py", line 85 in _run_code
File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/runpy.py", line 192 in _run_module_as_main
*** Error code 1
vstinner commentedon Oct 7, 2019
Oh, that comes from "timeout = PROGRESS_UPDATE" with "PROGRESS_UPDATE = 30.0".
But it means that the main process was stuck for longer than 60 seconds!?
The main process is supposed to write an update "running: ..." every 30 seconds.
This buildbot is very slow. Maybe 30 and 60 seconds limits are too low?
vstinner commentedon Oct 8, 2019
New changeset 0ec618a by Victor Stinner in branch 'master':
bpo-37531: regrtest ignores output on timeout (GH-16659)
0ec618a
miss-islington commentedon Oct 9, 2019
New changeset 598bfa4 by Miss Islington (bot) in branch '3.7':
bpo-37531: regrtest ignores output on timeout (GH-16659)
598bfa4
miss-islington commentedon Oct 9, 2019
New changeset 359a197 by Miss Islington (bot) in branch '3.8':
bpo-37531: regrtest ignores output on timeout (GH-16659)
359a197
vstinner commentedon Oct 9, 2019
The initial issue has been fixed in branches 3.7, 3.8 and master. I close the issue. Thanks to everybody who help to make these changes possible and helped to debug regressions.
It's not perfect, regrtest has to workaround bpo-38207 bug, but at least regrtest should no longer runs for several hours or days.
The main regrtest process now kills a worker process running longer than timeout seconds. In practice, it's timeout x 1.5, to give time to the faulthandler watchdog to kill the process.
If regrtest hits bpo-38207 bug, regrtest takes 2 min 30 sec to exit in the worst case.
I changed this timeout from 1 minute to 5 minutes.
I propose to continue the discussion in bpo-38207 "subprocess: On Windows, Popen.kill() + Popen.communicate() is blocking until all processes using the pipe close the pipe".
I implemented workarounds in regrtest so regrtest doesn't block *forever*: it only waits 30 seconds until a worker thread completes, and I added a timeout of 2 minutes on Python exit of the main regrtest process to prevent blocking in threading._shutdown(), because of the blocked threads.
I didn't check on my Windows VM if I have the issue. If someone sees more than one typeperf.exe process, please open a separated issue.
Note: typeperf.exe is the correct spelling (typepref.exe is a typo ;-)).
vstinner commentedon Oct 16, 2019
New changeset a661392 by Victor Stinner in branch 'master':
bpo-37531: regrtest now catchs ProcessLookupError (GH-16827)
a661392
vstinner commentedon Apr 14, 2020
New changeset 67b8a1f by Victor Stinner in branch '3.8':
[3.8] Update libregrtest from master (GH-19516)
67b8a1f
vstinner commentedon Apr 14, 2020
New changeset b894b66 by Victor Stinner in branch '3.7':
Update libregrtest from master (GH-19517)
b894b66