Skip to content

Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS #81712

Closed

Description

@nanjekyejoannah
BPO 37531
Nosy @vstinner, @jkloth, @koobs, @eryksun, @miss-islington, @nanjekyejoannah
PRs
  • bpo-37531: Fix regrtest timeout for subprocesses #14679
  • bpo-37531: Fix regrtest timeout for subprocesses #15072
  • [3.8] bpo-37531: Fix regrtest timeout for subprocesses (GH-15072) #15279
  • [3.7] bpo-37531: Fix regrtest timeout for subprocesses (GH-15072) #15280
  • bpo-37531: Enhance regrtest multiprocess timeout #15345
  • bpo-37531: Fix regrtest _timedout() function on timeout #15419
  • [3.8] bpo-37531: Enhance regrtest multiprocess timeout (GH-15345) #15871
  • [3.7] bpo-37531: Enhance regrtest multiprocess timeout (GH-15345) #15874
  • bpo-37531: regrtest main process uses shorter timeout #16220
  • [3.7] bpo-37531: regrtest main process uses shorter timeout (GH-16220) #16223
  • [3.8] bpo-37531: regrtest main process uses shorter timeout (GH-16220) #16224
  • bpo-37531: Skip test_regrtest.test_multiprocessing_timeout() on Windows #16247
  • bpo-37531: Skip test_regrtest.test_multiprocessing_timeout() on all platforms #16282
  • [3.8] bpo-37531: sync regrtest with master branch #16285
  • [3.7] bpo-37531: sync regrtest with master branch (GH-16285) #16289
  • bpo-37531: regrtest ignores output on timeout #16659
  • [3.8] bpo-37531: regrtest ignores output on timeout (GH-16659) #16676
  • [3.7] bpo-37531: regrtest ignores output on timeout (GH-16659) #16677
  • bpo-37531: regrtest now catchs ProcessLookupError #16827
  • Files
  • kill_timeout.py
  • test_kill.py
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-10-09.15:06:10.652>
    created_at = <Date 2019-07-09.19:34:22.877>
    labels = ['3.8', '3.7', 'tests', '3.9', 'type-crash']
    title = 'Fix regrtest timeout for subprocesses: regrtest -jN --timeout=SECONDS'
    updated_at = <Date 2020-04-14.23:05:24.157>
    user = 'https://github.com/nanjekyejoannah'

    bugs.python.org fields:

    activity = <Date 2020-04-14.23:05:24.157>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-10-09.15:06:10.652>
    closer = 'vstinner'
    components = ['Tests']
    creation = <Date 2019-07-09.19:34:22.877>
    creator = 'nanjekyejoannah'
    dependencies = []
    files = ['48555', '48556']
    hgrepos = []
    issue_num = 37531
    keywords = ['patch']
    message_count = 51.0
    messages = ['347576', '349689', '349703', '349704', '349708', '350009', '350012', '350014', '350017', '350020', '350023', '350044', '350057', '350058', '350060', '350061', '350132', '350181', '350183', '350191', '350233', '350246', '350272', '350329', '350578', '351399', '351517', '351713', '351714', '351716', '351815', '352273', '352619', '352632', '352633', '352674', '352679', '352726', '352730', '352796', '352804', '352806', '354078', '354088', '354223', '354270', '354272', '354277', '354817', '366410', '366459']
    nosy_count = 7.0
    nosy_names = ['vstinner', 'jkloth', 'jeremy.kloth', 'koobs', 'eryksun', 'miss-islington', 'nanjekyejoannah']
    pr_nums = ['14679', '15072', '15279', '15280', '15345', '15419', '15871', '15874', '16220', '16223', '16224', '16247', '16282', '16285', '16289', '16659', '16676', '16677', '16827']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'crash'
    url = 'https://bugs.python.org/issue37531'
    versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

    Activity

    1. nanjekyejoannah commented on Jul 9, 2019

      @nanjekyejoannah
      ContributorAuthor

      Reported by Victor Stinner.

      regrtest has a --timeout parameter which calls
      faulthandler.dump_traceback_later(timeout), but sometimes it's not
      enough to kill a test. regrtest should kill a worker process if it
      runs longer than --timeout seconds.

    2. vstinner commented on Aug 14, 2019

      @vstinner
      Member

      New changeset b0c8369 by Victor Stinner in branch 'master':
      bpo-37531: Fix regrtest timeout for subprocesses (GH-15072)
      b0c8369

    3. vstinner commented on Aug 14, 2019

      @vstinner
      Member

      New changeset 93bee6a by Victor Stinner in branch '3.7':
      [3.7] bpo-37531: Fix regrtest timeout for subprocesses (GH-15072) (GH-15280)
      93bee6a

    4. vstinner commented on Aug 14, 2019

      @vstinner
      Member

      New changeset d85c567 by Victor Stinner in branch '3.8':
      [3.8] bpo-37531: Fix regrtest timeout for subprocesses (GH-15072) (GH-15279)
      d85c567

    5. vstinner commented on Aug 14, 2019

      @vstinner
      Member

      Thanks Joannah! That's a nice enhancement which should help to stabilize buildbots even more.

      The next step is bpo-37711: "regrtest: re-run failed tests in subprocesses".

    6. jkloth commented on Aug 20, 2019

      @jkloth
      Contributor

      It seems that an issue still exists.

      https://buildbot.python.org/all/#/builders/130/builds/1050

      has been running for nearly 9 hours at this point. I can leave it "stuck" if there is some diagnostics that would be beneficial. But I would prefer not to have this impeding other testing for too long.

    7. vstinner commented on Aug 20, 2019

      @vstinner
      Member

      I just tested the 3.7 branch manually: regrtest works as expected.

      I'm using this local changes:

      diff --git a/Lib/test/libregrtest/runtest.py b/Lib/test/libregrtest/runtest.py
      index e7dce180cb..7d996d6918 100644
      --- a/Lib/test/libregrtest/runtest.py
      +++ b/Lib/test/libregrtest/runtest.py
      @@ -119,7 +119,7 @@ def _runtest(ns, test_name):
       
           use_timeout = (ns.timeout is not None)
           if use_timeout:
      -        faulthandler.dump_traceback_later(ns.timeout, exit=True)
      +        pass #faulthandler.dump_traceback_later(ns.timeout, exit=True)
       
           start_time = time.perf_counter()
           try:
      diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
      index 8032da0530..e21399c840 100644
      --- a/Lib/test/test_os.py
      +++ b/Lib/test/test_os.py
      @@ -89,6 +89,7 @@ class FileTests(unittest.TestCase):
           tearDown = setUp
       
           def test_access(self):
      +        import time; time.sleep(3600)
               f = os.open(support.TESTFN, os.O_CREAT|os.O_RDWR)
               os.close(f)
               self.assertTrue(os.access(support.TESTFN, os.W_OK))

      Then I ran:

      vstinner@apu$ ./python -m test -j2 test_os -m test_access --timeout=3
      Run tests in parallel using 2 child processes
      0:00:04 load avg: 0.59 [1/1/1] test_os timed out (4 sec 512 ms)

      == Tests result: FAILURE ==

      1 test failed:
      test_os

      Total duration: 4 sec 545 ms
      Tests result: FAILURE

      As you can see, test_os is killed after 4.5 seconds, as expected (it uses timeout x 1.5, since faulthandler is supposed to be first watchdog with timeout seconds).

      --

      Maybe something is wrong with test_concurrent_futures on Windows. Maybe subprocess.Popen.kill() is unable to kill the process immediately for an unknown reason?

      Or maybe subprocess.Popen.communicate(timeout=self.timeout) didn't fail with TimeoutExpired?

      Sadly, libregrtest is quite when it kills a worker process because of timeout, since the following output should be enough to understand what happened. But here the worker was not killed, or failed to be killed, or something else...

    8. vstinner commented on Aug 20, 2019

      @vstinner
      Member

      I can leave it "stuck" if there is some diagnostics that would be beneficial.

      Can you check the process hierarchy? I would like to know how many worker processes are still running under the main regrtest process. I expect to see exactly one. I don't know how to investigate more on such issue on Windows.

    9. jeremykloth commented on Aug 20, 2019

      jeremyklothmannequin
      Mannequin

      Can you check the process hierarchy? I would like to know how many worker processes are still running under the main regrtest process. I expect to see exactly one. I don't know how to investigate more on such issue on Windows.

      There are no child processes of the main test process. There are 5
      orphaned processes from multiprocessing. Their parent process is gone
      (assuming the parent_pid argument is correct). There is also an
      orphaned typeperf.exe process, probably also from the killed process.

    10. 30 remaining items

    11. vstinner commented on Sep 18, 2019

      @vstinner
      Member

      It seems like the new regrtest design doesn't work as I expected.

      Calling popen.kill() + popen.stdout.close() + popen.stderr.close() from a thread B does not always interrupt popen.communicate() in thread A.

      See https://bugs.python.org/issue38207#msg352729 for an example on Linux where communicate() still blocks until all child processes complete.

      I see different options:

      • Revert changes to restore regrtest old design which didn't have these new issues

      • Find a way to fix bpo-38207 on all platforms: be able to interrupt communicate() as soon as the process is killed and/or when all pipes are closed.

      • Call communicate() with shorter timeout to workaround the blocking communicate() issue (bpo-38207).

      • Maybe experiment asyncio which supports asynchronous subprocess.

      asyncio subprocess uses overlapped operations which can be cancelled. So maybe it isn't affected by bpo-38207.

    12. vstinner commented on Sep 19, 2019

      @vstinner
      Member

      New changeset b2dd2dd by Victor Stinner in branch 'master':
      bpo-37531: Skip test_regrtest.test_multiprocessing_timeout() on all platforms (GH-16282)
      b2dd2dd

    13. vstinner commented on Sep 19, 2019

      @vstinner
      Member

      New changeset fb7746d by Victor Stinner in branch '3.8':
      bpo-37531: sync regrtest with master branch (GH-16285)
      fb7746d

    14. vstinner commented on Sep 19, 2019

      @vstinner
      Member

      New changeset 5e1400a by Victor Stinner in branch '3.7':
      bpo-37531: sync regrtest with master branch (GH-16285) (GH-16289)
      5e1400a

    15. vstinner commented on Oct 7, 2019

      @vstinner
      Member

      Oh, regrtest main process was killed by faulthandler watchdog: it was blocked in _get_result() for longer than 1 minute.

      Extract of _get_result():

              use_faulthandler = (self.ns.timeout is not None)
              timeout = PROGRESS_UPDATE
              while True:
                  if use_faulthandler:
                      faulthandler.dump_traceback_later(timeout * 2.0, exit=True)
              # wait for a thread
              try:
                  return self.output.get(timeout=timeout)
              except queue.Empty:
                  pass
      
                  # display progress
                  running = get_running(self.workers)
                  if running and not self.ns.pgo:
                      self.log('running: %s' % ', '.join(running))

      self.ns.timeout is supposed to be 1500.

      I don't understand why regrtest got a timeout of 60 seconds?

      https://buildbot.python.org/all/#/builders/168/builds/1583

      LD_LIBRARY_PATH=/usr/home/buildbot/python/3.x.koobs-freebsd-current/build ./python ./Tools/scripts/run_tests.py -j 1 -u all -W --slowest --fail-env-changed --timeout=1500 -j2 -j4
      == CPython 3.9.0a0 (heads/master:ed8efd8e2c, Oct 7 2019, 21:38:27) [Clang 8.0.1 (tags/RELEASE_801/final 366581)]
      == FreeBSD-13.0-CURRENT-amd64-64bit-ELF little-endian
      == cwd: /usr/home/buildbot/python/3.x.koobs-freebsd-current/build/build/test_python_67537
      == CPU count: 4
      == encodings: locale=UTF-8, FS=utf-8
      Using random seed 1307598
      0:00:00 load avg: 3.41 Run tests in parallel using 4 child processes
      (...)
      0:00:11 load avg: 3.55 [ 12/419] test_extcall passed
      0:00:13 load avg: 3.55 [ 13/419] test_pydoc passed
      0:00:14 load avg: 3.55 [ 14/419] test_devpoll skipped
      test_devpoll skipped -- test works only on Solaris OS family
      0:00:15 load avg: 3.55 [ 15/419] test_class passed
      0:00:17 load avg: 3.50 [ 16/419] test_ntpath passed
      Timeout (0:01:00)!
      Thread 0x0000000802d30c00 (most recent call first):
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/selectors.py", line 415 in select
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/subprocess.py", line 1964 in _communicate
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/subprocess.py", line 1113 in communicate
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 199 in _run_process
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 228 in _runtest
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 265 in run
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/threading.py", line 944 in _bootstrap_inner
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/threading.py", line 902 in _bootstrap

      Thread 0x0000000802d2fd00 (most recent call first):
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/selectors.py", line 415 in select
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/subprocess.py", line 1964 in _communicate
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/subprocess.py", line 1113 in communicate
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 199 in _run_process
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 228 in _runtest
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 265 in run
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/threading.py", line 944 in _bootstrap_inner
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/threading.py", line 902 in _bootstrap

      Thread 0x0000000802d2f800 (most recent call first):
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/selectors.py", line 415 in select
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/subprocess.py", line 1964 in _communicate
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/subprocess.py", line 1113 in communicate
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 199 in _run_process
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 228 in _runtest
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 265 in run
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/threading.py", line 944 in _bootstrap_inner
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/threading.py", line 902 in _bootstrap

      Thread 0x0000000802d30200 (most recent call first):
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/selectors.py", line 415 in select
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/subprocess.py", line 1964 in _communicate
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/subprocess.py", line 1113 in communicate
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 199 in _run_process
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 228 in _runtest
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 265 in run
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/threading.py", line 944 in _bootstrap_inner
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/threading.py", line 902 in _bootstrap

      Thread 0x0000000800acc000 (most recent call first):
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/threading.py", line 307 in wait
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/queue.py", line 179 in get
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 362 in _get_result
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 413 in run_tests
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/runtest_mp.py", line 434 in run_tests_multiprocess
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/main.py", line 513 in run_tests
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/main.py", line 687 in _main
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/main.py", line 634 in main
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/libregrtest/main.py", line 712 in main
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/main.py", line 2 in <module>
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/runpy.py", line 85 in _run_code
      File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/runpy.py", line 192 in _run_module_as_main
      *** Error code 1

    16. vstinner commented on Oct 7, 2019

      @vstinner
      Member

      I don't understand why regrtest got a timeout of 60 seconds?

      Oh, that comes from "timeout = PROGRESS_UPDATE" with "PROGRESS_UPDATE = 30.0".

      But it means that the main process was stuck for longer than 60 seconds!?

      The main process is supposed to write an update "running: ..." every 30 seconds.

      This buildbot is very slow. Maybe 30 and 60 seconds limits are too low?

    17. vstinner commented on Oct 8, 2019

      @vstinner
      Member

      New changeset 0ec618a by Victor Stinner in branch 'master':
      bpo-37531: regrtest ignores output on timeout (GH-16659)
      0ec618a

    18. miss-islington commented on Oct 9, 2019

      @miss-islington
      Contributor

      New changeset 598bfa4 by Miss Islington (bot) in branch '3.7':
      bpo-37531: regrtest ignores output on timeout (GH-16659)
      598bfa4

    19. miss-islington commented on Oct 9, 2019

      @miss-islington
      Contributor

      New changeset 359a197 by Miss Islington (bot) in branch '3.8':
      bpo-37531: regrtest ignores output on timeout (GH-16659)
      359a197

    20. vstinner commented on Oct 9, 2019

      @vstinner
      Member

      The initial issue has been fixed in branches 3.7, 3.8 and master. I close the issue. Thanks to everybody who help to make these changes possible and helped to debug regressions.

      It's not perfect, regrtest has to workaround bpo-38207 bug, but at least regrtest should no longer runs for several hours or days.

      regrtest should kill a worker process if it runs longer than --timeout seconds.

      The main regrtest process now kills a worker process running longer than timeout seconds. In practice, it's timeout x 1.5, to give time to the faulthandler watchdog to kill the process.

      If regrtest hits bpo-38207 bug, regrtest takes 2 min 30 sec to exit in the worst case.

      But it means that the main process was stuck for longer than 60 seconds!? The main process is supposed to write an update "running: ..." every 30 seconds. This buildbot is very slow. Maybe 30 and 60 seconds limits are too low?

      I changed this timeout from 1 minute to 5 minutes.

      Calling popen.kill() + popen.stdout.close() + popen.stderr.close() from a thread B does not always interrupt popen.communicate() in thread A.

      I propose to continue the discussion in bpo-38207 "subprocess: On Windows, Popen.kill() + Popen.communicate() is blocking until all processes using the pipe close the pipe".

      I implemented workarounds in regrtest so regrtest doesn't block *forever*: it only waits 30 seconds until a worker thread completes, and I added a timeout of 2 minutes on Python exit of the main regrtest process to prevent blocking in threading._shutdown(), because of the blocked threads.

      I don't understand why you get 2 typeperf.exe process: only the main process should spawn one. Worker processes don't need it and should not spawn it.

      I didn't check on my Windows VM if I have the issue. If someone sees more than one typeperf.exe process, please open a separated issue.

      Note: typeperf.exe is the correct spelling (typepref.exe is a typo ;-)).

    21. vstinner commented on Oct 16, 2019

      @vstinner
      Member

      New changeset a661392 by Victor Stinner in branch 'master':
      bpo-37531: regrtest now catchs ProcessLookupError (GH-16827)
      a661392

    22. vstinner commented on Apr 14, 2020

      @vstinner
      Member

      New changeset 67b8a1f by Victor Stinner in branch '3.8':
      [3.8] Update libregrtest from master (GH-19516)
      67b8a1f

    23. vstinner commented on Apr 14, 2020

      @vstinner
      Member

      New changeset b894b66 by Victor Stinner in branch '3.7':
      Update libregrtest from master (GH-19517)
      b894b66

    24. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.7 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifetestsTests in the Lib/test dirtype-crashA hard crash of the interpreter, possibly with a core dump

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions