Skip to content

Python on Windows disables all C runtime library assertions #49054

Closed

Description

@mhammond
BPO 4804
Nosy @loewis, @mhammond, @amauryfa, @kristjanvalur
Files
  • crterror.patch
  • crterror.patch
  • crterror.patch: a thread safe attempt at fine grained CRT error control
  • fopen.patch
  • crt_report_hook.patch
  • check_fd.patch
  • __pioinfo.patch
  • __pioinfo.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2010-07-19.23:20:40.492>
    created_at = <Date 2009-01-02.04:09:39.306>
    labels = ['type-bug', 'OS-windows']
    title = 'Python on Windows disables all C runtime library assertions'
    updated_at = <Date 2010-07-19.23:20:40.490>
    user = 'https://github.com/mhammond'

    bugs.python.org fields:

    activity = <Date 2010-07-19.23:20:40.490>
    actor = 'BreamoreBoy'
    assignee = 'none'
    closed = True
    closed_date = <Date 2010-07-19.23:20:40.492>
    closer = 'BreamoreBoy'
    components = ['Windows']
    creation = <Date 2009-01-02.04:09:39.306>
    creator = 'mhammond'
    dependencies = []
    files = ['12715', '12796', '12802', '12916', '12919', '12942', '12950', '12953']
    hgrepos = []
    issue_num = 4804
    keywords = ['patch', 'needs review']
    message_count = 62.0
    messages = ['78753', '78766', '78767', '78768', '78769', '78770', '78780', '78842', '78848', '78854', '78892', '78897', '79731', '79762', '79782', '79786', '79803', '80172', '80191', '80196', '80197', '80209', '80214', '80215', '80859', '80865', '80890', '80904', '80905', '80906', '80911', '80931', '80932', '80933', '80934', '80935', '80936', '80937', '80938', '80956', '80959', '80998', '81000', '81040', '81072', '81074', '81079', '81093', '81133', '81135', '81160', '81258', '81262', '81271', '81275', '81321', '81459', '81535', '81549', '81552', '108689', '110838']
    nosy_count = 6.0
    nosy_names = ['loewis', 'mhammond', 'amaury.forgeotdarc', 'kristjan.jonsson', 'ocean-city', 'BreamoreBoy']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'patch review'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue4804'
    versions = ['Python 2.6', 'Python 3.1', 'Python 2.7', 'Python 3.2']

    Activity

    1. mhammond commented on Jan 2, 2009

      @mhammond
      ContributorAuthor

      This block in exceptions.c:

      #if defined _MSC_VER && _MSC_VER >= 1400 && defined(__STDC_SECURE_LIB__)
      ...
          /* turn off assertions in debug mode */
          prevCrtReportMode = _CrtSetReportMode(_CRT_ASSERT, 0);
      #endif

      Does exactly what the comment says it does - disables all assertions.
      It disables *all* CRT assertions in the process, which includes some
      very useful ones related to memory corruption, and all 'assert' and
      'ASSERT' statements in all Python extension modules.

      The change was introduced in:
      """
      r46894 | kristjan.jonsson | 2006-06-13 01:45:12 +1000 (Tue, 13 Jun 2006)
      | 2 lines

      Fix the CRT argument error handling for VisualStudio .NET 2005. Install
      a CRT error handler and disable the assertion for debug builds. This
      causes CRT to set errno to EINVAL.
      This update fixes crash cases in the test suite where the default CRT
      error handler would cause process exit.
      """

      which seems like a very large hammer to be using (ie, the problem
      causing the assertions to blow should probably have been fixed). I'd
      like to remove these 2 lines. Any objections, or should I upload the
      trivial patch?

    2. amauryfa commented on Jan 2, 2009

      @amauryfa
      Contributor

      But then you get a nasty pop-up window on code like this:

      >> fd = os.open("foo", 0)
      >> os.close(fd)
      >> os.close(fd)

      Instead of a gentle "OSError: [Errno 9] Bad file descriptor"

    3. kristjanvalur commented on Jan 2, 2009

      kristjanvalurmannequin
      Mannequin

      Yes, this is the biggest reason why this is required.
      This, and and os.closerange().
      I have browsed the CRT source and I find no way to determine if a fd is actually valid, before calling any of the functions that check for it.
      Reintroducing the assertion will prevent a debug build python from passing the testsuite.
      Much as I think this is useful (turning on the crt tests helped me find a couple of python bugs recently) it unfortunately needs to be off because of os.close(). All other troublesome cases (strftime(), open()) have workarounds.

      Kristján

    4. kristjanvalur commented on Jan 2, 2009

      kristjanvalurmannequin
      Mannequin

      Btw,
      It occurred to me that a more gentle way to do this is to disable the functionality only around close().
      This is far less intrusive, for example, in an embedded environment.
      I must confess that I had completely forgototten about the disabling of the assertion and only stumbled on this recently when I was trying to find out why none of my _ASSERTS in eve worked! Making wholesale CRT settings certainly should be avoided.

      So, as I said, we could very well avoid this by just wrapping a few instances of close() with a CrtSetErrorHandler calls and friends. With the caveat that these are not threadsafe, that is, it is a process global setting for the CRT. But toggling it during a limited scope surely is less intrusive than modifying it for the entire process permanently.

      Kristján

    5. mhammond commented on Jan 2, 2009

      @mhammond
      ContributorAuthor

      It would be interesting to know which tests actually fail. If the tests
      are explicitly checking a bad fd, then IMO it makes more sense for that
      test to simply be avoided in debug builds and nothing of value would be
      left untested. OTOH, I'd also be happy with just disabling them around
      the 2 functions known to cause issues - anything other than leaving
      assertions globally disabled for the process!

    6. mhammond commented on Jan 2, 2009

      @mhammond
      ContributorAuthor

      I guess another option is to expose it via msvcrt and let the test
      themselves disable it?

    7. kristjanvalur commented on Jan 2, 2009

      kristjanvalurmannequin
      Mannequin

      Ok, I'll prepare a patch to the trunk. I've had some experience with this and should have the code lying around somewhere. I'll let you know once I´ve created an issue.
      K

    8. loewis commented on Jan 2, 2009

      loewismannequin
      Mannequin

      As long as the CRT contains bogus assertions (that violate standard C),
      I think Python rightfully should continue to disable assertions. If
      applications desire assertions, they should explicitly turn them on
      (provided there is an API to do so).

    9. kristjanvalur commented on Jan 2, 2009

      kristjanvalurmannequin
      Mannequin

      We've had this discussion before, you know, 2006.
      MS is asserting on preconditions that are undefined by the standard.
      As such, they are not in violation. In fact, it is foolish by us to invoke undefined behaviour, because it may, proverbially, result in the formatting of the hard drive :)

      K

    10. loewis commented on Jan 2, 2009

      loewismannequin
      Mannequin

      We've had this discussion before, you know, 2006.
      MS is asserting on preconditions that are undefined by the standard.
      As such, they are not in violation.

      I remember, and you were already wrong back then :-)

      MS asserts (in winsig.c) that the signal number in signal() is one
      of the explicit list of supported or ignored signals; if it isn't,
      it aborts. This is in violation to ISO C, 7.14.1.1p8, which specifies
      that SIG_ERR must be returned and errno set.

      It is true that Python *also* invokes undefined behavior in certain
      cases; contributions to eliminate these are welcome.

    11. mhammond commented on Jan 2, 2009

      @mhammond
      ContributorAuthor

      Can anyone point me at a test that failed in this case? A quick look
      over winsig.c shows no asserts. I didn't compare the vs2005 crt source
      though, so its highly likely I just missed it...

      On the broader point, it could be argued that if it is the apps
      responsibility to re-enable assertions, it is also the apps
      responsibility to disable them in the first place. If other libraries
      on the system attempted the same as Python, it would obviously be
      undefined if Python's attempt had any actual affect - the last called
      would win. As we found with the old locale() issues, we really can't
      rely on Python setting a global shared state to keep things nice. From
      a pragmatic POV, if we know the MS crt asserts on certain constants
      signal numbers, can't we just avoid passing those numbers?

    12. loewis commented on Jan 2, 2009

      loewismannequin
      Mannequin

      Can anyone point me at a test that failed in this case?

      See bpo-1350409 (and subsequently bpo-1167262 and bpo-1311784). Python
      fails to start when assertions are turned on (haven't tested for
      VS 2008, though).

      If you were looking for a test case that fails specifically - there
      might none. However, try Amaury's example.

      On the broader point, it could be argued that if it is the apps
      responsibility to re-enable assertions, it is also the apps
      responsibility to disable them in the first place.

      Ideally, assertions should be enabled in debug builds, and disabled
      in release builds, and there should be no runtime configuration.
      Unfortunately, Microsoft chose to add bogus assertions, so this general
      rule is useless within the context of MSC.

      (it's not just that the signal issue is a standards violation. Assuming
      that _close apparently follows POSIX close, it should always set
      errno to EBADF for a double-close, rather than raising an assertion. It
      is conforming in release mode, but loses conformance in debug mode)

      From
      a pragmatic POV, if we know the MS crt asserts on certain constants
      signal numbers, can't we just avoid passing those numbers?

      It's the other way 'round: it rejects all but a list of known signals.
      Still, it would be possible to hack around - one such hack is currently
      implemented.

    13. kristjanvalur commented on Jan 13, 2009

      kristjanvalurmannequin
      Mannequin

      I submit to you a patch to the trunk which narrows down the problem. In
      stead of throwing the switch for the whole process, we selectively
      disable the hard error checking for those two cases that require it:
      The file open mode string (unless we want to reimplement the crt mode
      string parser) and the posix file descriptor methods. The remaining
      case, strftime, we fix with a proper argument checking function.

      Yesterday I put in some tests in to the testsuite to excercise this
      functionality (revision 68547)

    14. 36 remaining items

    15. kristjanvalur commented on Feb 4, 2009

      kristjanvalurmannequin
      Mannequin

      Checked in r69268, to check strftime() and fopen()

    16. ocean-city commented on Feb 4, 2009

      ocean-citymannequin
      Mannequin

      Here is the patch implementing msg81093. (check_fd.patch)
      I tested this on VC6. (test_os passed) I hope this also works on VC9 as
      well.

    17. kristjanvalur commented on Feb 6, 2009

      kristjanvalurmannequin
      Mannequin

      I've taken the patch from Hirokazu and enhanced it:

      1. it needed work to function with Visual Studio 2008
      2. It now exposes a function so that _fileio.c can make use of it too.
      3. Turned off the CRT manipulation in exceptions.c
      4. Fixed minor problems, e.g. with dup2
      5. Added comments

      A VS2008 compilation runs the testsuite just fine. Maybe Hirozaku can
      test this with his VS2005?

    18. ocean-city commented on Feb 6, 2009

      ocean-citymannequin
      Mannequin

      Sorry, I don't have VS2005.

      By the way, _PyVerify_fd seems to be required to VC6 too. Because
      fdopen(fd >= _NHANDLE_) crashes on debug build and fdopen(bad fd <
      _NHANDLE_) won't set errno to EBADF.

    19. kristjanvalur commented on Feb 6, 2009

      kristjanvalurmannequin
      Mannequin

      I see. I had thought your code was for VS2005 (VC8)
      I have changed the patch to work with VS2005 as well.
      I don't think we need to worry about VS2003 so much, as I don't think
      it is an officially supported compiler for the current python
      versions. Also, the problem with close() et al only started when we
      ported the code to Visual Studio 2005, and this is the version of
      VisualStudio that starts to make assertions about its file descriptors.
      Anyway, I upload an updated version of the patch, which copes with
      VS2005 as well as VS2008

    20. ocean-city commented on Feb 6, 2009

      ocean-citymannequin
      Mannequin

      I agree. Please focus on _MSC_VER >= 1400. I'll post new issue about VC6
      after this issue will be solved.

    21. loewis commented on Feb 6, 2009

      loewismannequin
      Mannequin

      I think the comment (an invalid fd would be a C program bug)
      misrepresents the facts. Please don't check it in as-is. An invalid file
      descriptor is *not* assertable. The authority on file descriptors, the
      POSIX standard, specifies for, say, write(2)

      [...] Otherwise, -1 shall be returned and errno set to indicate the error.
      [...] [EBADF] The fildes argument is not a valid file descriptor open
      for writing.

      This is consistent throughout the entire API: the behavior for invalid
      file descriptors is *not* undefined, and not even
      implementation-defined. If the CRT would conform to the relevant,
      long-existing specifications it tries to emulate, it would just set
      errno to EBADF, and be done.

      There should be a global declaration of _Py_verify_fd. I would also
      suggest that it becomes a constant macro unless _MSC_VER is defined.

    22. kristjanvalur commented on Feb 9, 2009

      kristjanvalurmannequin
      Mannequin

      Ok, how about this comment, Martin?

      /* Microsoft CRT in VS2005 and higher will verify that a filehandle is

      • valid and throw an assertion if it isn't.
      • Normally, an invalid fd is likely to be a C program error and
        therefore
      • an assertion can be useful, but it does contradict the POSIX standard
      • which for write(2) states:
      • "Otherwise, -1 shall be returned and errno set to indicate the
        error."
      • "[EBADF] The fildes argument is not a valid file descriptor open
        for
      • writing."
        
      • Furthermore, python allows the user to enter any old integer
      • as a fd and should merely raise a python exception on error.
      • The Microsoft CRT doesn't provide an official way to check for the
      • validity of a file descriptor, but we can emulate its internal
        behaviour
      • by using the exported __pinfo data member and knowledge of the
      • internal structures involved.
      • The structures below must be updated for each version of visual
        studio
      • according to the file internal.h in the CRT source, until MS comes
      • up with a less hacky way to do this.
      • (all of this is to avoid globally modifying the CRT behaviour using
      • _set_invalid_parameter_handler() and _CrtSetReportMode())
        */

      Also, I've added the following to fileobject.h, not coming up with a
      better place:

      #if defined _MSC_VER && _MSC_VER >= 1400
      /* A routine to check if a file descriptor is valid on Windows.  
      Returns 0
       * and sets errno to EBADF if it isn't.  This is to avoid Assertions
       * from various functions in the Windows CRT beginning with
       * Visual Studio 2005
       */
      int _PyVerify_fd(int fd);
      #else
      #define _PyVerify_fd(A) (1) /* dummy */
      #endif
    23. loewis commented on Feb 10, 2009

      loewismannequin
      Mannequin

      The revised patch looks fine to me, please apply.

    24. kristjanvalur commented on Feb 10, 2009

      kristjanvalurmannequin
      Mannequin

      Committed r69495

    25. ocean-city commented on Feb 10, 2009

      ocean-citymannequin
      Mannequin

      I opened new issue related to VC6 as bpo-5204.

    26. amauryfa commented on Jun 26, 2010

      @amauryfa
      Contributor

      Why is this issue still open?

    27. BreamoreBoy commented on Jul 19, 2010

      BreamoreBoymannequin
      Mannequin

      Closed as stated committed r69495.

    28. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      OS-windowstype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions