You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Python on Windows disables all C runtime library assertions #49054
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee=Noneclosed_at=<Date2010-07-19.23:20:40.492>created_at=<Date2009-01-02.04:09:39.306>labels= ['type-bug', 'OS-windows']
title='Python on Windows disables all C runtime library assertions'updated_at=<Date2010-07-19.23:20:40.490>user='https://github.com/mhammond'
#if defined _MSC_VER&&_MSC_VER >= 1400&& defined(__STDC_SECURE_LIB__)
...
/* turn off assertions in debug mode */prevCrtReportMode=_CrtSetReportMode(_CRT_ASSERT, 0);
#endif
Does exactly what the comment says it does - disables all assertions.
It disables *all* CRT assertions in the process, which includes some
very useful ones related to memory corruption, and all 'assert' and
'ASSERT' statements in all Python extension modules.
The change was introduced in:
"""
r46894 | kristjan.jonsson | 2006-06-13 01:45:12 +1000 (Tue, 13 Jun 2006)
| 2 lines
Fix the CRT argument error handling for VisualStudio .NET 2005. Install
a CRT error handler and disable the assertion for debug builds. This
causes CRT to set errno to EINVAL.
This update fixes crash cases in the test suite where the default CRT
error handler would cause process exit.
"""
which seems like a very large hammer to be using (ie, the problem
causing the assertions to blow should probably have been fixed). I'd
like to remove these 2 lines. Any objections, or should I upload the
trivial patch?
Instead of a gentle "OSError: [Errno 9] Bad file descriptor"
kristjanvalur commented on Jan 2, 2009
kristjanvalurmannequin
Mannequin
More actions
Yes, this is the biggest reason why this is required.
This, and and os.closerange().
I have browsed the CRT source and I find no way to determine if a fd is actually valid, before calling any of the functions that check for it.
Reintroducing the assertion will prevent a debug build python from passing the testsuite.
Much as I think this is useful (turning on the crt tests helped me find a couple of python bugs recently) it unfortunately needs to be off because of os.close(). All other troublesome cases (strftime(), open()) have workarounds.
Kristján
kristjanvalur commented on Jan 2, 2009
kristjanvalurmannequin
Mannequin
More actions
Btw,
It occurred to me that a more gentle way to do this is to disable the functionality only around close().
This is far less intrusive, for example, in an embedded environment.
I must confess that I had completely forgototten about the disabling of the assertion and only stumbled on this recently when I was trying to find out why none of my _ASSERTS in eve worked! Making wholesale CRT settings certainly should be avoided.
So, as I said, we could very well avoid this by just wrapping a few instances of close() with a CrtSetErrorHandler calls and friends. With the caveat that these are not threadsafe, that is, it is a process global setting for the CRT. But toggling it during a limited scope surely is less intrusive than modifying it for the entire process permanently.
It would be interesting to know which tests actually fail. If the tests
are explicitly checking a bad fd, then IMO it makes more sense for that
test to simply be avoided in debug builds and nothing of value would be
left untested. OTOH, I'd also be happy with just disabling them around
the 2 functions known to cause issues - anything other than leaving
assertions globally disabled for the process!
I guess another option is to expose it via msvcrt and let the test
themselves disable it?
kristjanvalur commented on Jan 2, 2009
kristjanvalurmannequin
Mannequin
More actions
Ok, I'll prepare a patch to the trunk. I've had some experience with this and should have the code lying around somewhere. I'll let you know once I´ve created an issue.
K
loewis commented on Jan 2, 2009
loewismannequin
Mannequin
More actions
As long as the CRT contains bogus assertions (that violate standard C),
I think Python rightfully should continue to disable assertions. If
applications desire assertions, they should explicitly turn them on
(provided there is an API to do so).
kristjanvalur commented on Jan 2, 2009
kristjanvalurmannequin
Mannequin
More actions
We've had this discussion before, you know, 2006.
MS is asserting on preconditions that are undefined by the standard.
As such, they are not in violation. In fact, it is foolish by us to invoke undefined behaviour, because it may, proverbially, result in the formatting of the hard drive :)
K
loewis commented on Jan 2, 2009
loewismannequin
Mannequin
More actions
We've had this discussion before, you know, 2006.
MS is asserting on preconditions that are undefined by the standard.
As such, they are not in violation.
I remember, and you were already wrong back then :-)
MS asserts (in winsig.c) that the signal number in signal() is one
of the explicit list of supported or ignored signals; if it isn't,
it aborts. This is in violation to ISO C, 7.14.1.1p8, which specifies
that SIG_ERR must be returned and errno set.
It is true that Python *also* invokes undefined behavior in certain
cases; contributions to eliminate these are welcome.
Can anyone point me at a test that failed in this case? A quick look
over winsig.c shows no asserts. I didn't compare the vs2005 crt source
though, so its highly likely I just missed it...
On the broader point, it could be argued that if it is the apps
responsibility to re-enable assertions, it is also the apps
responsibility to disable them in the first place. If other libraries
on the system attempted the same as Python, it would obviously be
undefined if Python's attempt had any actual affect - the last called
would win. As we found with the old locale() issues, we really can't
rely on Python setting a global shared state to keep things nice. From
a pragmatic POV, if we know the MS crt asserts on certain constants
signal numbers, can't we just avoid passing those numbers?
loewis commented on Jan 2, 2009
loewismannequin
Mannequin
More actions
Can anyone point me at a test that failed in this case?
See bpo-1350409 (and subsequently bpo-1167262 and bpo-1311784). Python
fails to start when assertions are turned on (haven't tested for
VS 2008, though).
If you were looking for a test case that fails specifically - there
might none. However, try Amaury's example.
On the broader point, it could be argued that if it is the apps
responsibility to re-enable assertions, it is also the apps
responsibility to disable them in the first place.
Ideally, assertions should be enabled in debug builds, and disabled
in release builds, and there should be no runtime configuration.
Unfortunately, Microsoft chose to add bogus assertions, so this general
rule is useless within the context of MSC.
(it's not just that the signal issue is a standards violation. Assuming
that _close apparently follows POSIX close, it should always set
errno to EBADF for a double-close, rather than raising an assertion. It
is conforming in release mode, but loses conformance in debug mode)
From
a pragmatic POV, if we know the MS crt asserts on certain constants
signal numbers, can't we just avoid passing those numbers?
It's the other way 'round: it rejects all but a list of known signals.
Still, it would be possible to hack around - one such hack is currently
implemented.
kristjanvalur commented on Jan 13, 2009
kristjanvalurmannequin
Mannequin
More actions
I submit to you a patch to the trunk which narrows down the problem. In
stead of throwing the switch for the whole process, we selectively
disable the hard error checking for those two cases that require it:
The file open mode string (unless we want to reimplement the crt mode
string parser) and the posix file descriptor methods. The remaining
case, strftime, we fix with a proper argument checking function.
Yesterday I put in some tests in to the testsuite to excercise this
functionality (revision 68547)
36 remaining items
kristjanvalur commented on Feb 4, 2009
kristjanvalurmannequin
Mannequin
More actions
Checked in r69268, to check strftime() and fopen()
ocean-city commented on Feb 4, 2009
ocean-citymannequin
Mannequin
More actions
Here is the patch implementing msg81093. (check_fd.patch)
I tested this on VC6. (test_os passed) I hope this also works on VC9 as
well.
kristjanvalur commented on Feb 6, 2009
kristjanvalurmannequin
Mannequin
More actions
I've taken the patch from Hirokazu and enhanced it:
it needed work to function with Visual Studio 2008
It now exposes a function so that _fileio.c can make use of it too.
Turned off the CRT manipulation in exceptions.c
Fixed minor problems, e.g. with dup2
Added comments
A VS2008 compilation runs the testsuite just fine. Maybe Hirozaku can
test this with his VS2005?
ocean-city commented on Feb 6, 2009
ocean-citymannequin
Mannequin
More actions
Sorry, I don't have VS2005.
By the way, _PyVerify_fd seems to be required to VC6 too. Because
fdopen(fd >= _NHANDLE_) crashes on debug build and fdopen(bad fd <
_NHANDLE_) won't set errno to EBADF.
kristjanvalur commented on Feb 6, 2009
kristjanvalurmannequin
Mannequin
More actions
I see. I had thought your code was for VS2005 (VC8)
I have changed the patch to work with VS2005 as well.
I don't think we need to worry about VS2003 so much, as I don't think
it is an officially supported compiler for the current python
versions. Also, the problem with close() et al only started when we
ported the code to Visual Studio 2005, and this is the version of
VisualStudio that starts to make assertions about its file descriptors.
Anyway, I upload an updated version of the patch, which copes with
VS2005 as well as VS2008
ocean-city commented on Feb 6, 2009
ocean-citymannequin
Mannequin
More actions
I agree. Please focus on _MSC_VER >= 1400. I'll post new issue about VC6
after this issue will be solved.
loewis commented on Feb 6, 2009
loewismannequin
Mannequin
More actions
I think the comment (an invalid fd would be a C program bug)
misrepresents the facts. Please don't check it in as-is. An invalid file
descriptor is *not* assertable. The authority on file descriptors, the
POSIX standard, specifies for, say, write(2)
[...] Otherwise, -1 shall be returned and errno set to indicate the error.
[...] [EBADF] The fildes argument is not a valid file descriptor open
for writing.
This is consistent throughout the entire API: the behavior for invalid
file descriptors is *not* undefined, and not even
implementation-defined. If the CRT would conform to the relevant,
long-existing specifications it tries to emulate, it would just set
errno to EBADF, and be done.
There should be a global declaration of _Py_verify_fd. I would also
suggest that it becomes a constant macro unless _MSC_VER is defined.
kristjanvalur commented on Feb 9, 2009
kristjanvalurmannequin
Mannequin
More actions
Ok, how about this comment, Martin?
/* Microsoft CRT in VS2005 and higher will verify that a filehandle is
valid and throw an assertion if it isn't.
Normally, an invalid fd is likely to be a C program error and
therefore
an assertion can be useful, but it does contradict the POSIX standard
which for write(2) states:
"Otherwise, -1 shall be returned and errno set to indicate the
error."
"[EBADF] The fildes argument is not a valid file descriptor open
for
writing."
Furthermore, python allows the user to enter any old integer
as a fd and should merely raise a python exception on error.
The Microsoft CRT doesn't provide an official way to check for the
validity of a file descriptor, but we can emulate its internal
behaviour
by using the exported __pinfo data member and knowledge of the
internal structures involved.
The structures below must be updated for each version of visual
studio
according to the file internal.h in the CRT source, until MS comes
up with a less hacky way to do this.
(all of this is to avoid globally modifying the CRT behaviour using
_set_invalid_parameter_handler() and _CrtSetReportMode())
*/
Also, I've added the following to fileobject.h, not coming up with a
better place:
#if defined _MSC_VER&&_MSC_VER >= 1400/* A routine to check if a file descriptor is valid on Windows. Returns 0 * and sets errno to EBADF if it isn't. This is to avoid Assertions * from various functions in the Windows CRT beginning with * Visual Studio 2005 */int_PyVerify_fd(intfd);
#else#define_PyVerify_fd(A) (1) /* dummy */#endif
Activity
mhammond commentedon Jan 2, 2009
This block in exceptions.c:
Does exactly what the comment says it does - disables all assertions.
It disables *all* CRT assertions in the process, which includes some
very useful ones related to memory corruption, and all 'assert' and
'ASSERT' statements in all Python extension modules.
The change was introduced in:
"""
r46894 | kristjan.jonsson | 2006-06-13 01:45:12 +1000 (Tue, 13 Jun 2006)
| 2 lines
Fix the CRT argument error handling for VisualStudio .NET 2005. Install
a CRT error handler and disable the assertion for debug builds. This
causes CRT to set errno to EINVAL.
This update fixes crash cases in the test suite where the default CRT
error handler would cause process exit.
"""
which seems like a very large hammer to be using (ie, the problem
causing the assertions to blow should probably have been fixed). I'd
like to remove these 2 lines. Any objections, or should I upload the
trivial patch?
amauryfa commentedon Jan 2, 2009
But then you get a nasty pop-up window on code like this:
Instead of a gentle "OSError: [Errno 9] Bad file descriptor"
kristjanvalur commentedon Jan 2, 2009
Yes, this is the biggest reason why this is required.
This, and and os.closerange().
I have browsed the CRT source and I find no way to determine if a fd is actually valid, before calling any of the functions that check for it.
Reintroducing the assertion will prevent a debug build python from passing the testsuite.
Much as I think this is useful (turning on the crt tests helped me find a couple of python bugs recently) it unfortunately needs to be off because of os.close(). All other troublesome cases (strftime(), open()) have workarounds.
Kristján
kristjanvalur commentedon Jan 2, 2009
Btw,
It occurred to me that a more gentle way to do this is to disable the functionality only around close().
This is far less intrusive, for example, in an embedded environment.
I must confess that I had completely forgototten about the disabling of the assertion and only stumbled on this recently when I was trying to find out why none of my _ASSERTS in eve worked! Making wholesale CRT settings certainly should be avoided.
So, as I said, we could very well avoid this by just wrapping a few instances of close() with a CrtSetErrorHandler calls and friends. With the caveat that these are not threadsafe, that is, it is a process global setting for the CRT. But toggling it during a limited scope surely is less intrusive than modifying it for the entire process permanently.
Kristján
mhammond commentedon Jan 2, 2009
It would be interesting to know which tests actually fail. If the tests
are explicitly checking a bad fd, then IMO it makes more sense for that
test to simply be avoided in debug builds and nothing of value would be
left untested. OTOH, I'd also be happy with just disabling them around
the 2 functions known to cause issues - anything other than leaving
assertions globally disabled for the process!
mhammond commentedon Jan 2, 2009
I guess another option is to expose it via msvcrt and let the test
themselves disable it?
kristjanvalur commentedon Jan 2, 2009
Ok, I'll prepare a patch to the trunk. I've had some experience with this and should have the code lying around somewhere. I'll let you know once I´ve created an issue.
K
loewis commentedon Jan 2, 2009
As long as the CRT contains bogus assertions (that violate standard C),
I think Python rightfully should continue to disable assertions. If
applications desire assertions, they should explicitly turn them on
(provided there is an API to do so).
kristjanvalur commentedon Jan 2, 2009
We've had this discussion before, you know, 2006.
MS is asserting on preconditions that are undefined by the standard.
As such, they are not in violation. In fact, it is foolish by us to invoke undefined behaviour, because it may, proverbially, result in the formatting of the hard drive :)
K
loewis commentedon Jan 2, 2009
I remember, and you were already wrong back then :-)
MS asserts (in winsig.c) that the signal number in signal() is one
of the explicit list of supported or ignored signals; if it isn't,
it aborts. This is in violation to ISO C, 7.14.1.1p8, which specifies
that SIG_ERR must be returned and errno set.
It is true that Python *also* invokes undefined behavior in certain
cases; contributions to eliminate these are welcome.
mhammond commentedon Jan 2, 2009
Can anyone point me at a test that failed in this case? A quick look
over winsig.c shows no asserts. I didn't compare the vs2005 crt source
though, so its highly likely I just missed it...
On the broader point, it could be argued that if it is the apps
responsibility to re-enable assertions, it is also the apps
responsibility to disable them in the first place. If other libraries
on the system attempted the same as Python, it would obviously be
undefined if Python's attempt had any actual affect - the last called
would win. As we found with the old locale() issues, we really can't
rely on Python setting a global shared state to keep things nice. From
a pragmatic POV, if we know the MS crt asserts on certain constants
signal numbers, can't we just avoid passing those numbers?
loewis commentedon Jan 2, 2009
See bpo-1350409 (and subsequently bpo-1167262 and bpo-1311784). Python
fails to start when assertions are turned on (haven't tested for
VS 2008, though).
If you were looking for a test case that fails specifically - there
might none. However, try Amaury's example.
Ideally, assertions should be enabled in debug builds, and disabled
in release builds, and there should be no runtime configuration.
Unfortunately, Microsoft chose to add bogus assertions, so this general
rule is useless within the context of MSC.
(it's not just that the signal issue is a standards violation. Assuming
that _close apparently follows POSIX close, it should always set
errno to EBADF for a double-close, rather than raising an assertion. It
is conforming in release mode, but loses conformance in debug mode)
It's the other way 'round: it rejects all but a list of known signals.
Still, it would be possible to hack around - one such hack is currently
implemented.
kristjanvalur commentedon Jan 13, 2009
I submit to you a patch to the trunk which narrows down the problem. In
stead of throwing the switch for the whole process, we selectively
disable the hard error checking for those two cases that require it:
The file open mode string (unless we want to reimplement the crt mode
string parser) and the posix file descriptor methods. The remaining
case, strftime, we fix with a proper argument checking function.
Yesterday I put in some tests in to the testsuite to excercise this
functionality (revision 68547)
36 remaining items
kristjanvalur commentedon Feb 4, 2009
Checked in r69268, to check strftime() and fopen()
ocean-city commentedon Feb 4, 2009
Here is the patch implementing msg81093. (check_fd.patch)
I tested this on VC6. (test_os passed) I hope this also works on VC9 as
well.
kristjanvalur commentedon Feb 6, 2009
I've taken the patch from Hirokazu and enhanced it:
A VS2008 compilation runs the testsuite just fine. Maybe Hirozaku can
test this with his VS2005?
ocean-city commentedon Feb 6, 2009
Sorry, I don't have VS2005.
By the way, _PyVerify_fd seems to be required to VC6 too. Because
fdopen(fd >= _NHANDLE_) crashes on debug build and fdopen(bad fd <
_NHANDLE_) won't set errno to EBADF.
kristjanvalur commentedon Feb 6, 2009
I see. I had thought your code was for VS2005 (VC8)
I have changed the patch to work with VS2005 as well.
I don't think we need to worry about VS2003 so much, as I don't think
it is an officially supported compiler for the current python
versions. Also, the problem with close() et al only started when we
ported the code to Visual Studio 2005, and this is the version of
VisualStudio that starts to make assertions about its file descriptors.
Anyway, I upload an updated version of the patch, which copes with
VS2005 as well as VS2008
ocean-city commentedon Feb 6, 2009
I agree. Please focus on _MSC_VER >= 1400. I'll post new issue about VC6
after this issue will be solved.
loewis commentedon Feb 6, 2009
I think the comment (an invalid fd would be a C program bug)
misrepresents the facts. Please don't check it in as-is. An invalid file
descriptor is *not* assertable. The authority on file descriptors, the
POSIX standard, specifies for, say, write(2)
[...] Otherwise, -1 shall be returned and errno set to indicate the error.
[...] [EBADF] The fildes argument is not a valid file descriptor open
for writing.
This is consistent throughout the entire API: the behavior for invalid
file descriptors is *not* undefined, and not even
implementation-defined. If the CRT would conform to the relevant,
long-existing specifications it tries to emulate, it would just set
errno to EBADF, and be done.
There should be a global declaration of _Py_verify_fd. I would also
suggest that it becomes a constant macro unless _MSC_VER is defined.
kristjanvalur commentedon Feb 9, 2009
Ok, how about this comment, Martin?
/* Microsoft CRT in VS2005 and higher will verify that a filehandle is
therefore
error."
for
behaviour
studio
*/
Also, I've added the following to fileobject.h, not coming up with a
better place:
loewis commentedon Feb 10, 2009
The revised patch looks fine to me, please apply.
kristjanvalur commentedon Feb 10, 2009
Committed r69495
ocean-city commentedon Feb 10, 2009
I opened new issue related to VC6 as bpo-5204.
amauryfa commentedon Jun 26, 2010
Why is this issue still open?
BreamoreBoy commentedon Jul 19, 2010
Closed as stated committed r69495.