You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test_warnings.CEnvironmentVariableTests.test_nonascii fails under an ascii terminal #52835
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee='https://github.com/vstinner'closed_at=<Date2010-09-17.23:41:02.436>created_at=<Date2010-05-01.11:05:18.503>labels= ['type-bug', 'tests']
title='test_warnings.CEnvironmentVariableTests.test_nonascii fails under an ascii terminal'updated_at=<Date2010-09-17.23:41:02.434>user='https://github.com/voidspace'
If you run test_warnings.py under an ascii terminal (at least on Mac OS X) then test_warnings.CEnvironmentVariableTests.test_nonascii fails. Perhaps the test should be skipped?
Traceback (most recent call last):
File "Lib/test/test_warnings.py", line 758, in <module>
test_main()
File "Lib/test/test_warnings.py", line 753, in test_main
PyEnvironmentVariableTests
File "/compile/python-trunk3/Lib/test/support.py", line 1053, in run_unittest
_run_suite(suite)
File "/compile/python-trunk3/Lib/test/support.py", line 1027, in _run_suite
result = runner.run(suite)
File "/compile/python-trunk3/Lib/unittest/runner.py", line 158, in run
result.printErrors()
File "/compile/python-trunk3/Lib/unittest/runner.py", line 109, in printErrorsself.printErrorList('FAIL', self.failures)
File "/compile/python-trunk3/Lib/unittest/runner.py", line 116, in printErrorListself.stream.writeln("%s"% err)
File "/compile/python-trunk3/Lib/unittest/runner.py", line 24, in writelnself.write(arg)
UnicodeEncodeError: 'ascii' codec can't encode character '\xf3' in position 121: ordinal not in range(128)
Looks like the traceback here actually comes from unittest trying to report the failure.
Victor has proposed a patch for the traceback problem for regrtest, I think. I haven't looked at it, but I wonder if there is something that can instead be done to make unittest work in cases like this when run in an ascii terminal. See bpo-8522.
What does bpo-8522 have to do with it - did you mean a different issue? In unittest it could catch the UnicodeEncodeError and write the ascii repr instead.
The bug only occurs on Mac OS X because file system encoding is hardcoded to UTF-8 on this OS and the test is skipped if the file system encoding is ASCII.
But the bug is not specific to test_warnings, as mentionned by R. David Murray: bpo-8533 is the real bug. test_warnings is just an example to reproduce it.
I just commited a fix for bpo-8533: it should fix also this issue.
Attached patch creates PySys_AddWarnOptionUnicode() function and decodes PYTHONWARNINGS variable with the file system encoding (as it is done for os.environ, to be consistent) instead of the locale encoding.
The patch only concerns the POSIX version. Windows is not affected by this issue (see below).
--
This issue occurs if the file system encoding is different than the locale encoding (especially if the locale encoding is ASCII). It only occurs on Mac OS because on this OS, the file system encoding is hardcoded to UTF-8 whereas the locale encoding... depends on the locale.
I reproduced the bug on Linux by hardcoding the file system encoding to UTF-8.
--
About the test output (UnicodeEncodeError: 'ascii' codec can't encode character '\xf3' ...): I think that Michael Foord executed directly Lib/test/test_warnings.py instead of using "Lib/test/regrtest.py -v test_warnings". I only patched regrtest.py to use backslashreplace error handler on sys.stdout.
It's not the same error, PYTHONWARNINGS is decoded from the wrong encoding: locale encodind instead of utf-8. r84731 should fix this bug (at least, it restores the encoding used because my last commit, r84694).
With r84731, Python decodes PYTHONWARNINGS with strict error handler on Mac OS X, instead of surrogateescape error handler. I don't think that it matters because PYTHONWARNINGS is not supposed to contain raw bytes, and anyway, Mac OS X rejects invalid utf-8 filenames.
Activity
voidspace commentedon May 1, 2010
If you run test_warnings.py under an ascii terminal (at least on Mac OS X) then test_warnings.CEnvironmentVariableTests.test_nonascii fails. Perhaps the test should be skipped?
======================================================================
FAIL: test_nonascii (main.CEnvironmentVariableTests)
----------------------------------------------------------------------
Looks like the traceback here actually comes from unittest trying to report the failure.
bitdancer commentedon May 1, 2010
Victor has proposed a patch for the traceback problem for regrtest, I think. I haven't looked at it, but I wonder if there is something that can instead be done to make unittest work in cases like this when run in an ascii terminal. See bpo-8522.
voidspace commentedon May 1, 2010
What does bpo-8522 have to do with it - did you mean a different issue? In unittest it could catch the UnicodeEncodeError and write the ascii repr instead.
bitdancer commentedon May 1, 2010
Heh, yes, I meant bpo-8533.
vstinner commentedon May 2, 2010
The bug only occurs on Mac OS X because file system encoding is hardcoded to UTF-8 on this OS and the test is skipped if the file system encoding is ASCII.
But the bug is not specific to test_warnings, as mentionned by R. David Murray: bpo-8533 is the real bug. test_warnings is just an example to reproduce it.
I just commited a fix for bpo-8533: it should fix also this issue.
vstinner commentedon May 17, 2010
Does my fix this bpo-8533 fixes this issue or not? I guess yes, but I would prefer a confirmation.
vstinner commentedon May 18, 2010
The test was introduced by r80066 (issue bpo-7301).
vstinner commentedon May 18, 2010
os.environ is decoded with sys.getfilesystemencoding(), whereas PYTHONWARNINGS is decoded with the locale encoding (locale.getpreferredencoding()) :-/
vstinner commentedon May 18, 2010
Attached patch creates PySys_AddWarnOptionUnicode() function and decodes PYTHONWARNINGS variable with the file system encoding (as it is done for os.environ, to be consistent) instead of the locale encoding.
The patch only concerns the POSIX version. Windows is not affected by this issue (see below).
--
This issue occurs if the file system encoding is different than the locale encoding (especially if the locale encoding is ASCII). It only occurs on Mac OS because on this OS, the file system encoding is hardcoded to UTF-8 whereas the locale encoding... depends on the locale.
I reproduced the bug on Linux by hardcoding the file system encoding to UTF-8.
--
About the test output (UnicodeEncodeError: 'ascii' codec can't encode character '\xf3' ...): I think that Michael Foord executed directly Lib/test/test_warnings.py instead of using "Lib/test/regrtest.py -v test_warnings". I only patched regrtest.py to use backslashreplace error handler on sys.stdout.
vstinner commentedon May 19, 2010
tarek tested on Mac OS X: the patch fixes test_warnings issue.
vstinner commentedon May 19, 2010
Ok, I commited my patch: r81358 (py3k). I'm waiting for the buildbots before backporting the fix to 3.1.
vstinner commentedon May 19, 2010
Oh, Python 3.1 doesn't use the PYTHONWARNINGS variable: commit blocked in 3.1 (r81362).
florentx commentedon Aug 22, 2010
A similar issue occurs on "x86 debian parallel" buildbot:
======================================================================
FAIL: test_nonascii (test.test_warnings.CEnvironmentVariableTests)
----------------------------------------------------------------------
http://www.python.org/dev/buildbot/builders/x86%20debian%20parallel%203.x/builds/80
florentx commentedon Aug 22, 2010
There's a failure on the same buildbot, "x86 debian parallel", with test test_subprocess (test_undecodable_env).
This test was introduced with issue bpo-8391.
The test_subprocess and test_warnings failures are specific to this buildbot.
vstinner commentedon Aug 22, 2010
I know test_warnings failure and I will try to fix it soon.
vstinner commentedon Sep 10, 2010
It should be fixed by r84694.
florentx commentedon Sep 11, 2010
Still happens with r84709 on PPC Tiger 3.x
======================================================================
FAIL: test_nonascii (test.test_warnings.CEnvironmentVariableTests)
----------------------------------------------------------------------
======================================================================
FAIL: test_nonascii (test.test_warnings.PyEnvironmentVariableTests)
----------------------------------------------------------------------
Ran 71 tests in 9.113s
FAILED (failures=2)
http://www.python.org/dev/buildbot/builders/PPC%20Tiger%203.x/builds/566
vstinner commentedon Sep 12, 2010
It's not the same error, PYTHONWARNINGS is decoded from the wrong encoding: locale encodind instead of utf-8. r84731 should fix this bug (at least, it restores the encoding used because my last commit, r84694).
With r84731, Python decodes PYTHONWARNINGS with strict error handler on Mac OS X, instead of surrogateescape error handler. I don't think that it matters because PYTHONWARNINGS is not supposed to contain raw bytes, and anyway, Mac OS X rejects invalid utf-8 filenames.
vstinner commentedon Sep 17, 2010
I don't see any test_warnings anymore on http://code.google.com/p/bbreport/wiki/PythonBuildbotReport. Close this issue.