Skip to content

test_warnings.CEnvironmentVariableTests.test_nonascii fails under an ascii terminal #52835

Closed

Description

@voidspace
BPO 8589
Nosy @brettcannon, @vstinner, @bitdancer, @voidspace, @florentx
Files
  • sys_warnoptions_encoding.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/vstinner'
    closed_at = <Date 2010-09-17.23:41:02.436>
    created_at = <Date 2010-05-01.11:05:18.503>
    labels = ['type-bug', 'tests']
    title = 'test_warnings.CEnvironmentVariableTests.test_nonascii fails under an ascii terminal'
    updated_at = <Date 2010-09-17.23:41:02.434>
    user = 'https://github.com/voidspace'

    bugs.python.org fields:

    activity = <Date 2010-09-17.23:41:02.434>
    actor = 'vstinner'
    assignee = 'vstinner'
    closed = True
    closed_date = <Date 2010-09-17.23:41:02.436>
    closer = 'vstinner'
    components = ['Tests']
    creation = <Date 2010-05-01.11:05:18.503>
    creator = 'michael.foord'
    dependencies = []
    files = ['17389']
    hgrepos = []
    issue_num = 8589
    keywords = ['patch', 'buildbot']
    message_count = 19.0
    messages = ['104709', '104716', '104718', '104721', '104774', '105895', '106010', '106012', '106013', '106054', '106079', '106093', '114695', '114697', '114703', '116057', '116106', '116153', '116742']
    nosy_count = 5.0
    nosy_names = ['brett.cannon', 'vstinner', 'r.david.murray', 'michael.foord', 'flox']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = None
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue8589'
    versions = ['Python 3.2']

    Activity

    1. voidspace commented on May 1, 2010

      @voidspace
      ContributorAuthor

      If you run test_warnings.py under an ascii terminal (at least on Mac OS X) then test_warnings.CEnvironmentVariableTests.test_nonascii fails. Perhaps the test should be skipped?

      ======================================================================
      FAIL: test_nonascii (main.CEnvironmentVariableTests)
      ----------------------------------------------------------------------

      Traceback (most recent call last):
        File "Lib/test/test_warnings.py", line 758, in <module>
          test_main()
        File "Lib/test/test_warnings.py", line 753, in test_main
          PyEnvironmentVariableTests
        File "/compile/python-trunk3/Lib/test/support.py", line 1053, in run_unittest
          _run_suite(suite)
        File "/compile/python-trunk3/Lib/test/support.py", line 1027, in _run_suite
          result = runner.run(suite)
        File "/compile/python-trunk3/Lib/unittest/runner.py", line 158, in run
          result.printErrors()
        File "/compile/python-trunk3/Lib/unittest/runner.py", line 109, in printErrors
          self.printErrorList('FAIL', self.failures)
        File "/compile/python-trunk3/Lib/unittest/runner.py", line 116, in printErrorList
          self.stream.writeln("%s" % err)
        File "/compile/python-trunk3/Lib/unittest/runner.py", line 24, in writeln
          self.write(arg)
      UnicodeEncodeError: 'ascii' codec can't encode character '\xf3' in position 121: ordinal not in range(128)

      Looks like the traceback here actually comes from unittest trying to report the failure.

    2. bitdancer commented on May 1, 2010

      @bitdancer
      Member

      Victor has proposed a patch for the traceback problem for regrtest, I think. I haven't looked at it, but I wonder if there is something that can instead be done to make unittest work in cases like this when run in an ascii terminal. See bpo-8522.

    3. voidspace commented on May 1, 2010

      @voidspace
      ContributorAuthor

      What does bpo-8522 have to do with it - did you mean a different issue? In unittest it could catch the UnicodeEncodeError and write the ascii repr instead.

    4. bitdancer commented on May 1, 2010

      @bitdancer
      Member

      Heh, yes, I meant bpo-8533.

    5. vstinner commented on May 2, 2010

      @vstinner
      Member

      The bug only occurs on Mac OS X because file system encoding is hardcoded to UTF-8 on this OS and the test is skipped if the file system encoding is ASCII.

      But the bug is not specific to test_warnings, as mentionned by R. David Murray: bpo-8533 is the real bug. test_warnings is just an example to reproduce it.

      I just commited a fix for bpo-8533: it should fix also this issue.

    6. vstinner commented on May 17, 2010

      @vstinner
      Member

      Does my fix this bpo-8533 fixes this issue or not? I guess yes, but I would prefer a confirmation.

    7. vstinner commented on May 18, 2010

      @vstinner
      Member

      The test was introduced by r80066 (issue bpo-7301).

    8. vstinner commented on May 18, 2010

      @vstinner
      Member

      os.environ is decoded with sys.getfilesystemencoding(), whereas PYTHONWARNINGS is decoded with the locale encoding (locale.getpreferredencoding()) :-/

    9. vstinner commented on May 18, 2010

      @vstinner
      Member

      Attached patch creates PySys_AddWarnOptionUnicode() function and decodes PYTHONWARNINGS variable with the file system encoding (as it is done for os.environ, to be consistent) instead of the locale encoding.

      The patch only concerns the POSIX version. Windows is not affected by this issue (see below).

      --

      This issue occurs if the file system encoding is different than the locale encoding (especially if the locale encoding is ASCII). It only occurs on Mac OS because on this OS, the file system encoding is hardcoded to UTF-8 whereas the locale encoding... depends on the locale.

      I reproduced the bug on Linux by hardcoding the file system encoding to UTF-8.

      --

      About the test output (UnicodeEncodeError: 'ascii' codec can't encode character '\xf3' ...): I think that Michael Foord executed directly Lib/test/test_warnings.py instead of using "Lib/test/regrtest.py -v test_warnings". I only patched regrtest.py to use backslashreplace error handler on sys.stdout.

    10. vstinner commented on May 19, 2010

      @vstinner
      Member

      tarek tested on Mac OS X: the patch fixes test_warnings issue.

    11. vstinner commented on May 19, 2010

      @vstinner
      Member

      Ok, I commited my patch: r81358 (py3k). I'm waiting for the buildbots before backporting the fix to 3.1.

    12. vstinner commented on May 19, 2010

      @vstinner
      Member

      Oh, Python 3.1 doesn't use the PYTHONWARNINGS variable: commit blocked in 3.1 (r81362).

    13. florentx commented on Aug 22, 2010

      florentxmannequin
      Mannequin

      A similar issue occurs on "x86 debian parallel" buildbot:

      ======================================================================
      FAIL: test_nonascii (test.test_warnings.CEnvironmentVariableTests)
      ----------------------------------------------------------------------

      Traceback (most recent call last):
        File "/home2/buildbot2/slave/3.x.loewis-parallel/build/Lib/test/test_warnings.py", line 731, in test_nonascii
          "['ignore:DeprecaciónWarning']".encode('utf-8'))
      AssertionError: b'' != b"['ignore:Deprecaci\xc3\xb3nWarning']"

      http://www.python.org/dev/buildbot/builders/x86%20debian%20parallel%203.x/builds/80

    14. florentx commented on Aug 22, 2010

      florentxmannequin
      Mannequin

      There's a failure on the same buildbot, "x86 debian parallel", with test test_subprocess (test_undecodable_env).

      This test was introduced with issue bpo-8391.

      The test_subprocess and test_warnings failures are specific to this buildbot.

    15. vstinner commented on Aug 22, 2010

      @vstinner
      Member

      I know test_warnings failure and I will try to fix it soon.

    16. vstinner commented on Sep 10, 2010

      @vstinner
      Member

      It should be fixed by r84694.

    17. florentx commented on Sep 11, 2010

      florentxmannequin
      Mannequin

      Still happens with r84709 on PPC Tiger 3.x

      ======================================================================
      FAIL: test_nonascii (test.test_warnings.CEnvironmentVariableTests)
      ----------------------------------------------------------------------

      Traceback (most recent call last):
        File "/Users/buildbot/buildarea/3.x.parc-tiger-1/build/Lib/test/test_warnings.py", line 731, in test_nonascii
          "['ignore:Deprecaci\xf3nWarning']".encode('utf-8'))
      AssertionError: b"['ignore:Deprecaci\xc3\x83\xc2\xb3nWarning']" != b"['ignore:Deprecaci\xc3\xb3nWarning']"

      ======================================================================
      FAIL: test_nonascii (test.test_warnings.PyEnvironmentVariableTests)
      ----------------------------------------------------------------------

      Traceback (most recent call last):
        File "/Users/buildbot/buildarea/3.x.parc-tiger-1/build/Lib/test/test_warnings.py", line 731, in test_nonascii
          "['ignore:Deprecaci\xf3nWarning']".encode('utf-8'))
      AssertionError: b"['ignore:Deprecaci\xc3\x83\xc2\xb3nWarning']" != b"['ignore:Deprecaci\xc3\xb3nWarning']"

      Ran 71 tests in 9.113s

      FAILED (failures=2)

      http://www.python.org/dev/buildbot/builders/PPC%20Tiger%203.x/builds/566

    18. vstinner commented on Sep 12, 2010

      @vstinner
      Member

      Still happens with r84709 on PPC Tiger 3.x

      It's not the same error, PYTHONWARNINGS is decoded from the wrong encoding: locale encodind instead of utf-8. r84731 should fix this bug (at least, it restores the encoding used because my last commit, r84694).

      With r84731, Python decodes PYTHONWARNINGS with strict error handler on Mac OS X, instead of surrogateescape error handler. I don't think that it matters because PYTHONWARNINGS is not supposed to contain raw bytes, and anyway, Mac OS X rejects invalid utf-8 filenames.

    19. vstinner commented on Sep 17, 2010

      @vstinner
      Member

      I don't see any test_warnings anymore on http://code.google.com/p/bbreport/wiki/PythonBuildbotReport. Close this issue.

    20. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    testsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions