Skip to content

logging to file + encoding #49420

Closed
Closed

Description

@shamilbi
mannequin
BPO 5170
Nosy @vsajip
Files
  • encoding-test.zip: Simple test script and resulting output
  • unnamed
  • unnamed
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/vsajip'
    closed_at = <Date 2009-04-22.12:14:26.783>
    created_at = <Date 2009-02-06.16:07:46.195>
    labels = ['type-bug', 'library']
    title = 'logging to file + encoding'
    updated_at = <Date 2009-04-22.12:14:26.782>
    user = 'https://bugs.python.org/shamilbi'

    bugs.python.org fields:

    activity = <Date 2009-04-22.12:14:26.782>
    actor = 'vinay.sajip'
    assignee = 'vinay.sajip'
    closed = True
    closed_date = <Date 2009-04-22.12:14:26.783>
    closer = 'vinay.sajip'
    components = ['Library (Lib)']
    creation = <Date 2009-02-06.16:07:46.195>
    creator = 'shamilbi'
    dependencies = []
    files = ['12959', '13726', '13734']
    hgrepos = []
    issue_num = 5170
    keywords = []
    message_count = 11.0
    messages = ['81274', '81297', '81378', '81411', '86198', '86222', '86224', '86226', '86282', '86287', '86291']
    nosy_count = 2.0
    nosy_names = ['vinay.sajip', 'shamilbi']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = None
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue5170'
    versions = ['Python 2.6']

    Activity

    1. shamilbi commented on Feb 6, 2009

      shamilbimannequin
      MannequinAuthor

      if i configure logging into a file with encoding = 'cp1251' and do
      logger.debug(u'...') then i get crash with UnicodeError

      i suggest reimplementing method FileHandler.emit():
      ...
      if isinstance(msg, unicode):
      stream.write(f % msg) # it works!
      ...

    2. added
      type-crashA hard crash of the interpreter, possibly with a core dump
      stdlibStandard Library Python modules in the Lib/ directory
      on Feb 6, 2009
    3. vsajip commented on Feb 6, 2009

      @vsajip
      Member

      The attached test script and output file appear to show logging working
      correctly. The script writes a log message including the Cyrillic text
      доброе утро (Good morning) to a CP1251-encoded file, test.log. Opening
      this file in a Unicode-aware editor (I used BabelPad,
      http://www.babelstone.co.uk/Software/BabelPad.html) appears to show the
      Cyrillic characters correctly: see the screenshot at

      http://img5.imageshack.us/img5/799/cp1251zw2.png

      Marking as pending. Can you give more specifics about your problem?

    4. shamilbi commented on Feb 8, 2009

      shamilbimannequin
      MannequinAuthor

      test_log.py:
      -----------
      #! -- coding: windows-1251 --

      import logging
      
      logger = logging.getLogger('test_log')
      logger.addHandler(logging.FileHandler('test.log', encoding='cp1251'))
      logger.setLevel(logging.DEBUG)
      
      logger.debug(u'Привет')    # russian Hello

      exception:
      ---------

      Traceback (most recent call last):
        File "e:\bin\python\0\lib\logging\__init__.py", line 765, in emit
          self.stream.write(fs % msg.encode("UTF-8"))
        File "e:\bin\python\0\lib\codecs.py", line 686, in write
          return self.writer.write(data)
        File "e:\bin\python\0\lib\codecs.py", line 351, in write
          data, consumed = self.encode(object, self.errors)
        File "e:\bin\python\0\lib\encodings\cp1251.py", line 12, in encode
          return codecs.charmap_encode(input,errors,encoding_table)
      UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0:
      ordinal not in range(128)
    5. vsajip commented on Feb 8, 2009

      @vsajip
      Member

      Sorry, misread the original issue and tested with Python 2.5 rather than
      2.6. This is a regression; fix and additional test case checked into
      trunk and release26-maint.

    6. shamilbi commented on Apr 20, 2009

      shamilbimannequin
      MannequinAuthor

      (python 2.6.2, WinXP)
      logging to console stopped working. Here is a workaround:

      logging/__init__.py:
      class StreamHandler(Handler):
          ...
          def emit(self, record):
          ...
                          if (isinstance(msg, unicode) or
                              getattr(stream, 'encoding', None) is None):
      -                         stream.write(fs % msg)
      +                         if stream == sys.stdout:
      +                             print(msg)
      +                         else:
      +                             stream.write(fs % msg)
    7. added
      type-bugAn unexpected behavior, bug, or error
      and removed
      type-crashA hard crash of the interpreter, possibly with a core dump
      on Apr 20, 2009
    8. vsajip commented on Apr 21, 2009

      @vsajip
      Member

      Can you retry with setting the "encoding" attribute of the file to
      "cp1251"? That should work and that should be the appropriate method to
      avoid the problem.

      test_logging.py in the Python distribution has a test which exercises
      Unicode functionality using cp1251, does that test work in your environment?

    9. shamilbi commented on Apr 21, 2009

      shamilbimannequin
      MannequinAuthor

      Can you retry with setting the "encoding" attribute of the file to
      "cp1251"? That should work and that should be the appropriate method to
      avoid the problem.

      test_logging.py in the Python distribution has a test which exercises
      Unicode functionality using cp1251, does that test work in your
      environment?

      logging to file is OK (since 2.6.2), logging to console __was__ OK (2.6.1),
      not now (2.6.2)

      shamil

    10. vsajip commented on Apr 21, 2009

      @vsajip
      Member

      Trunk and release26-maint were recently changed (r71657, r71658) to use
      the following logic, which differs from the code snippet you posted.

                      if (isinstance(msg, unicode) and
                          getattr(stream, 'encoding', None)):
                          stream.write(fs.decode(stream.encoding) % msg)
                      else:
                          stream.write(fs % msg)
      

      If the stream is stderr and you are passing a unicode msg, the else
      branch will not be taken; as long as the stream has an encoding
      attribute, it should output correctly.

      The change was made when another, similar issue was posted by another
      user (issue bpo-5768).

      Can you confirm what happens with the current code as it is in
      release26-maint? спасибо!

    11. shamilbi commented on Apr 22, 2009

      shamilbimannequin
      MannequinAuthor

      Trunk and release26-maint were recently changed (r71657, r71658) to use
      the following logic, which differs from the code snippet you posted.

                     if (isinstance(msg, unicode) and
                         getattr(stream, 'encoding', None)):
                         stream.write(fs.decode(stream.encoding) % msg)
                      else:
                         stream.write(fs % msg)
      

      If the stream is stderr and you are passing a unicode msg, the else
      branch will not be taken; as long as the stream has an encoding
      attribute, it should output correctly.

      The change was made when another, similar issue was posted by another
      user (issue bpo-5768).

      Can you confirm what happens with the current code as it is in
      release26-maint? спасибо!

      it still doesn't work for console (but OK for files).

      the following works in both cases:
      if (isinstance(msg, unicode) and
      getattr(stream, 'encoding', None) and
      (stream == sys.stdout or stream == sys.stderr)):
      stream.write(fs % msg.encode(stream.encoding))
      else:
      stream.write(fs % msg)

      i think it's all about the difference betwin print(msg) and
      sys.stdout.write('%s\n' % msg)

      shamil

    12. vsajip commented on Apr 22, 2009

      @vsajip
      Member

      i think it's all about the difference betwin print(msg) and
      sys.stdout.write('%s\n' % msg)

      Yes. Actually, it's about what streams will accept. For example, a
      stream opened with codecs.open(encoding='cp1251') will accept a Unicode
      string and encode it before output. Likewise, a StringIO wrapped inside
      a writer obtained from the codecs module.

      However, stdout and stderr are of type file and seem to behave
      differently - they behave as expected if I encode the value written to
      them, but fail if I write a Unicode string to them.

      I don't think special-casing for sys.stdout and sys.stderr in the
      logging code is the correct approach. Better to put a conditional
      encoding step when it's needed - now I just need to figure out all the
      cases when it's needed ;-)

    13. vsajip commented on Apr 22, 2009

      @vsajip
      Member

      Fix checked into trunk and release26-maint.

    14. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions