Skip to content

utf-8 encoding error #51294

Closed
Closed

Description

@ArcRiley
mannequin
BPO 7045
Nosy @loewis, @amauryfa, @benjaminp, @ezio-melotti
Superseder
  • bpo-3297: Python interpreter uses Unicode surrogate pairs only before the pyc is created
  • Files
  • t.py
  • u.py
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2009-10-28.01:07:55.167>
    created_at = <Date 2009-10-03.03:46:46.589>
    labels = ['interpreter-core', 'type-bug']
    title = 'utf-8 encoding error'
    updated_at = <Date 2009-10-28.01:07:55.153>
    user = 'https://bugs.python.org/ArcRiley'

    bugs.python.org fields:

    activity = <Date 2009-10-28.01:07:55.153>
    actor = 'benjamin.peterson'
    assignee = 'none'
    closed = True
    closed_date = <Date 2009-10-28.01:07:55.167>
    closer = 'benjamin.peterson'
    components = ['Interpreter Core']
    creation = <Date 2009-10-03.03:46:46.589>
    creator = 'ArcRiley'
    dependencies = []
    files = ['15028', '15032']
    hgrepos = []
    issue_num = 7045
    keywords = []
    message_count = 10.0
    messages = ['93475', '93476', '93482', '93486', '93489', '93510', '93513', '93514', '93515', '94596']
    nosy_count = 6.0
    nosy_names = ['loewis', 'amaury.forgeotdarc', 'Rhamphoryncus', 'benjamin.peterson', 'ezio.melotti', 'ArcRiley']
    pr_nums = []
    priority = 'normal'
    resolution = 'duplicate'
    stage = None
    status = 'closed'
    superseder = '3297'
    type = 'behavior'
    url = 'https://bugs.python.org/issue7045'
    versions = ['Python 3.1']

    Activity

    1. ArcRiley commented on Oct 3, 2009

      ArcRileymannequin
      MannequinAuthor

      The following is a minimal example which does not work under Python
      3.1.1 but functions as expected on Pyhton 2.6 and 3.0.

      Python 3.1.1 believes the single UTF-8 glyph is two entirely different
      (and illegal) unicode characters:

      Traceback (most recent call last):
        File "t.py", line 2, in <module>
          print('𐑛')
      UnicodeEncodeError: 'utf-8' codec can't encode character '\ud801' in
      position 0: surrogates not allowed

      Test system is Ubuntu 9.10-beta 32-bit

    2. added
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      type-bugAn unexpected behavior, bug, or error
      on Oct 3, 2009
    3. ArcRiley commented on Oct 3, 2009

      ArcRileymannequin
      MannequinAuthor

      While t.py only bugs on 3.1, the following happens with 3.0 as well:

      >>> line = '𐑑𐑧𐑕𐑑𐑦𐑙'
      >>> first = '𐑑'
      >>> first
      '𐑑'
      >>> line[0]
      '\ud801'
      >>> line[0] == first
      False
      
      And with 2.6:
      >>> line = u'𐑑𐑧𐑕𐑑𐑦𐑙'
      >>> first = u'𐑑'
      >>> first
      u'\ud801\udc51'
    4. loewis commented on Oct 3, 2009

      loewismannequin
      Mannequin

      I can't reproduce that; it prints fine for me.

      Notice that it is perfectly fine for Python to represent this as two
      code points in UCS-2 mode (so that len(s)==2); this is called UTF-16.

    5. ezio-melotti commented on Oct 3, 2009

      @ezio-melotti
      Member

      I can't reproduce it either on Ubuntu 9.04 32-bit. I tried both from the
      terminal and from the file, using Py3.2a0.

      As Martin said, the fact that in narrow builds of Python the codepoints
      outside the BMP are represented with two surrogate pairs is a known
      "issue". This is how UTF-16 works, even if it has some problematic
      side-effects.
      In your example 'line[0]' is not equal to 'first' because line[0] is the
      codepoint of the first surrogate and 'first' is a scalar value that
      represents the SHAVIAN LETTER TOT (U+010451).

      Regarding the traceback you pasted in the first post, have you used
      print('𐑑') or print(line[0])?

      This is what I get using line[0]:
      >>> line = '𐑑𐑧𐑕𐑑𐑦𐑙'
      >>> first = '𐑑'
      >>> print(line[0])
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
      UnicodeEncodeError: 'utf-8' codec can't encode character '\ud801' in
      position 0: surrogates not allowed

      In this case you are getting an error because lone surrogates are
      invalid and they can't be encoded. If you use line[:2] instead it works
      because it takes both the surrogates:

      >>> print(line[0:2])
      𐑑
      >>> first == line[0:2]
      True

      If you really got that error with print('𐑛'), then bpo-3297 could be related.

      Can you also try this and see what it prints?
      >>> import sys
      >>> sys.maxunicode
    6. ArcRiley commented on Oct 3, 2009

      ArcRileymannequin
      MannequinAuthor
      Python 3.1.1 (r311:74480, Sep 13 2009, 22:19:17)
      [GCC 4.4.1] on linux2
      Type "help", "copyright", "credits" or "license" for more information.
      >>> import sys
      >>> sys.maxunicode
      1114111
      >>> u = '𐑑'
      >>> print(u)
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
      UnicodeEncodeError: 'utf-8' codec can't encode character '\ud801' in
      position 0: surrogates not allowed
    7. amauryfa commented on Oct 3, 2009

      @amauryfa
      Contributor

      The page:
      http://www.fileformat.info/info/unicode/char/d801/index.htm
      has a big warning saying that "U+D801 is not a valid unicode character."

      The problem is similar to bpo-6697, and lead to the same question:
      should python validate utf-8 input, and refuse invalid unicode characters?

    8. ArcRiley commented on Oct 3, 2009

      ArcRileymannequin
      MannequinAuthor

      Amaury, you are absolutely correct, \ud801 is not a valid unicode glyph,
      however I am not giving Python \ud801, I am giving Python '𐑑' (==
      '\U00010451').

      I am attaching a different short example that demonstrates that Python
      is mishandling UTF-8 on both the interactive terminal and in scripts, u.py

      The output should be the same, but on Python 3.1.1 compiled for wide
      unicode it reports two different values. As someone on #python-dev
      found '𐑑'.encode('utf-16').decode('utf-16') outputs the correct value.

    9. Rhamphoryncus commented on Oct 3, 2009

      Rhamphoryncusmannequin
      Mannequin

      I believe this is a duplicate of issue bpo-3297. When given a high unicode
      scalar value directly in the source (rather than in escaped form) python
      will split it into surrogates, even on a UTF-32 build where those
      surrogates are nonsensical and ill-formed.

      Patches for Issue bpo-3672 probably made this more visible.

    10. ArcRiley commented on Oct 3, 2009

      ArcRileymannequin
      MannequinAuthor

      This behavior is identical whether u.py or u.pyc is run on my systems,
      where that previous ticket concerns differing behavior.

      Though it is obviously related.

    11. benjaminp commented on Oct 28, 2009

      @benjaminp
      Contributor

      This is a duplicate of bpo-3297, and Adam's patch there fixes it.

    12. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      interpreter-core(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions