Skip to content

Add PyUnicode_DecodeLocale and PyUnicode_DecodeLocaleAndSize #57769

Closed

Description

@vstinner
BPO 13560
Nosy @loewis, @vstinner, @ezio-melotti, @skrah
Files
  • pyunicode_decodelocale.patch
  • pyunicode_decodelocale-2.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2011-12-17.06:15:00.660>
    created_at = <Date 2011-12-08.23:02:16.224>
    labels = ['expert-unicode']
    title = 'Add PyUnicode_DecodeLocale and PyUnicode_DecodeLocaleAndSize'
    updated_at = <Date 2011-12-17.06:15:00.658>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2011-12-17.06:15:00.658>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2011-12-17.06:15:00.660>
    closer = 'vstinner'
    components = ['Unicode']
    creation = <Date 2011-12-08.23:02:16.224>
    creator = 'vstinner'
    dependencies = []
    files = ['23886', '23894']
    hgrepos = []
    issue_num = 13560
    keywords = ['patch']
    message_count = 6.0
    messages = ['149060', '149116', '149654', '149655', '149658', '149661']
    nosy_count = 5.0
    nosy_names = ['loewis', 'vstinner', 'ezio.melotti', 'skrah', 'python-dev']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = None
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue13560'
    versions = ['Python 3.3']

    Activity

    1. vstinner commented on Dec 8, 2011

      @vstinner
      MemberAuthor

      To decode byte string from the locale encoding (LC_CTYPE), PyUnicode_DecodeFSDefault() can be used, but this function uses a constant encoding set at startup (the locale encoding at startup). The right method is currently to call _Py_char2wchar() and then PyUnicode_FromWideChar(). _Py_char2wchar() is a low level function, it doesn't raise nice Python exception, but just return NULL on error and write a message to stderr using fprintf() (!).

      Attached patch adds PyUnicode_DecodeLocale() and PyUnicode_DecodeLocaleAndSize() to offer a high level API to decode data from the *current* locale encoding. These functions fail with an OSError or MemoryError if decoding fails (instead of a generic ValueError), and don't write to stderr anymore. They are a surrogateescape argument to choose to escape undecodable bytes or to fail with an error.

      The patch only uses the function in _localemodule.c, but other functions may have to be fixed to use the new function. The tzname_encoding.patch of issue bpo-5905 should maybe use it for example.

    2. vstinner commented on Dec 9, 2011

      @vstinner
      MemberAuthor

      I fixed issue bpo-5905 (strptime fails in non-UTF locale). The fix is not enough if the locale is changed in Python.

      Update the patch to fix time.strftime() (if wcsftime() is not available).

    3. vstinner commented on Dec 17, 2011

      @vstinner
      MemberAuthor

      changeset: 74002:279b0aee0cfb
      user: Victor Stinner <victor.stinner@haypocalc.com>
      date: Fri Dec 16 23:56:01 2011 +0100
      files: Doc/c-api/unicode.rst Include/unicodeobject.h Modules/_localemodule.c Modules/main.c Modules/timemodule.c
      description:
      Add PyUnicode_DecodeLocaleAndSize() and PyUnicode_DecodeLocale()

      • PyUnicode_DecodeLocaleAndSize() and PyUnicode_DecodeLocale() decode a string
        from the current locale encoding
      • _Py_char2wchar() writes an "error code" in the size argument to indicate
        if the function failed because of memory allocation failure or because of a
        decoding error. The function doesn't write the error message directly to
        stderr.
      • Fix time.strftime() (if wcsftime() is missing): decode strftime() result
        from the current locale encoding, not from the filesystem encoding.
    4. python-dev commented on Dec 17, 2011

      python-devmannequin
      Mannequin

      New changeset 88198b93ff2f by Victor Stinner in branch 'default':
      Issue bpo-13560: Add PyUnicode_EncodeLocale()
      http://hg.python.org/cpython/rev/88198b93ff2f

      New changeset 51412b4b81ae by Victor Stinner in branch 'default':
      Issue bpo-13560: os.strerror() now uses the current locale encoding instead of UTF-8
      http://hg.python.org/cpython/rev/51412b4b81ae

    5. python-dev commented on Dec 17, 2011

      python-devmannequin
      Mannequin

      New changeset 07802351ccad by Victor Stinner in branch 'default':
      Issue bpo-13560: Locale codec functions use the classic "errors" parameter,
      http://hg.python.org/cpython/rev/07802351ccad

    6. vstinner commented on Dec 17, 2011

      @vstinner
      MemberAuthor

      Ok, I think that the current code is good enough to close the issue. I opened a more global issue about the Python codec: bpo-13619.

    7. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions