You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add PyUnicode_DecodeLocale and PyUnicode_DecodeLocaleAndSize #57769
To decode byte string from the locale encoding (LC_CTYPE), PyUnicode_DecodeFSDefault() can be used, but this function uses a constant encoding set at startup (the locale encoding at startup). The right method is currently to call _Py_char2wchar() and then PyUnicode_FromWideChar(). _Py_char2wchar() is a low level function, it doesn't raise nice Python exception, but just return NULL on error and write a message to stderr using fprintf() (!).
Attached patch adds PyUnicode_DecodeLocale() and PyUnicode_DecodeLocaleAndSize() to offer a high level API to decode data from the *current* locale encoding. These functions fail with an OSError or MemoryError if decoding fails (instead of a generic ValueError), and don't write to stderr anymore. They are a surrogateescape argument to choose to escape undecodable bytes or to fail with an error.
The patch only uses the function in _localemodule.c, but other functions may have to be fixed to use the new function. The tzname_encoding.patch of issue bpo-5905 should maybe use it for example.
PyUnicode_DecodeLocaleAndSize() and PyUnicode_DecodeLocale() decode a string
from the current locale encoding
_Py_char2wchar() writes an "error code" in the size argument to indicate
if the function failed because of memory allocation failure or because of a
decoding error. The function doesn't write the error message directly to
stderr.
Fix time.strftime() (if wcsftime() is missing): decode strftime() result
from the current locale encoding, not from the filesystem encoding.
Activity
vstinner commentedon Dec 8, 2011
To decode byte string from the locale encoding (LC_CTYPE), PyUnicode_DecodeFSDefault() can be used, but this function uses a constant encoding set at startup (the locale encoding at startup). The right method is currently to call _Py_char2wchar() and then PyUnicode_FromWideChar(). _Py_char2wchar() is a low level function, it doesn't raise nice Python exception, but just return NULL on error and write a message to stderr using fprintf() (!).
Attached patch adds PyUnicode_DecodeLocale() and PyUnicode_DecodeLocaleAndSize() to offer a high level API to decode data from the *current* locale encoding. These functions fail with an OSError or MemoryError if decoding fails (instead of a generic ValueError), and don't write to stderr anymore. They are a surrogateescape argument to choose to escape undecodable bytes or to fail with an error.
The patch only uses the function in _localemodule.c, but other functions may have to be fixed to use the new function. The tzname_encoding.patch of issue bpo-5905 should maybe use it for example.
vstinner commentedon Dec 9, 2011
I fixed issue bpo-5905 (strptime fails in non-UTF locale). The fix is not enough if the locale is changed in Python.
Update the patch to fix time.strftime() (if wcsftime() is not available).
vstinner commentedon Dec 17, 2011
changeset: 74002:279b0aee0cfb
user: Victor Stinner <victor.stinner@haypocalc.com>
date: Fri Dec 16 23:56:01 2011 +0100
files: Doc/c-api/unicode.rst Include/unicodeobject.h Modules/_localemodule.c Modules/main.c Modules/timemodule.c
description:
Add PyUnicode_DecodeLocaleAndSize() and PyUnicode_DecodeLocale()
from the current locale encoding
if the function failed because of memory allocation failure or because of a
decoding error. The function doesn't write the error message directly to
stderr.
from the current locale encoding, not from the filesystem encoding.
python-dev commentedon Dec 17, 2011
New changeset 88198b93ff2f by Victor Stinner in branch 'default':
Issue bpo-13560: Add PyUnicode_EncodeLocale()
http://hg.python.org/cpython/rev/88198b93ff2f
New changeset 51412b4b81ae by Victor Stinner in branch 'default':
Issue bpo-13560: os.strerror() now uses the current locale encoding instead of UTF-8
http://hg.python.org/cpython/rev/51412b4b81ae
python-dev commentedon Dec 17, 2011
New changeset 07802351ccad by Victor Stinner in branch 'default':
Issue bpo-13560: Locale codec functions use the classic "errors" parameter,
http://hg.python.org/cpython/rev/07802351ccad
vstinner commentedon Dec 17, 2011
Ok, I think that the current code is good enough to close the issue. I opened a more global issue about the Python codec: bpo-13619.