You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Create PyUnicode_AsWideCharString() function #54188
PyUnicode_AsWideChar() doesn't merge surrogate pairs on a system with 32 bits wchar_t and Python compiled in narrow mode (sizeof(wchar_t) == 4 and sizeof(Py_UNICODE) == 2) => see issue bpo-8670.
It is not easy to fix this problem because the callers of PyUnicode_AsWideChar() suppose that the output (wide character) string has the same length (in character) than the input (PyUnicode) string (suppose that sizeof(wchar_t) == sizeof(Py_UNICODE)). And PyUnicode_AsWideChar() doesn't write nul character at the end if the output string is truncated.
To prepare this change, a new PyUnicode_AsWideCharString() function would help because it does compute the size of the output buffer (whereas PyUnicode_AsWideChar() requires the output buffer in an argument).
Attached patch implements it:
-------
/* Convert the Unicode object to a wide character string. The output string
always ends with a nul character. If size is not NULL, write the number of
wide characters (including the final nul character) into *size.
Returns a buffer allocated by PyMem_Alloc() (use PyMem_Free() to free it) on
success. On error, returns NULL and *size is undefined. */
PyAPI_FUNC(wchar_t*) PyUnicode_AsWideCharString(
PyUnicodeObject*unicode, /* Unicode object */Py_ssize_t*size/* number of characters of the result */
);
PyUnicode_AsWideChar() doesn't merge surrogate pairs on a system with 32 bits wchar_t and Python compiled in narrow mode (sizeof(wchar_t) == 4 and sizeof(Py_UNICODE) == 2) => see issue bpo-8670.
It is not easy to fix this problem because the callers of PyUnicode_AsWideChar() suppose that the output (wide character) string has the same length (in character) than the input (PyUnicode) string (suppose that sizeof(wchar_t) == sizeof(Py_UNICODE)). And PyUnicode_AsWideChar() doesn't write nul character at the end if the output string is truncated.
To prepare this change, a new PyUnicode_AsWideCharString() function would help because it does compute the size of the output buffer (whereas PyUnicode_AsWideChar() requires the output buffer in an argument).
Activity
vstinner commentedon Sep 29, 2010
PyUnicode_AsWideChar() doesn't merge surrogate pairs on a system with 32 bits wchar_t and Python compiled in narrow mode (sizeof(wchar_t) == 4 and sizeof(Py_UNICODE) == 2) => see issue bpo-8670.
It is not easy to fix this problem because the callers of PyUnicode_AsWideChar() suppose that the output (wide character) string has the same length (in character) than the input (PyUnicode) string (suppose that sizeof(wchar_t) == sizeof(Py_UNICODE)). And PyUnicode_AsWideChar() doesn't write nul character at the end if the output string is truncated.
To prepare this change, a new PyUnicode_AsWideCharString() function would help because it does compute the size of the output buffer (whereas PyUnicode_AsWideChar() requires the output buffer in an argument).
Attached patch implements it:
-------
/* Convert the Unicode object to a wide character string. The output string
always ends with a nul character. If size is not NULL, write the number of
wide characters (including the final nul character) into *size.
Returns a buffer allocated by PyMem_Alloc() (use PyMem_Free() to free it) on
success. On error, returns NULL and *size is undefined. */
vstinner commentedon Sep 29, 2010
New version of the patch:
Keep the call to PyUnicode_AsWideChar() in:
malemburg commentedon Sep 29, 2010
STINNER Victor wrote:
Great idea !
amauryfa commentedon Sep 29, 2010
+1 from me as well.
But shouldn't PyUnicode_AsWideCharString() merge surrogate pairs when it can? The implementation doesn't do this.
vstinner commentedon Sep 29, 2010
I don't want to do two different things at the same time. My plan is:
PyUnicode_AsWideCharString()) to convert surrogates when needed
So, you agree with the API (and the documentation)?
vstinner commentedon Sep 29, 2010
I fixed in this issue in multiple commits:
Well, you can now directly patch the documentation. I think that the API is simple and fine :-)