You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
c_types.c_wchar should not assume that sizeof(wchar_t) == sizeof(Py_UNICODE) #52916
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee='https://github.com/theller'closed_at=<Date2010-10-02.11:55:58.229>created_at=<Date2010-05-09.06:38:33.027>labels= ['ctypes', 'type-bug', 'expert-unicode']
title='c_types.c_wchar should not assume that sizeof(wchar_t) ==\tsizeof(Py_UNICODE)'updated_at=<Date2010-10-02.15:33:48.410>user='https://bugs.python.org/stutzbach'
About the issue, the problem is in function u_set(). This function should use PyUnicode_AsWideChar() but PyUnicode_AsWideChar() doesn't support surrogates... whereas PyUnicode_FromWideChar() does support surrogates.
Fix PyUnicode_AsWideChar() to support surrogates (Py_UNICODE: 2 bytes, wchar_t: 4 bytes)
u_set() of _ctypes uses PyUnicode_AsWideChar()
add a test (skipped if sizeof(wchar_t) is smaller than 4 bytes)
It's too late to fix Python2: 2.7 beta 2 was released (it doesn't support non BMP characters for chr()/ord()).
TODO:
I'm not sure that my patch on PyUnicode_AsWideChar() works if sizeof(wchar_t)==2
Test the patch on Windows
Check that it doesn't break other functions calling PyUnicode_AsWideChar()
ctypes, _locale.strcoll() and time.strftime() use PyUnicode_AsWideChar(). time has interesting comments:
/* This assumes that PyUnicode_AsWideChar doesn't do any UTF-16 expansion. */if (PyUnicode_AsWideChar((PyUnicodeObject*)format,
(wchar_t*)PyBytes_AS_STRING(tmpfmt),
PyUnicode_GetSize(format)+1) == (size_t)-1)
/* This shouldn't fail. */Py_FatalError("PyUnicode_AsWideChar failed");
bpo-9979 proposes to create a new PyUnicode_AsWideCharString() function.
stutzbach commented on Sep 29, 2010
stutzbachmannequin
MannequinAuthor
More actions
I know enough about Unicode to have reported this bug, but I don't feel knowledgeable enough about Python's Unicode implementation to comment on your suggested solution.
I'm adding the other people listed in Misc/maintainers.rst as interested in Unicode to the nosy list on this issue and the one you just linked to.
I don't know how to test "if Py_UNICODE_SIZE == 4 && SIZEOF_WCHAR_T == 2". On Windows, sizeof(wchar_t) is 2, but it looks like Python is not prepared to have Py_UNICODE != wchar_t for is Windows implementation.
wchar_t is 32 bits long on Linux and Mac OS X. So how can I test it? Or should we just drop support of "Py_UNICODE_SIZE == 4 && SIZEOF_WCHAR_T == 2"?
stutzbach commented on Oct 1, 2010
stutzbachmannequin
MannequinAuthor
More actions
I, too, can't think of any platforms where Py_UNICODE_SIZE == 4 && SIZEOF_WCHAR_T == 2 and I'm not sure what the previous policy has been. Have you noticed any other code that would set a precedent?
If no one else chimes in, perhaps ask on IRC or python-dev?
I don't know how to test "if Py_UNICODE_SIZE == 4 && SIZEOF_WCHAR_T == 2". On Windows, sizeof(wchar_t) is 2, but it looks like Python is not prepared to have Py_UNICODE != wchar_t for is Windows implementation.
wchar_t is 32 bits long on Linux and Mac OS X. So how can I test it? Or should we just drop support of "Py_UNICODE_SIZE == 4 && SIZEOF_WCHAR_T == 2"?
You can tweak the Windows pyconfig.h to use UCS4, AFAIK, if you want to
test drive this case.
But it's probably easier to configure with "gcc -fshort-wchar" on
Linux :-)
changed the title [-]c_types.c_wchar should not assume that sizeof(wchar_t) == sizeof(Py_UNICODE)[/-][+]c_types.c_wchar should not assume that sizeof(wchar_t) == sizeof(Py_UNICODE)[/+]on Oct 1, 2010
stutzbach commented on Oct 1, 2010
stutzbachmannequin
MannequinAuthor
More actions
You can tweak the Windows pyconfig.h to use UCS4, AFAIK, if you want to
test drive this case.
I seem to recall seeing some other code that assumed Windows implied UCS2. Proceed with caution. ;-)
But it's probably easier to configure with "gcc -fshort-wchar" on
Linux :-)
libc will still be using sizeof(wchar_t) == 4, though. Won't that cause Bad Things to happen when calling libc wide-character functions?
> You can tweak the Windows pyconfig.h to use UCS4, AFAIK, if you want to
> test drive this case.
I seem to recall seeing some other code that assumed Windows implied UCS2. Proceed with caution. ;-)
Probably, yes. I've never tried it myself.
> But it's probably easier to configure with "gcc -fshort-wchar" on
> Linux :-)
libc will still be using sizeof(wchar_t) == 4, though. Won't that cause Bad Things to happen when calling libc wide-character functions?
Sure, but this is just about testing an interface, not running
applications :-)
Here's what the GCC man-page has to say:
-fshort-wchar
Override the underlying type for wchar_t to be short unsigned int instead
of the default for the target. This option is useful for building
programs to run under WINE.
Warning: the -fshort-wchar switch causes GCC to generate code that is not
binary compatible with code generated without that switch. Use it to
conform to a non-default application binary interface.
implement unicode_aswidechar() for 16 bits wchar_t and 32 bits Py_UNICODE
PyUnicode_AsWideWcharString() returns the number of wide characters excluding the nul character as does PyUnicode_AsWideChar()
For 16 bits wchar_t and 32 bits Py_UNICODE, I extracted the "as wide char" unicode functions into a small C file and compiled it with -fshort-wchar on Linux.
r85172 changes PyUnicode_AsWideCharString() (don't count the trailing nul character in the output size) and add unit tests.
r85173 patches unicode_aswidechar() to supports non-BMP characters for all known wchar_t/Py_UNICODE size combinaisons (2/2, 2/4 and 4/2).
I noticed that PyUnicode_AsWideChar() and PyUnicode_AsWideCharString() accept embeded nul characters. I don't know if it is a bug or an expected behaviour. Anyway, there is now a test for this case.
Since this was a bugfix, it should be merged back into 2.7, yes?
Mmmh, the fix requires to change PyUnicode_AsWideChar() function (support non-BMP characters and surrogate pairs) (and maybe also to create PyUnicode_AsWideCharString()). I don't really want to change such important function in a stable branch (Python2).
Is it really important to support non-BMP characters for ctypes.c_wchar in Python2? I would like to say: if you want better unicode support, use Python 3. And Python 3.2 if it's possible :-)
stutzbach commented on Oct 2, 2010
stutzbachmannequin
MannequinAuthor
More actions
Since I noticed the bug through source code inspection and no one has reported it occurring in practice, that sounds reasonable to me.
Activity
stutzbach commentedon May 9, 2010
Using a UCS2 Python on a platform with a 32-bit wchar_t, the following code throws an exception (but should not):
>>> ctypes.c_wchar('\u10000') Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: one character unicode string expectedThe trouble is in the u_set() function in Modules/_ctypes/cfield.c. The corresponding u_get() function looks correct.
On a UCS4 Python running on a system with a 16-bit wchar_t, u_set() will corrupt the data by silently truncating the character to 16-bits.
For reference, Linux and Mac OS use a 32-bit wchar_t while Windows uses a 16-bit wchar_t.
vstinner commentedon May 13, 2010
Support of characters outside the Unicode BMP (code > 0xffff) is not complete in narrow build (sizeof(Py_UNICODE) == 2) for Python2:
It looks better in Python3:
About the issue, the problem is in function u_set(). This function should use PyUnicode_AsWideChar() but PyUnicode_AsWideChar() doesn't support surrogates... whereas PyUnicode_FromWideChar() does support surrogates.
vstinner commentedon May 13, 2010
Patch for Python3:
It's too late to fix Python2: 2.7 beta 2 was released (it doesn't support non BMP characters for chr()/ord()).
TODO:
ctypes, _locale.strcoll() and time.strftime() use PyUnicode_AsWideChar(). time has interesting comments:
vstinner commentedon Sep 29, 2010
bpo-9979 proposes to create a new PyUnicode_AsWideCharString() function.
stutzbach commentedon Sep 29, 2010
I know enough about Unicode to have reported this bug, but I don't feel knowledgeable enough about Python's Unicode implementation to comment on your suggested solution.
I'm adding the other people listed in Misc/maintainers.rst as interested in Unicode to the nosy list on this issue and the one you just linked to.
vstinner commentedon Oct 1, 2010
Update the patch for the new PyUnicode_AsWideCharString() function:
The patch is not finished: I have to implement "#elif Py_UNICODE_SIZE == 4 && SIZEOF_WCHAR_T == 2" case.
vstinner commentedon Oct 1, 2010
Patch version 3:
vstinner commentedon Oct 1, 2010
I don't know how to test "if Py_UNICODE_SIZE == 4 && SIZEOF_WCHAR_T == 2". On Windows, sizeof(wchar_t) is 2, but it looks like Python is not prepared to have Py_UNICODE != wchar_t for is Windows implementation.
wchar_t is 32 bits long on Linux and Mac OS X. So how can I test it? Or should we just drop support of "Py_UNICODE_SIZE == 4 && SIZEOF_WCHAR_T == 2"?
stutzbach commentedon Oct 1, 2010
I, too, can't think of any platforms where Py_UNICODE_SIZE == 4 && SIZEOF_WCHAR_T == 2 and I'm not sure what the previous policy has been. Have you noticed any other code that would set a precedent?
If no one else chimes in, perhaps ask on IRC or python-dev?
malemburg commentedon Oct 1, 2010
STINNER Victor wrote:
You can tweak the Windows pyconfig.h to use UCS4, AFAIK, if you want to
test drive this case.
But it's probably easier to configure with "gcc -fshort-wchar" on
Linux :-)
Dropping support for this is not a good idea.
[-]c_types.c_wchar should not assume that sizeof(wchar_t) == sizeof(Py_UNICODE)[/-][+]c_types.c_wchar should not assume that sizeof(wchar_t) == sizeof(Py_UNICODE)[/+]stutzbach commentedon Oct 1, 2010
I seem to recall seeing some other code that assumed Windows implied UCS2. Proceed with caution. ;-)
libc will still be using sizeof(wchar_t) == 4, though. Won't that cause Bad Things to happen when calling libc wide-character functions?
malemburg commentedon Oct 1, 2010
Daniel Stutzbach wrote:
Probably, yes. I've never tried it myself.
Sure, but this is just about testing an interface, not running
applications :-)
Here's what the GCC man-page has to say:
vstinner commentedon Oct 1, 2010
Patch version 4:
For 16 bits wchar_t and 32 bits Py_UNICODE, I extracted the "as wide char" unicode functions into a small C file and compiled it with -fshort-wchar on Linux.
vstinner commentedon Oct 1, 2010
Ooops, I lost my patch to fix the initial (ctypes) issue. Here is an updated patch: ctypes_nonbmp.patch (which needs aswidechar_nonbmp-4.patch).
vstinner commentedon Oct 2, 2010
r85172 changes PyUnicode_AsWideCharString() (don't count the trailing nul character in the output size) and add unit tests.
r85173 patches unicode_aswidechar() to supports non-BMP characters for all known wchar_t/Py_UNICODE size combinaisons (2/2, 2/4 and 4/2).
I noticed that PyUnicode_AsWideChar() and PyUnicode_AsWideCharString() accept embeded nul characters. I don't know if it is a bug or an expected behaviour. Anyway, there is now a test for this case.
vstinner commentedon Oct 2, 2010
r85174+r85177: ctypes.c_wchar supports non-BMP characters with 32 bits wchar_t => fix this issue
(I commited also an unwanted change on _testcapi to fix r85172 in r85174: r85175 reverts this change, and r85176 fixes the _testcapi bug again)
vstinner commentedon Oct 2, 2010
Oh, and 4/4 ;-)
stutzbach commentedon Oct 2, 2010
Thanks for working on this!
Since this was a bugfix, it should be merged back into 2.7, yes?
vstinner commentedon Oct 2, 2010
Mmmh, the fix requires to change PyUnicode_AsWideChar() function (support non-BMP characters and surrogate pairs) (and maybe also to create PyUnicode_AsWideCharString()). I don't really want to change such important function in a stable branch (Python2).
Is it really important to support non-BMP characters for ctypes.c_wchar in Python2? I would like to say: if you want better unicode support, use Python 3. And Python 3.2 if it's possible :-)
stutzbach commentedon Oct 2, 2010
Since I noticed the bug through source code inspection and no one has reported it occurring in practice, that sounds reasonable to me.