You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Use Py_UCS4 instead of Py_UNICODE in unicodectype.c #49377
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee='https://github.com/amauryfa'closed_at=<Date2010-08-19.12:56:22.263>created_at=<Date2009-02-02.01:31:39.408>labels= ['type-bug', 'expert-unicode']
title='Use Py_UCS4 instead of Py_UNICODE in unicodectype.c'updated_at=<Date2010-08-19.12:56:22.261>user='https://bugs.python.org/bupjae'
>>> license
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python30\lib\site.py", line 372, in __repr__
self.__setup()
File "C:\Python30\lib\site.py", line 359, in __setup
data = fp.read()
File "C:\Python30\lib\io.py", line 1724, in read
decoder.decode(self.buffer.read(), final=True))
File "C:\Python30\lib\io.py", line 1295, in decode
output = self.decoder.decode(input, final=final)
UnicodeDecodeError: 'cp949' codec can't decode bytes in position 15164-
15165: il
legal multibyte sequence
>>> chr(0x10000)
'\U00010000'
>>> chr(0x11000)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python30\lib\io.py", line 1491, in write
b = encoder.encode(s)
UnicodeEncodeError: 'cp949' codec can't encode character '\ud804' in
position 1:
illegal multibyte sequence
>>>
I also can't understand why chr(0x10000) and chr(0x11000) has different
behavior
Here (winxpsp2, Py3, cp850-terminal) the license works fine:
>>> license
Type license() to see the full license text
and license() works as well.
I get this output for the chr()s:
>>> chr(0x10000)
'\U00010000'
>>> chr(0x11000)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Programs\Python30\lib\io.py", line 1491, in write
b = encoder.encode(s)
File "C:\Programs\Python30\lib\encodings\cp850.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position
1-2: character maps to <undefined>
I believe that chr(0x10000) and chr(0x11000) should have the opposite
behavior.
U+10000 (LINEAR B SYLLABLE B008 A) belongs to the 'Lo' category and
should be printed (and possibly raise a UnicodeError, see bpo-5110 1), U+11000 belongs to the 'Cn' category and should be escaped2.
On Linux with Py3 and a UTF-8 terminal, chr(0x10000) prints '\U00010000'
and chr(0x11000) prints the char (actually I see two boxes, but it
shouldn't be a problem of Python). The license() works fine too.
Also note that with cp850 the error message is 'character maps to
<undefined>' and with cp949 is 'illegal multibyte sequence'.
Python 2.7a0 (trunk:68963M, Jan 30 2009, 00:49:28)
>>> import unicodedata
>>> unicodedata.category(u"\U00010000")
'Lo'
>>> unicodedata.category(u"\U00011000")
'Cn'
>>> unicodedata.category(unichr(0x10000))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: unichr() arg not in range(0x10000) (narrow Python build)
Why unichr() fails whereas \Uxxxxxxxx works?
>>> len(u"\U00010000")
2
>>> ord(u"\U00010000")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: ord() expected a character, but string of length 2 found
FWIW, on Python3 it seems to work:
>>> import unicodedata
>>> unicodedata.category("\U00010000")
'Lo'
>>> unicodedata.category("\U00011000")
'Cn'
>>> unicodedata.category(chr(0x10000))
'Lo'
>>> unicodedata.category(chr(0x11000))
'Cn'
>>> ord(chr(0x10000)), 0x10000
(65536, 65536)
>>> ord(chr(0x11000)), 0x11000
(69632, 69632)
I'm using a narrow build too:
>>> import sys
>>> sys.maxunicode
65535
>>> len('\U00010000')
2
>>> ord('\U00010000')
65536
On Python2 unichr() is supposed to raise a ValueError on a narrow build
if the value is greater than 0xFFFF 1, but if the characters above
0xFFFF can be represented with u"\Uxxxxxxxx" there should be a way to
fix unichr so it can return them. Python3 already does it with chr().
Maybe we should open a new issue for this if it's not present already.
Since r56395, ord() and chr() accept and return surrogate pairs even in
narrow builds.
The goal is to remove most differences between narrow and wide unicode
builds (except for string lengths, indices or slices)
To address this problem, I suggest to change all functions in
unicodectype.c so that they accept Py_UCS4 characters (instead of
Py_UNICODE).
This would be a binary-incompatible change; and --with-wctype-functions
would have an effect only if sizeof(wchar_t)==4 (instead of the current
condition sizeof(wchar_t)==sizeof(PY_UNICODE_TYPE))
Since r56395, ord() and chr() accept and return surrogate pairs even in
narrow builds.
The goal is to remove most differences between narrow and wide unicode
builds (except for string lengths, indices or slices)
To address this problem, I suggest to change all functions in
unicodectype.c so that they accept Py_UCS4 characters (instead of
Py_UNICODE).
-1.
That would cause major breakage in the C API and is not inline with the
intention of having a Py_UNICODE type in the first place.
Users who are interested in UCS4 builds should simply use UCS4 builds.
This would be a binary-incompatible change; and --with-wctype-functions
would have an effect only if sizeof(wchar_t)==4 (instead of the current
condition sizeof(wchar_t)==sizeof(PY_UNICODE_TYPE))
--with-wctype-functions was scheduled for removal many releases ago,
but I never got around to it. The only reason it's still there is
that some Linux distribution use this config option (AFAIR, RedHat).
I'd be +1 on removing the option in 3.0.1 or deprecating it in
3.0.1 and removing it in 3.1.
It's not useful in any way, and causes compatibility problems
with regular builds.
amaury> Since r56395, ord() and chr() accept and return surrogate pairs
amaury> even in narrow builds.
Note: My examples are made with Python 2.x.
The goal is to remove most differences between narrow and wide unicode
builds (except for string lengths, indices or slices)
It would be nice to get the same behaviour in Python 2.x and 3.x to help
migration from Python2 to Python3 ;-)
unichr() (in Python 2.x) documentation is correct. But I would approciate to
support surrogates using unichr() which means also changing ord() behaviour.
To address this problem, I suggest to change all functions in
unicodectype.c so that they accept Py_UCS4 characters (instead of
Py_UNICODE).
Why? Using surrogates, you can use 16-bits Py_UNICODE to store non-BMP
characters (code > 0xffff).
--
I can open a new issue if you agree that we can change unichr() / ord()
behaviour on narrow build. We may ask on the mailing list?
amaury> Since r56395, ord() and chr() accept and return surrogate pairs
amaury> even in narrow builds.
Note: My examples are made with Python 2.x.
> The goal is to remove most differences between narrow and wide unicode
> builds (except for string lengths, indices or slices)
It would be nice to get the same behaviour in Python 2.x and 3.x to help
migration from Python2 to Python3 ;-)
unichr() (in Python 2.x) documentation is correct. But I would approciate to
support surrogates using unichr() which means also changing ord() behaviour.
This is not possible for unichr() in Python 2.x, since applications
always expect len(unichr(x)) == 1.
Changing ord() would be possible in Python 2.x is easier, since
this would only extend the range of returned values for UCS2
builds.
> To address this problem, I suggest to change all functions in
> unicodectype.c so that they accept Py_UCS4 characters (instead of
> Py_UNICODE).
Why? Using surrogates, you can use 16-bits Py_UNICODE to store non-BMP
characters (code > 0xffff).
--
I can open a new issue if you agree that we can change unichr() / ord()
behaviour on narrow build. We may ask on the mailing list?
Not if you recompile. I don't see how this breaks the API at the C level.
and is not inline with the intention of having a Py_UNICODE
type in the first place.
Py_UNICODE is still used as the allocation unit for unicode strings.
To get correct results, we need a way to access the whole unicode
database even on ucs2 builds; it's possible with the unicodedata module,
why not from C?
lemburg> This is not possible for unichr() in Python 2.x, since applications
lemburg> always expect len(unichr(x)) == 1
Oh, ok.
lemburg> Changing ord() would be possible in Python 2.x is easier, since
lemburg> this would only extend the range of returned values for UCS2
lemburg> builds.
ord() of Python3 (narrow build) rejects surrogate characters:
'\U00010000'
>>> len(chr(0x10000))
2
>>> ord(0x10000)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: ord() expected string of length 1, but int found
It looks that narrow builds with surrogates have some more problems...
Test with U+10000: "LINEAR B SYLLABLE B008 A", category: Letter, Other.
Not if you recompile. I don't see how this breaks the API at the C level.
Well, then try to look at such a change from a C extension
writer's perspective.
They'd have to change all their function calls and routines to work
with Py_UCS4.
Supporting both the old API and the new one would
be nearly impossible and require either an adapter API or a lot
of #ifdef'ery.
Please remember that the public Python C API is not only meant for
Python developers. It's main purpose is for it to be used by
other developers extending or embedding Python and those developers
use different release cycles and want to support more than just the
bleeding edge Python version.
Python has a long history of providing very stable APIs, both in
C and in Python.
FWIW: The last major change in the C API (the change to Py_ssize_t
from Python 2.4 to 2.5) has not even propogated to all major C
extensions yet. It's only now that people start to realize problems
with this, since their extensions start failing with segfaults
on 64-bit machines.
That said, we can of course provide additional UCS4 APIs for
certain things and also provide conversion helpers between
Py_UNICODE and Py_UCS4 where needed.
> and is not inline with the intention of having a Py_UNICODE
> type in the first place.
Py_UNICODE is still used as the allocation unit for unicode strings.
To get correct results, we need a way to access the whole unicode
database even on ucs2 builds; it's possible with the unicodedata module,
why not from C?
I must be missing some detail, but what does the Unicode database
have to do with the unicodeobject.c C API ?
haypo> ord() of Python3 (narrow build) rejects surrogate characters:
haypo> '\U00010000'
haypo> >>> len(chr(0x10000))
haypo> 2
haypo> >>> ord(0x10000)
haypo> TypeError: ord() expected string of length 1, but int found
ord() works fine on Py3, you probably meant to do
>>> ord('\U00010000')
65536
or
>>> ord(chr(0x10000))
65536
In Py3 is also stated that it accepts surrogate pairs (help(ord)).
Py2 instead doesn't support them:
>>> ord(u'\U00010000')
TypeError: ord() expected a character, but string of length 2 found
I must be missing some detail, but what does the Unicode database
have to do with the unicodeobject.c C API ?
Ah, now I understand your concerns. My suggestion is to change only the 20 functions in
unicodectype.c: _PyUnicode_IsAlpha, _PyUnicode_ToLowercase... and no change in
unicodeobject.c at all.
They all take a single code point as argument, some also return a single code point.
Changing these functions is backwards compatible.
I join a patch so we can argue on concrete code (tests are missing).
Another effect of the patch: unicodedata.numeric('\N{AEGEAN NUMBER TWO}') can return 2.0.
The str.isalpha() (and others) methods did not change: they still split the surrogate pairs.
Marc-Andre Lemburg wrote:
> I was never a fan of the Unicode repr() change to begin with. The
> repr() of an object should work in almost all cases.
I still think that bpo-5110 should be fixed (there's also a patch to fix the issue on Windows). If you agree please comment there and/or reopen that issue.
Could you explain what this bit is about ?
-#if defined(HAVE_USABLE_WCHAR_T) && defined(WANT_WCTYPE_FUNCTIONS)
+#if defined(Py_UNICODE_WIDE) && defined(WANT_WCTYPE_FUNCTIONS)
On Windows at least, HAVE_USABLE_WCHAR_T is True, this means that Py_Unicode can be converted to wchar_t. But now that Py_UNICODE_ISSPACE() takes Py_UCS4, it cannot be converted to wchar_t anymore.
Now that the unicode database functions claim to use Py_UCS4, the functions of wctypes.h are usable only if they also support Py_UCS4.
OTOH the symbol WANT_WCTYPE_FUNCTIONS is defined only if ./configure is called with --with-wctype-functions, I don't expect it to be common.
BTW, the comment says that "This reduces the interpreter's code size". I don't really agree, these functions are two-liners.
> Could you explain what this bit is about ?
> -#if defined(HAVE_USABLE_WCHAR_T) && defined(WANT_WCTYPE_FUNCTIONS)
> +#if defined(Py_UNICODE_WIDE) && defined(WANT_WCTYPE_FUNCTIONS)
On Windows at least, HAVE_USABLE_WCHAR_T is True, this means that Py_Unicode can be converted to wchar_t. But now that Py_UNICODE_ISSPACE() takes Py_UCS4, it cannot be converted to wchar_t anymore.
Right, but you still have to check whether wchar_t is usable, don't
you ? If wchar_t is 2 bytes on a non-Windows platform, using the
wctypes functions won't work either.
Now that the unicode database functions claim to use Py_UCS4, the functions of wctypes.h are usable only if they also support Py_UCS4.
OTOH the symbol WANT_WCTYPE_FUNCTIONS is defined only if ./configure is called with --with-wctype-functions, I don't expect it to be common.
BTW, the comment says that "This reduces the interpreter's code size". I don't really agree, these functions are two-liners.
True.
The support for the wctype functions should have been remove long ago,
since they cause subtle incompatibilities between Python builds. I should
have probably never added it in the first place... people were worried
about the size of the type record tables at the time, which is why
I thought it would be a good idea to try to optionally use the C lib
functions.
The comment was true before the Python type tables were changed
into a type record database: the switch used to remove the
Python tables required for those functions. With the type records
database, this is no longer the case, since the records are also
being used for properties that are not exposed via wctype functions.
The 'if' in 'gettyperecord'. (I would also rewrite that as "if (code > 0x10FFFF)", it looks more readable to me.)
The patch seems OK to me. In the NEWS message 'python' should be capitalized and I would also mention .isprintable() and possibly other functions that are affected directly -- mentioning repr() is good too, but it's only affected indirectly.
The 'if' in 'gettyperecord'. (I would also rewrite that as "if (code > 0x10FFFF)", it looks more readable to me.)
Ah, good catch !
The patch seems OK to me. In the NEWS message 'python' should be capitalized and I would also mention .isprintable() and possibly other functions that are affected directly -- mentioning repr() is good too, but it's only affected indirectly.
Activity
bupjae commentedon Feb 2, 2009
I also can't understand why chr(0x10000) and chr(0x11000) has different
behavior
ezio-melotti commentedon Feb 2, 2009
Here (winxpsp2, Py3, cp850-terminal) the license works fine: >>> license Type license() to see the full license textand license() works as well.
I believe that chr(0x10000) and chr(0x11000) should have the opposite
behavior.
U+10000 (LINEAR B SYLLABLE B008 A) belongs to the 'Lo' category and
should be printed (and possibly raise a UnicodeError, see bpo-5110
1), U+11000 belongs to the 'Cn' category and should be escaped2.
On Linux with Py3 and a UTF-8 terminal, chr(0x10000) prints '\U00010000'
and chr(0x11000) prints the char (actually I see two boxes, but it
shouldn't be a problem of Python). The license() works fine too.
Also note that with cp850 the error message is 'character maps to
<undefined>' and with cp949 is 'illegal multibyte sequence'.
amauryfa commentedon Feb 2, 2009
There were non-ascii characters in the Windows license file. This was
corrected with r67860.
This other problem is because on a narrow unicode build,
Py_UNICODE_ISPRINTABLE takes a 16bit integer.
And indeed,
vstinner commentedon Feb 3, 2009
I don't understand the behaviour of unichr():
Why unichr() fails whereas \Uxxxxxxxx works?
ezio-melotti commentedon Feb 3, 2009
On Python2 unichr() is supposed to raise a ValueError on a narrow build
if the value is greater than 0xFFFF 1, but if the characters above
0xFFFF can be represented with u"\Uxxxxxxxx" there should be a way to
fix unichr so it can return them. Python3 already does it with chr().
Maybe we should open a new issue for this if it's not present already.
amauryfa commentedon Feb 3, 2009
Since r56395, ord() and chr() accept and return surrogate pairs even in
narrow builds.
The goal is to remove most differences between narrow and wide unicode
builds (except for string lengths, indices or slices)
To address this problem, I suggest to change all functions in
unicodectype.c so that they accept Py_UCS4 characters (instead of
Py_UNICODE).
This would be a binary-incompatible change; and --with-wctype-functions
would have an effect only if sizeof(wchar_t)==4 (instead of the current
condition sizeof(wchar_t)==sizeof(PY_UNICODE_TYPE))
malemburg commentedon Feb 3, 2009
On 2009-02-03 13:39, Amaury Forgeot d'Arc wrote:
-1.
That would cause major breakage in the C API and is not inline with the
intention of having a Py_UNICODE type in the first place.
Users who are interested in UCS4 builds should simply use UCS4 builds.
--with-wctype-functions was scheduled for removal many releases ago,
but I never got around to it. The only reason it's still there is
that some Linux distribution use this config option (AFAIR, RedHat).
I'd be +1 on removing the option in 3.0.1 or deprecating it in
3.0.1 and removing it in 3.1.
It's not useful in any way, and causes compatibility problems
with regular builds.
vstinner commentedon Feb 3, 2009
amaury> Since r56395, ord() and chr() accept and return surrogate pairs
amaury> even in narrow builds.
Note: My examples are made with Python 2.x.
It would be nice to get the same behaviour in Python 2.x and 3.x to help
migration from Python2 to Python3 ;-)
unichr() (in Python 2.x) documentation is correct. But I would approciate to
support surrogates using unichr() which means also changing ord() behaviour.
Why? Using surrogates, you can use 16-bits Py_UNICODE to store non-BMP
characters (code > 0xffff).
--
I can open a new issue if you agree that we can change unichr() / ord()
behaviour on narrow build. We may ask on the mailing list?
malemburg commentedon Feb 3, 2009
On 2009-02-03 14:14, STINNER Victor wrote:
This is not possible for unichr() in Python 2.x, since applications
always expect len(unichr(x)) == 1.
Changing ord() would be possible in Python 2.x is easier, since
this would only extend the range of returned values for UCS2
builds.
amauryfa commentedon Feb 3, 2009
Not if you recompile. I don't see how this breaks the API at the C level.
Py_UNICODE is still used as the allocation unit for unicode strings.
To get correct results, we need a way to access the whole unicode
database even on ucs2 builds; it's possible with the unicodedata module,
why not from C?
My motivation for the change is this post:
http://mail.python.org/pipermail/python-dev/2008-July/080900.html
vstinner commentedon Feb 3, 2009
lemburg> This is not possible for unichr() in Python 2.x, since applications
lemburg> always expect len(unichr(x)) == 1
Oh, ok.
lemburg> Changing ord() would be possible in Python 2.x is easier, since
lemburg> this would only extend the range of returned values for UCS2
lemburg> builds.
ord() of Python3 (narrow build) rejects surrogate characters:
It looks that narrow builds with surrogates have some more problems...
Test with U+10000: "LINEAR B SYLLABLE B008 A", category: Letter, Other.
Correct result (Python 2.5, wide build):
Error in Python3 (narrow build):
Unicode ranges, all in the category "Other, Surrogate":
malemburg commentedon Feb 3, 2009
On 2009-02-03 14:50, Amaury Forgeot d'Arc wrote:
Well, then try to look at such a change from a C extension
writer's perspective.
They'd have to change all their function calls and routines to work
with Py_UCS4.
Supporting both the old API and the new one would
be nearly impossible and require either an adapter API or a lot
of #ifdef'ery.
Please remember that the public Python C API is not only meant for
Python developers. It's main purpose is for it to be used by
other developers extending or embedding Python and those developers
use different release cycles and want to support more than just the
bleeding edge Python version.
Python has a long history of providing very stable APIs, both in
C and in Python.
FWIW: The last major change in the C API (the change to Py_ssize_t
from Python 2.4 to 2.5) has not even propogated to all major C
extensions yet. It's only now that people start to realize problems
with this, since their extensions start failing with segfaults
on 64-bit machines.
That said, we can of course provide additional UCS4 APIs for
certain things and also provide conversion helpers between
Py_UNICODE and Py_UCS4 where needed.
I must be missing some detail, but what does the Unicode database
have to do with the unicodeobject.c C API ?
There are certainly other ways to make Python deal with surrogates
in more cases than the ones we already support.
ezio-melotti commentedon Feb 3, 2009
amauryfa commentedon Feb 3, 2009
Ah, now I understand your concerns. My suggestion is to change only the 20 functions in
unicodectype.c: _PyUnicode_IsAlpha, _PyUnicode_ToLowercase... and no change in
unicodeobject.c at all.
They all take a single code point as argument, some also return a single code point.
Changing these functions is backwards compatible.
I join a patch so we can argue on concrete code (tests are missing).
Another effect of the patch: unicodedata.numeric('\N{AEGEAN NUMBER TWO}') can return 2.0.
The str.isalpha() (and others) methods did not change: they still split the surrogate pairs.
30 remaining items
malemburg commentedon Jul 8, 2010
Ezio Melotti wrote:
Let's discuss this on bpo-9198.
amauryfa commentedon Jul 8, 2010
amauryfa commentedon Jul 8, 2010
A new patch, generated on top of r82662
malemburg commentedon Jul 8, 2010
Amaury Forgeot d'Arc wrote:
Could you explain what this bit is about ?
@@ -349,7 +313,7 @@
configure Python using --with-wctype-functions. This reduces the
interpreter's code size. */
-#if defined(HAVE_USABLE_WCHAR_T) && defined(WANT_WCTYPE_FUNCTIONS)
+#if defined(Py_UNICODE_WIDE) && defined(WANT_WCTYPE_FUNCTIONS)
amauryfa commentedon Jul 8, 2010
A new patch that doesn't remove an important check, avoids a crash when the C macro is called with a huge number. thanks Ezio.
amauryfa commentedon Jul 8, 2010
On Windows at least, HAVE_USABLE_WCHAR_T is True, this means that Py_Unicode can be converted to wchar_t. But now that Py_UNICODE_ISSPACE() takes Py_UCS4, it cannot be converted to wchar_t anymore.
Now that the unicode database functions claim to use Py_UCS4, the functions of wctypes.h are usable only if they also support Py_UCS4.
OTOH the symbol WANT_WCTYPE_FUNCTIONS is defined only if ./configure is called with --with-wctype-functions, I don't expect it to be common.
BTW, the comment says that "This reduces the interpreter's code size". I don't really agree, these functions are two-liners.
malemburg commentedon Jul 9, 2010
Amaury Forgeot d'Arc wrote:
Right, but you still have to check whether wchar_t is usable, don't
you ? If wchar_t is 2 bytes on a non-Windows platform, using the
wctypes functions won't work either.
The line should probably read:
True.
The support for the wctype functions should have been remove long ago,
since they cause subtle incompatibilities between Python builds. I should
have probably never added it in the first place... people were worried
about the size of the type record tables at the time, which is why
I thought it would be a good idea to try to optionally use the C lib
functions.
The comment was true before the Python type tables were changed
into a type record database: the switch used to remove the
Python tables required for those functions. With the type records
database, this is no longer the case, since the records are also
being used for properties that are not exposed via wctype functions.
malemburg commentedon Jul 9, 2010
Amaury Forgeot d'Arc wrote:
Could you please be more specific on what you changed ?
At least visually, there don't appear to be any differences between
v4 and v5 of the patch.
Thanks,
Marc-Andre Lemburg
eGenix.com
2010-07-19: EuroPython 2010, Birmingham, UK 9 days to go
::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
ezio-melotti commentedon Jul 9, 2010
The 'if' in 'gettyperecord'. (I would also rewrite that as "if (code > 0x10FFFF)", it looks more readable to me.)
The patch seems OK to me. In the NEWS message 'python' should be capitalized and I would also mention .isprintable() and possibly other functions that are affected directly -- mentioning repr() is good too, but it's only affected indirectly.
malemburg commentedon Jul 9, 2010
Ezio Melotti wrote:
Ah, good catch !
amauryfa commentedon Jul 9, 2010
str.isprintable() &co are not changed by this patch, because they enumerate Py_UNICODE units and do not join surrogates. See bpo-9200
amauryfa commentedon Jul 9, 2010
In this 6th patch, the wctype part was changed as suggested.
there is one more condition, Py_UNICODE_WIDE:
-#if defined(HAVE_USABLE_WCHAR_T) && defined(WANT_WCTYPE_FUNCTIONS)
+#if defined(WANT_WCTYPE_FUNCTIONS) && defined(HAVE_USABLE_WCHAR_T) && defined(Py_UNICODE_WIDE)
amauryfa commentedon Aug 19, 2010
Committed with r84177.