You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee='https://github.com/warsaw'closed_at=<Date2008-10-30.21:40:25.807>created_at=<Date2008-10-27.13:29:14.027>labels= ['interpreter-core', 'type-bug', 'release-blocker']
title='Lower case file system encoding'updated_at=<Date2008-10-30.22:38:38.348>user='https://github.com/tiran'
Python should lower case the file system encoding in Python/pythonrun.c.
On several occasions Python optimizes code paths for lower case
encodings like "utf-8" or "latin-1". On my Ubuntu system the file system
encoding is upper case ("UTF-8") and the optimizations aren't used. This
also causes problems with sub interpreters bpo-3723 initstdio() in the sub
interpreter fails because "UTF-8" must be looked up in the codecs and
encoding registry while "utf-8" works like a charm.
if (codeset) {
if (!Py_FileSystemDefaultEncoding) {
char*p;
for (p=codeset; *p; p++)
*p=tolower(*p);
Py_FileSystemDefaultEncoding=codeset;
}
elsefree(codeset);
}
Python/codecs.c:normalizestring() does a similar job. Maybe a new method
"char* PyCodec_NormalizeEncodingName(const char*)" could be introduced
for the problem.
Converting to the lower case doesn't solve the problem: if the locale
is "utf8" and Python checks for "utf-8", the optimization will fail.
Another example: iso-8859-1, latin-1 or latin1?
A correct patch would be to get the most common name of the charset
and make sure that Python C code always use this name.
The lower-casing doesn't hurt, since that's done anyway during codec
lookup, but I'd be -1 on making this try to duplicate the aliasing
already done by the encodings package.
I can't agree with that evaluation. In cases where Python would fail
without this patch (i.e. because the file system encoding cannot be
found during startup), this solution doesn't work well in general - it
only works if the file system encoding happens to be UTF-8. If the file
system encoding is not in the list of "builtin" codec names, startup
would still fail.
r67057 addresses this case in a somewhat more general manner, by falling
back to ASCII during startup, for encoding file names. This should work
in the usual case where Python is in /usr/bin (say), but it's still
possible to make it fail, e.g. if the codecs are in /home/Питон (say),
on a system that uses koi8-r as the file system encoding, this bug would
persist despite the two patches that have been applied.
Le Thursday 30 October 2008 23:14:21 Martin v. Löwis, vous avez écrit :
I can't agree with that evaluation. In cases where Python would fail
without this patch (i.e. because the file system encoding cannot be
found during startup),
My patch doesn't change the way how Python get the file system encoding: it
just gets the "Python charset name" (eg. "utf-8" instead of "UTF8",
or "iso8859-1" instead of "latin-1"). The goal was to enable the
optimizations, especially with utf-8. It's not related to bpo-3723.
Activity
tiran commentedon Oct 27, 2008
Python should lower case the file system encoding in Python/pythonrun.c.
On several occasions Python optimizes code paths for lower case
encodings like "utf-8" or "latin-1". On my Ubuntu system the file system
encoding is upper case ("UTF-8") and the optimizations aren't used. This
also causes problems with sub interpreters bpo-3723 initstdio() in the sub
interpreter fails because "UTF-8" must be looked up in the codecs and
encoding registry while "utf-8" works like a charm.
The patch is trivial:
Python/codecs.c:normalizestring() does a similar job. Maybe a new method
"char* PyCodec_NormalizeEncodingName(const char*)" could be introduced
for the problem.
vstinner commentedon Oct 27, 2008
Converting to the lower case doesn't solve the problem: if the locale
is "utf8" and Python checks for "utf-8", the optimization will fail.
Another example: iso-8859-1, latin-1 or latin1?
A correct patch would be to get the most common name of the charset
and make sure that Python C code always use this name.
malemburg commentedon Oct 27, 2008
The lower-casing doesn't hurt, since that's done anyway during codec
lookup, but I'd be -1 on making this try to duplicate the aliasing
already done by the encodings package.
vstinner commentedon Oct 27, 2008
Here is a patch to get the "most common charset name": use
codecs.lookup(codeset).name.
tiran commentedon Oct 27, 2008
Victor's patch fixes the issue with bpo-3723.
malemburg commentedon Oct 27, 2008
+1 on adding Viktor's patch.
tiran commentedon Oct 28, 2008
Me, too! The solution is elegant and works well.
Barry still has to accept the patch, though.
tiran commentedon Oct 30, 2008
Fixed in r67055
loewis commentedon Oct 30, 2008
I can't agree with that evaluation. In cases where Python would fail
without this patch (i.e. because the file system encoding cannot be
found during startup), this solution doesn't work well in general - it
only works if the file system encoding happens to be UTF-8. If the file
system encoding is not in the list of "builtin" codec names, startup
would still fail.
r67057 addresses this case in a somewhat more general manner, by falling
back to ASCII during startup, for encoding file names. This should work
in the usual case where Python is in /usr/bin (say), but it's still
possible to make it fail, e.g. if the codecs are in /home/Питон (say),
on a system that uses koi8-r as the file system encoding, this bug would
persist despite the two patches that have been applied.
vstinner commentedon Oct 30, 2008
Le Thursday 30 October 2008 23:14:21 Martin v. Löwis, vous avez écrit :
My patch doesn't change the way how Python get the file system encoding: it
just gets the "Python charset name" (eg. "utf-8" instead of "UTF8",
or "iso8859-1" instead of "latin-1"). The goal was to enable the
optimizations, especially with utf-8. It's not related to bpo-3723.