You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
compile() doesn't ignore the source encoding when a string is passed in #48876
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee='https://github.com/benjaminp'closed_at=<Date2009-03-02.23:32:33.644>created_at=<Date2008-12-11.06:19:16.441>labels= ['interpreter-core', 'type-bug']
title="compile() doesn't ignore the source encoding when a string is passed in"updated_at=<Date2009-03-30.19:26:09.596>user='https://github.com/brettcannon'
When compile() is called with a string it is a reasonable assumption
that it has already been decoded. But this is not in fact the case and
leads to errors when trying to use non-ASCII identifiers::
>>> source = "# coding=latin-1\n\u00c6 = '\u00c6'"
>>> compile(source, '<test>', 'exec')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<test>", line 2
� = '�'
^
SyntaxError: invalid character in identifier
>>> compile(source.encode('latin-1'), '<test>', 'exec')
<code object <module> at 0x389cc8, file "<test>", line 2>
>>> source =b"# coding=cp1252\n\x94 = '\x94'".decode('cp1252')
>>> compile(source, '<test>', 'exec')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<test>", line 0
SyntaxError: unknown encoding: cp1252
The real error here is masked; just before the exception is set, there
is a pending
SyntaxError: 'charmap' codec can't decode byte 0x9d in position 18:
character maps to <undefined>
It seems that the source internal representation is correct utf-8, but
this is decoded again with the "coding=" cookie.
Here is what I have found out so far. Python/bltinmodule.c:builtin_compile takes in a PyObject and gets the
char * representation of that object and passes it to Python/pythonrun.c:Py_CompileStringFlags. Unfortunately no other
information is passed along in the call, including what the encoding
happens to be. This is unfortunate as builtin_compile makes sure that
the char* data is encoded using the default encoding before calling
Py_CompileStringFlags.
I just tried setting a PyCF flag to denote that the char* data is
encoded using the default encoding, but Parser/tokenizer.c is not
compiled against unicodeobject.c and thus one cannot use
PyUnicode_GetDefaultEncoding() to know what the data is stored as.
I'm going to try to explicitly convert to UTF-8 and see if that works.
The function decode_str() (Parser/tokenizer.c) is responsible to
detect the encoding using the BOM or the cookie ("coding: xxx").
decode_str() reencodes also the text to utf-8 if the encoding is
different than utf-8. I think that we can just skip this function if
the input text is already unicode (utf-8). Attached patch implements
this idea.
The patch introduces a new compiler flag (PyCF_IGNORE_COOKIE) and a
new parser flag (PyPARSE_IGNORE_COOKIE). The new compiler flag is set
by source_as_string() when the input is a PyUnicode object. "Ignore
cookie" is maybe not the best name for this flag.
With my patch, the first Brett's example displays:
$ ./python com2.py
Traceback (most recent call last):
File "com2.py", line 3, in<module>compile(source, '<test>', 'exec')
File "<test>", line 2
” ='”'^
SyntaxError: invalid character in identifier
The error cursor is not at the right column (bug related to the issue
2382 or introduced by my patch?).
The patch changes the public API: PyTokenizer_FromString() prototype
changed to get a new argument. I don't like changing public API. The
new argument should be a bit vector (flags) instead of a single bit
(ignore_cookie). We can avoid changing the public API by creating a
new function (eg. "PyTokenizer_FromUnicode" ;-)).
There are some old PyPARSE_xxx constants in Include/parsetok.h that
might be removed. PyPARSE_WITH_IS_KEYWORD value is 3 which is strange
since flags is a bit vector (changed with | and tested by &). But
PyPARSE_WITH_IS_KEYWORD is a dead constant (written in #if
0...#endif).
I will see when I can get to it (other stuff is taking priority). Not
going to assign to myself quite yet in case someone wants to beat me to
this. I won't lose track since I created the bug and have a saved query
to look at all bugs I make.
I don't like the change of API to PyTokenizer_FromString. I would prefer
another function like PyTokenizer_IgnoreCodingCookie() blows up when
parsing has already started.
The (char *) cast in PyTokenizer_FromString is unneeded.
You need to indent the "else" clause after you test for ignore_cookie.
I'd like to see a test that shows that byte strings still have their
cookies examined.
I don't like the change of API to PyTokenizer_FromString.
I would prefer another function like
PyTokenizer_IgnoreCodingCookie()
Ok, I created a new function PyTokenizer_FromUnicode(). I
choosed "FromUnicode" because the string is encoded in unicode (as
UTF-8, even if it's not the wchar_t* type).
The (char *) cast in PyTokenizer_FromString is unneeded.
The cast on the decode_str() result? It was already present in the
original code. I removed it in my new patch.
You need to indent the "else" clause after you test for
ignore_cookie.
Ooops, I always have problems to generate a diff because my editor
removes trailing spaces and then I have to ignore space changes to
create the diff.
I'd like to see a test that shows that byte strings still have their
cookies examined.
test_pep263 has already two tests using a "#coding:" header.
What should be the encoding and/or the type (str, bytes) of the "source"
string? Taking into account the encoding of the script which contains
this code, I have the feeling there is always something going wrong,
this can be a "non ascii" char in the source (encoded in utf-8!) or the
interactive interpreter does not accept very well a byte string
representing a utf-8 encoded string.
IDLE is not suffering from this. Its interactive interpreter is somehow
receiving "ucs-2 ready string" from tkinter.
I'm a little bit confused here (win2k, winXP sp2, Python 3.0.1).
I'm glad to have discovered this topic. I bumped into something similar
when I toyed with an interactive interpreter.
from code import InteractiveInterpreter
ii = InteractiveInterpreter()
source = ...
ii.runsource(source)
What should be the encoding and/or the type (str, bytes) of the "source"
string?
Off the top of my head it should be UTF-8. Otherwise it can probably be
bytes as long as it has universal newlines.
jmfauth commented on Mar 25, 2009
jmfauthmannequin
Mannequin
More actions
When I was preparing some test examples to be submitted here.
I noticed the module codeop.py used by the InteractiveInterpreter,
does not like byte strings very much.
IDLE, Python 3.0.1, winxp sp2
>>> source =b'print(999)'
>>> compile(source, '<in>', 'exec')
<code object <module> at 0x00AA5CC8, file "<in>", line 1>
>>> r =compile(source, '<in>', 'exec')
>>> exec(r)
999
>>> from code import InteractiveInterpreter
>>> ii = InteractiveInterpreter()
>>> ii.runsource(source)
Traceback (most recent call last):
File "<pyshell#6>", line 1, in <module>
ii.runsource(source)
File "C:\Python30\lib\code.py", line 63, in runsource
code = self.compile(source, filename, symbol)
File "C:\Python30\lib\codeop.py", line 168, in __call__
return _maybe_compile(self.compiler, source, filename, symbol)
File "C:\Python30\lib\codeop.py", line 70, in _maybe_compile
for line in source.split("\n"):
TypeError: Type str doesn't support the buffer API
>>>
>>> source ='print(999)'
>>> ii.runsource(source)
999
False
@jmfauth: Can you open a different issue for the IDLE issue?
jmfauth commented on Mar 25, 2009
jmfauthmannequin
Mannequin
More actions
Victor
Yes, I could, but I think this is not an IDLE issue, I'm just
using IDLE to illustrate the problem. I got the same when I'm
working from within an editor or with my interactive interpreter
I wrote for the fun.
>c:\python30\pythonw -u "uuu.py"
Traceback (most recent call last):
File "uuu.py", line 8, in <module>
ii.runsource(source)
File "c:\python30\lib\code.py", line 63, in runsource
code =self.compile(source, filename, symbol)
File "c:\python30\lib\codeop.py", line 168, in __call__return _maybe_compile(self.compiler, source, filename, symbol)
File "c:\python30\lib\codeop.py", line 70, in _maybe_compilefor line in source.split("\n"):
TypeError: Type str doesn't support the buffer API
>Exit code: 1
In a few words, my empirical understanding of the story.
Things are in good shape, but Python, itsself and its
components (compile, interactiveinterpreter module, runsource(),
exec(), ...) are lacking in consistency, the all accept
miscellanous arguments type or they are not strict enough in
their arguments acceptance. When they accept a str, which
encoding is accepted?
Python 3 is now using unicode as str. Of course, this is welcome,
but the caveat is that there are now two encodings in use. Code source
defaults to utf-8, but the str in code defaults to ucs-2/4 (eg. in
IDLE).
Maybe a solution is to have an optional encoding argument as we now
have everywhere eg. str(), open(), which defaults to one encoding.
Yes, I could, but I think this is not an IDLE issue
(...)
File "uuu.py", line 8, in <module>
ii.runsource(source)
(...)
File "c:\python30\lib\codeop.py", line 70, in _maybe_compile
for line in source.split("\n"):
TypeError: Type str doesn't support the buffer API
compile() works as expected. Your problem is related to
InteractiveInterpreter().runsource(source) which is part of IDLE. runsource()
is not compatible with bytes, only 'str' type is accepted.
The error comes from bytes.split(str): _maybe_compile() should use
source.split(b'\n') if source type is bytes. Or runsource() should reject
bytes directly.
issue bpo-4628: No universal newline support for compile() when using bytes
jmfauth commented on Mar 30, 2009
jmfauthmannequin
Mannequin
More actions
Quick feedback from a Windows user.
I made a few more tests with the freshly installed Pyton 3.1a1. The
compile() function is running very well.
As a side effect, it now possible to write an "execfile()" without
any problem. Tests with execfile() versions reading files as text
or as binary, with/without coding cookies, BOM, cp1252, cp850, cp437,
utf-8 cookie, utf-8 with bom, ....
(Of course, taking in account and managing universal newline).
Activity
brettcannon commentedon Dec 11, 2008
When compile() is called with a string it is a reasonable assumption
that it has already been decoded. But this is not in fact the case and
leads to errors when trying to use non-ASCII identifiers::
>>> source = "# coding=latin-1\n\u00c6 = '\u00c6'" >>> compile(source, '<test>', 'exec') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<test>", line 2 � = '�' ^ SyntaxError: invalid character in identifier >>> compile(source.encode('latin-1'), '<test>', 'exec') <code object <module> at 0x389cc8, file "<test>", line 2>amauryfa commentedon Dec 30, 2008
bpo-4742 is similar issue:
The real error here is masked; just before the exception is set, there
is a pending
SyntaxError: 'charmap' codec can't decode byte 0x9d in position 18:
character maps to <undefined>
It seems that the source internal representation is correct utf-8, but
this is decoded again with the "coding=" cookie.
brettcannon commentedon Jan 3, 2009
Here is what I have found out so far.
Python/bltinmodule.c:builtin_compile takes in a PyObject and gets the
char * representation of that object and passes it to
Python/pythonrun.c:Py_CompileStringFlags. Unfortunately no other
information is passed along in the call, including what the encoding
happens to be. This is unfortunate as builtin_compile makes sure that
the char* data is encoded using the default encoding before calling
Py_CompileStringFlags.
I just tried setting a PyCF flag to denote that the char* data is
encoded using the default encoding, but Parser/tokenizer.c is not
compiled against unicodeobject.c and thus one cannot use
PyUnicode_GetDefaultEncoding() to know what the data is stored as.
I'm going to try to explicitly convert to UTF-8 and see if that works.
brettcannon commentedon Jan 3, 2009
So explicitly converting to UTF-8 didn't work, or at least as simply as
I had hoped.
vstinner commentedon Jan 5, 2009
The function decode_str() (Parser/tokenizer.c) is responsible to
detect the encoding using the BOM or the cookie ("coding: xxx").
decode_str() reencodes also the text to utf-8 if the encoding is
different than utf-8. I think that we can just skip this function if
the input text is already unicode (utf-8). Attached patch implements
this idea.
The patch introduces a new compiler flag (PyCF_IGNORE_COOKIE) and a
new parser flag (PyPARSE_IGNORE_COOKIE). The new compiler flag is set
by source_as_string() when the input is a PyUnicode object. "Ignore
cookie" is maybe not the best name for this flag.
With my patch, the first Brett's example displays: $ ./python com2.py Traceback (most recent call last): File "com2.py", line 3, in <module> compile(source, '<test>', 'exec') File "<test>", line 2 ” = '”' ^ SyntaxError: invalid character in identifierThe error cursor is not at the right column (bug related to the issue
2382 or introduced by my patch?).
The patch changes the public API: PyTokenizer_FromString() prototype
changed to get a new argument. I don't like changing public API. The
new argument should be a bit vector (flags) instead of a single bit
(ignore_cookie). We can avoid changing the public API by creating a
new function (eg. "PyTokenizer_FromUnicode" ;-)).
There are some old PyPARSE_xxx constants in Include/parsetok.h that
might be removed. PyPARSE_WITH_IS_KEYWORD value is 3 which is strange
since flags is a bit vector (changed with | and tested by &). But
PyPARSE_WITH_IS_KEYWORD is a dead constant (written in #if
0...#endif).
vstinner commentedon Jan 5, 2009
Oops, I attached the wrong file :-p
vstinner commentedon Jan 5, 2009
I tried py3k_adjust_cursor_at_syntax_error_v2.patch (issue bpo-2382) and the
cursor is displayed at the right column:
$ ./python com2.py Traceback (most recent call last): ... File "<test>", line 2 ” = '”' ^So it's not a new bug ;-)
vstinner commentedon Jan 17, 2009
New version of my patch: add a regression test.
@brett.cannon: Could you review my patch?
brettcannon commentedon Jan 17, 2009
I will see when I can get to it (other stuff is taking priority). Not
going to assign to myself quite yet in case someone wants to beat me to
this. I won't lose track since I created the bug and have a saved query
to look at all bugs I make.
vstinner commentedon Jan 26, 2009
Ping! Can anyone review my patch?
benjaminp commentedon Jan 27, 2009
I don't like the change of API to PyTokenizer_FromString. I would prefer
another function like PyTokenizer_IgnoreCodingCookie() blows up when
parsing has already started.
The (char *) cast in PyTokenizer_FromString is unneeded.
You need to indent the "else" clause after you test for ignore_cookie.
I'd like to see a test that shows that byte strings still have their
cookies examined.
amauryfa commentedon Jan 27, 2009
Note that PyTokenizer_FromString is not an API function: it's not marked
with PyAPI_FUNC, and not exported from the windows DLL.
vstinner commentedon Jan 29, 2009
Ok, I created a new function PyTokenizer_FromUnicode(). I
choosed "FromUnicode" because the string is encoded in unicode (as
UTF-8, even if it's not the wchar_t* type).
The cast on the decode_str() result? It was already present in the
original code. I removed it in my new patch.
Ooops, I always have problems to generate a diff because my editor
removes trailing spaces and then I have to ignore space changes to
create the diff.
test_pep263 has already two tests using a "#coding:" header.
4 remaining items
vstinner commentedon Feb 26, 2009
Yeah!
Anyone to review and/or commit the last patch?
benjaminp commentedon Feb 27, 2009
I'll deal with it eventually.
benjaminp commentedon Mar 2, 2009
Fixed in r70112.
benjaminp commentedon Mar 2, 2009
Should this be backported?
vstinner commentedon Mar 3, 2009
It's the r70113 (not the 70112). I see that pitrou backported the fix
to 3.0.x. I think that it's enough, 2.x doesn't require the fix.
jmfauth commentedon Mar 24, 2009
I'm glad to have discovered this topic. I bumped into something similar
when I toyed with an interactive interpreter.
What should be the encoding and/or the type (str, bytes) of the "source"
string? Taking into account the encoding of the script which contains
this code, I have the feeling there is always something going wrong,
this can be a "non ascii" char in the source (encoded in utf-8!) or the
interactive interpreter does not accept very well a byte string
representing a utf-8 encoded string.
IDLE is not suffering from this. Its interactive interpreter is somehow
receiving "ucs-2 ready string" from tkinter.
I'm a little bit confused here (win2k, winXP sp2, Python 3.0.1).
brettcannon commentedon Mar 24, 2009
On Tue, Mar 24, 2009 at 09:24, Jean-Michel Fauth <report@bugs.python.org>wrote:
Off the top of my head it should be UTF-8. Otherwise it can probably be
bytes as long as it has universal newlines.
jmfauth commentedon Mar 25, 2009
When I was preparing some test examples to be submitted here.
I noticed the module codeop.py used by the InteractiveInterpreter,
does not like byte strings very much.
IDLE, Python 3.0.1, winxp sp2
vstinner commentedon Mar 25, 2009
@jmfauth: Can you open a different issue for the IDLE issue?
jmfauth commentedon Mar 25, 2009
Yes, I could, but I think this is not an IDLE issue, I'm just
using IDLE to illustrate the problem. I got the same when I'm
working from within an editor or with my interactive interpreter
I wrote for the fun.
Code in the editor:
# -- coding: cp1252 --
# Python 3.0.1, winxp sp2
Output:
My interactive interpreter
=======================
I realised and missed the fact the str() function is now accepting
an encoding argument (very good). With it, the above code works.
Code in the editor:
# -- coding: cp1252 --
# Python 3.0.1, winxp sp2
Output: (ok)
=======================
In a few words, my empirical understanding of the story.
Things are in good shape, but Python, itsself and its
components (compile, interactiveinterpreter module, runsource(),
exec(), ...) are lacking in consistency, the all accept
miscellanous arguments type or they are not strict enough in
their arguments acceptance. When they accept a str, which
encoding is accepted?
Python 3 is now using unicode as str. Of course, this is welcome,
but the caveat is that there are now two encodings in use. Code source
defaults to utf-8, but the str in code defaults to ucs-2/4 (eg. in
IDLE).
Maybe a solution is to have an optional encoding argument as we now
have everywhere eg. str(), open(), which defaults to one encoding.
I suspect the miscellaneous discussions one finds from people attempting
to write a "correct" execfile() for Python 3 are coming from this.
Regards.
vstinner commentedon Mar 25, 2009
compile() works as expected. Your problem is related to
InteractiveInterpreter().runsource(source) which is part of IDLE. runsource()
is not compatible with bytes, only 'str' type is accepted.
The error comes from bytes.split(str): _maybe_compile() should use
source.split(b'\n') if source type is bytes. Or runsource() should reject
bytes directly.
Yes, runsource() (only) works with the str type.
Please see issues:
jmfauth commentedon Mar 30, 2009
Quick feedback from a Windows user.
I made a few more tests with the freshly installed Pyton 3.1a1. The
compile() function is running very well.
As a side effect, it now possible to write an "execfile()" without
any problem. Tests with execfile() versions reading files as text
or as binary, with/without coding cookies, BOM, cp1252, cp850, cp437,
utf-8 cookie, utf-8 with bom, ....
(Of course, taking in account and managing universal newline).