You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Per Antoine's request, I wrote this test code, it isn't elegant, I whipped it together quickly; but it shows the issue. The issue may be one of my ignorance, but it does show the behavior I described in bpo-4953. Here's the output from the various test parameters that might be useful in running the test.
c:\python32\python.exe test.py test 1
['c:\\python32\\python.exe', 'test.py', '1']
All OK
c:\python32\python.exe test.py test 2
['c:\\python32\\python.exe', 'test.py', '2']
Not OK: b'abc\r\r\ndef\r\r\n'
c:\python32\python.exe test.py test 3
['c:\\python32\\python.exe', 'test.py', '3']
All OK
c:\python32\python.exe test.py test 4
['c:\\python32\\python.exe', 'test.py', '4']
Not OK: b'abc\r\r\ndef\r\r\n'
c:\python32\python.exe test.py test 1-u
['c:\\python32\\python.exe', '-u', 'test.py', '1-u']
All OK
c:\python32\python.exe test.py test 2-u
['c:\\python32\\python.exe', '-u', 'test.py', '2-u']
All OK
c:\python32\python.exe test.py test 3-u
['c:\\python32\\python.exe', '-u', 'test.py', '3-u']
All OK
c:\python32\python.exe test.py test 4-u
['c:\\python32\\python.exe', '-u', 'test.py', '4-u']
All OK
Note that test 2 and 4, which do not use the mscvrt stuff, have double \r: one sent by the code, and another added, apparently by MSC newline processing. test 2-u and 4-u, which are invoking the subprocess with Python's -u parameter, also do not exhibit the problem, even though the mscvrt stuff is not used. This seems to indicate that Python's -u parameter does approximately the same thing as my windows_binary function.
Seems like if Python already has code for this, that it would be nice to either make it more easily available to the user as an API (like my windows_binary function, invoked with a single line) in the io or sys modules (since it is used to affect sys.std* files).
And it would be nice if the function "worked cross-platform", even if it is a noop on most platforms.
vpython commented on Jan 6, 2011
vpythonmannequin
MannequinAuthor
More actions
tested on Windows, for those that aren't following bpo-4953
What are the results if, instead of piping through subprocess, you simply redirect stdout to a file (using "... > myfile.txt")?
vpython commented on Jan 6, 2011
vpythonmannequin
MannequinAuthor
More actions
The same. This can be tested with the same test program,
c:\python32\python.exe test.py 1 > test1.txt
similar for 2, 3, 4. Then add -u and repeat. All 8 cases produce the same results, either via a pipe, or with a redirected stdout.
vpython commented on Jan 6, 2011
vpythonmannequin
MannequinAuthor
More actions
Actually, it seems like this "-u" behaviour, should simply be the default for Python 3.x on Windows. The new IO subsystem seems to be able to add \r when desired anyway. And except for Notepad, most programs on Windows can deal with \r\n or solo \n anyway. \r\r\n doesn't cause too many problems for very many programs, but is (1) non-standard (2) wasteful of bytes (3) does cause problems for CGI programs, and likely some others... I haven't done a lot of testing with that case, but tried a few programs, and they dealt with it gracefully.
Actually, it seems like this "-u" behaviour, should simply be the
default for Python 3.x on Windows.
There is nothing in "-u" which should explain the behaviour you're
observing, so either way it's a bug. Making "-u" the default is quite
orthogonal.
vpython commented on Jan 6, 2011
vpythonmannequin
MannequinAuthor
More actions
Is there an easy way for me to find the code for -u? I haven't learned my way around the Python sources much, just peeked in modules that I've needed to fix or learn something from a little. I'm just surprised you think it is orthogonal, but I'm glad you agree it is a bug.
Is there an easy way for me to find the code for -u?
"-u" just passes 0 for the "buffering" argument to open() when creating
stdout and stderr. Otherwise "buffering" equals -1.
You can find equivalent code for open() in Lib/_pyio.py (the actual code
in is in C).
The practical difference is that "-u" removes the binary buffering layer
between the unicode layer and the raw file object:
This should make absolutely no difference as to newline handling, since
that is handled exclusively in TextIOWrapper (binary layers are
transparent by construction).
vpython commented on Jan 6, 2011
vpythonmannequin
MannequinAuthor
More actions
I can read and understand C well enough, having coded in it for about 40 years now... but I left C for Perl and Perl for Python, I try not to code in C when I don't have to, these days, as the P languages are more productive, overall.
But there has to be special handling somewhere for opening std*, because they are already open, unlike other files. That is no doubt where the bug is. Can you point me at that code?
But there has to be special handling somewhere for opening std*,
because they are already open, unlike other files. That is no doubt
where the bug is. Can you point me at that code?
I suppose the FileIO in _io is next to look at, wherever it can be found.
I don't get what you're looking for. FileIO is involved in both the
buffered and unbuffered cases, so it shouldn't make a difference.
In any case, please look into Modules/_io
So -u does do 2/3 of what my windows_binary() does :) Should I switch
my test case to use stderr to demonstrate that it doesn't help with
that?
Well, judging by the history of this code, selectively putting -u in
binary mode may be justified by the fact that Python 2 relied on the C
runtime's stdio FILE pointers, and therefore on the C runtime's own
newline translation. I would say that Python 3 should put all stdio fds
in binary mode, regardless of the -u switch.
vpython commented on Jan 6, 2011
vpythonmannequin
MannequinAuthor
More actions
Makes sense to me. Still should document the open file parameter when passed an fd, and either tell the user that it should be O_BINARY, or that it will be O_BINARYd for them, whichever technique is chosen. But having two newline techniques is bad, and if Python thinks it is tracking the file pointer, but Windows is doing newline translation for it, then it isn't likely tracking it correctly for random access IO. So I think the choice should be that any fd passed in to open on Windows should get O_BINARYd immediately.
Attached stdio_binary.patch always set stdin, stdout and stderr in binary mode on Windows (and not only with -u command line flag). I added the following comment, is it correct?
Not sure why I've been added, but I agree that all fds should always be in binary mode in Python 3.
vpython commented on Jan 6, 2011
vpythonmannequin
MannequinAuthor
More actions
Victor, Thanks for your interest and patches.
msg125530 points out the location of the code where _all_ fds could be O_BINARYed, when passed in to open. I think this would make all fds open in binary mode, per Guido's comment... he made exactly the comment I was hoping for, even though I didn't +nosy him... I believe this would catch std* along the way, and render your first patch unnecessary, but your second one would likely still be needed.
New patch setting O_BINARY mode using _setmode() (on Windows) to stdin, stdout, stderr, but also to all FileIO() objects.
The patch on Py_Main() is still needed because Python may use sys.stderr when it is still a stdprinter object, before the final TextIOWrapper object is created (before the io module is loaded).
I don't know the effect of _setmode(stderr, O_BINARY) on calls to fputs(stderr, ...) in Py_FatalError().
I would appreciate a review of parser_translate_newline.patch.
I don't know the effect of _setmode(stderr, O_BINARY) on calls to fputs(stderr, ...) in Py_FatalError().
On Windows it will write lines ending in LF only instead of CRLF. Most
tools to read text files should be able deal with that by now,
conditioned as they are by many years of Unix developers doing Windows
stuff on the side. (Somebody should test what Notepad does though. :-)
I don't know any other supported platforms where O_BINARY exists and
is not a no-op (but I haven't really looked :-).
I commited io_binary_windows.patch + parser_translate_newline.patch as r87824. I fixed the patch on the parser to avoid leak on newtok if translate_newlines() fails.
Activity
vpython commentedon Jan 6, 2011
Per Antoine's request, I wrote this test code, it isn't elegant, I whipped it together quickly; but it shows the issue. The issue may be one of my ignorance, but it does show the behavior I described in bpo-4953. Here's the output from the various test parameters that might be useful in running the test.
Note that test 2 and 4, which do not use the mscvrt stuff, have double \r: one sent by the code, and another added, apparently by MSC newline processing. test 2-u and 4-u, which are invoking the subprocess with Python's -u parameter, also do not exhibit the problem, even though the mscvrt stuff is not used. This seems to indicate that Python's -u parameter does approximately the same thing as my windows_binary function.
Seems like if Python already has code for this, that it would be nice to either make it more easily available to the user as an API (like my windows_binary function, invoked with a single line) in the io or sys modules (since it is used to affect sys.std* files).
And it would be nice if the function "worked cross-platform", even if it is a noop on most platforms.
vpython commentedon Jan 6, 2011
tested on Windows, for those that aren't following bpo-4953
pitrou commentedon Jan 6, 2011
What are the results if, instead of piping through subprocess, you simply redirect stdout to a file (using "... > myfile.txt")?
vpython commentedon Jan 6, 2011
The same. This can be tested with the same test program,
c:\python32\python.exe test.py 1 > test1.txt
similar for 2, 3, 4. Then add -u and repeat. All 8 cases produce the same results, either via a pipe, or with a redirected stdout.
vpython commentedon Jan 6, 2011
Actually, it seems like this "-u" behaviour, should simply be the default for Python 3.x on Windows. The new IO subsystem seems to be able to add \r when desired anyway. And except for Notepad, most programs on Windows can deal with \r\n or solo \n anyway. \r\r\n doesn't cause too many problems for very many programs, but is (1) non-standard (2) wasteful of bytes (3) does cause problems for CGI programs, and likely some others... I haven't done a lot of testing with that case, but tried a few programs, and they dealt with it gracefully.
pitrou commentedon Jan 6, 2011
There is nothing in "-u" which should explain the behaviour you're
observing, so either way it's a bug. Making "-u" the default is quite
orthogonal.
vpython commentedon Jan 6, 2011
Is there an easy way for me to find the code for -u? I haven't learned my way around the Python sources much, just peeked in modules that I've needed to fix or learn something from a little. I'm just surprised you think it is orthogonal, but I'm glad you agree it is a bug.
pitrou commentedon Jan 6, 2011
"-u" just passes 0 for the "buffering" argument to open() when creating
stdout and stderr. Otherwise "buffering" equals -1.
You can find equivalent code for open() in Lib/_pyio.py (the actual code
in is in C).
The practical difference is that "-u" removes the binary buffering layer
between the unicode layer and the raw file object:
This should make absolutely no difference as to newline handling, since
that is handled exclusively in TextIOWrapper (binary layers are
transparent by construction).
vpython commentedon Jan 6, 2011
I can read and understand C well enough, having coded in it for about 40 years now... but I left C for Perl and Perl for Python, I try not to code in C when I don't have to, these days, as the P languages are more productive, overall.
But there has to be special handling somewhere for opening std*, because they are already open, unlike other files. That is no doubt where the bug is. Can you point me at that code?
pitrou commentedon Jan 6, 2011
See initstdio() in Python/pythonrun.c
vpython commentedon Jan 6, 2011
I suppose the FileIO in _io is next to look at, wherever it can be found.
pitrou commentedon Jan 6, 2011
I don't get what you're looking for. FileIO is involved in both the
buffered and unbuffered cases, so it shouldn't make a difference.
In any case, please look into Modules/_io
4 remaining items
pitrou commentedon Jan 6, 2011
Well, judging by the history of this code, selectively putting -u in
binary mode may be justified by the fact that Python 2 relied on the C
runtime's stdio FILE pointers, and therefore on the C runtime's own
newline translation. I would say that Python 3 should put all stdio fds
in binary mode, regardless of the -u switch.
vpython commentedon Jan 6, 2011
Makes sense to me. Still should document the open file parameter when passed an fd, and either tell the user that it should be O_BINARY, or that it will be O_BINARYd for them, whichever technique is chosen. But having two newline techniques is bad, and if Python thinks it is tracking the file pointer, but Windows is doing newline translation for it, then it isn't likely tracking it correctly for random access IO. So I think the choice should be that any fd passed in to open on Windows should get O_BINARYd immediately.
vstinner commentedon Jan 6, 2011
Attached stdio_binary.patch always set stdin, stdout and stderr in binary mode on Windows (and not only with -u command line flag). I added the following comment, is it correct?
vstinner commentedon Jan 6, 2011
Using my patch, the interpreter fails with a SyntaxError on \r: attached patch translates newlines to avoid this problem.
vstinner commentedon Jan 6, 2011
See also issue bpo-4628.
gvanrossum commentedon Jan 6, 2011
Not sure why I've been added, but I agree that all fds should always be in binary mode in Python 3.
vpython commentedon Jan 6, 2011
Victor, Thanks for your interest and patches.
msg125530 points out the location of the code where _all_ fds could be O_BINARYed, when passed in to open. I think this would make all fds open in binary mode, per Guido's comment... he made exactly the comment I was hoping for, even though I didn't +nosy him... I believe this would catch std* along the way, and render your first patch unnecessary, but your second one would likely still be needed.
vstinner commentedon Jan 6, 2011
New patch setting O_BINARY mode using _setmode() (on Windows) to stdin, stdout, stderr, but also to all FileIO() objects.
The patch on Py_Main() is still needed because Python may use sys.stderr when it is still a stdprinter object, before the final TextIOWrapper object is created (before the io module is loaded).
I don't know the effect of _setmode(stderr, O_BINARY) on calls to fputs(stderr, ...) in Py_FatalError().
I would appreciate a review of parser_translate_newline.patch.
gvanrossum commentedon Jan 6, 2011
On Thu, Jan 6, 2011 at 1:20 PM, STINNER Victor <report@bugs.python.org> wrote:
On Windows it will write lines ending in LF only instead of CRLF. Most
tools to read text files should be able deal with that by now,
conditioned as they are by many years of Unix developers doing Windows
stuff on the side. (Somebody should test what Notepad does though. :-)
I don't know any other supported platforms where O_BINARY exists and
is not a no-op (but I haven't really looked :-).
vstinner commentedon Jan 7, 2011
I commited io_binary_windows.patch + parser_translate_newline.patch as r87824. I fixed the patch on the parser to avoid leak on newtok if translate_newlines() fails.
vstinner commentedon Jan 7, 2011
Tests pass on Windows 7 buildbot, the two other XP buildbots have unrelated issues. I also tested on my XP VM: all tests pass. So I close this issue.
vpython commentedon Jan 7, 2011
Thanks for your work on this Victor, and other commenters also.
vstinner commentedon Aug 3, 2012
For the record, this change (always set stdout and stderr in binary mode on Windows) introduced (at least???) the following regressions: