You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In some corner cases io.BufferedWriter raises an io.BlockingIOError
"lying" about the number of characters written.
I've added some tests and a small change to fix this issue.
I was first surprised by the fact that e.characters_written is not used
in the write() method; but _flush_unlocked() already adjusts the
_write_buf according to the original e.characters_written raised by the
underlying raw file. Everything is fine.
I suggest however to add some tests around the first "except
BlockingIOError". This would answer the question:
# XXX Why not just let the exception pass through?
For example, I modified a function in your patch:
# Subsequent calls to write() try to flush the raw file.
try:
bufio.write(b"x")
except io.BlockingIOError as e:
# Two more chars were written at the raw level
self.assertEqual(bufio._write_buf, write_buf[2:])
# But write() did not accept anything.
self.assertEqual(e.characters_written, 0)
else:
self.fail("BlockingIOError not raised")
severb commented on Nov 6, 2008
severbmannequin
MannequinAuthor
More actions
Thanks for your review, here's a new patch.
I've added a new test for the pre-flush condition and made the comments
less cryptic.
We have discussed this bug in the python developer chat yesterday. I
decided to wait until after the 3.0.0 release. The problem is not
critical enough for 3.0.0. I like to keep the amount of changes during
the RC phase to a minimum.
I do concur with the desire to restrict changes during RC phase. Do this
also mean that merges from trunk will be reduced to the strict minimum?
No global merge, only on a revision basis after review.
In this case we could apply the patch to the trunk, and let a future
global merge propagate the change to py3k.
severb commented on Dec 7, 2008
severbmannequin
MannequinAuthor
More actions
Christian, if the patch looks good to you I think now it's a good time
to commit it.
Here is a patch which replaces testWriteNonBlocking with a reasonable
implementation-independent test (it works with e.g. the io-c sandbox).
The new test also checks for the current problem, i.e. it passes with
the fix to io.py and fails without.
severb commented on Dec 31, 2008
severbmannequin
MannequinAuthor
More actions
Thanks for the new implementation of MockNonBlockWriterIO class. It
makes tests so much easier to read.
There are some minor things in your patch that I would change. For example:
# 1 byte will be written, the rest will be bufferedraw.block_on(b"k")
self.assertEquals(bufio.write(b"jklmn"), 5)
# ...raw.block_on(b"0")
The comment is misleading because in fact no byte is written at raw
level. That's because the data size is smaller than the buffer size and
the buffer is empty (was emptied by the last write call). All this
renders raw.block_on(b"k") call useless. I also think this is the
correct behavior regardless of implementation language of BufferedWriter
class i.e. no write call should write at raw level smaller chunks of
data than buffer's size unless it has to.
Your tests can't cover the pre-flush condition because max_buffer_size
equals buffer_size.
Unless you'll beat me to it or prove me wrong, I'll update your patch
next year.
The comment is misleading because in fact no byte is written at raw
level. That's because the data size is smaller than the buffer size and
the buffer is empty (was emptied by the last write call).
It depends on the implementation. A different implementation may use a
different algorithm.
I also think this is the
correct behavior regardless of implementation language of BufferedWriter
class i.e. no write call should write at raw level smaller chunks of
data than buffer's size unless it has to.
But how do you decide when it "has to"? Unless you want to constrain the
exact implemented algorithm, you can't do that in your tests.
severb commented on Dec 31, 2008
severbmannequin
MannequinAuthor
More actions
> The comment is misleading because in fact no byte is written at raw
> level. That's because the data size is smaller than the buffer size and
> the buffer is empty (was emptied by the last write call).
It depends on the implementation. A different implementation may use a
different algorithm.
I feel that no matter what implementation algorithm BufferedWriter uses
it shouldn't write smaller chunks of data than buffer's size or else the
buffer is useless.
> I also think this is the
> correct behavior regardless of implementation language of BufferedWriter
> class i.e. no write call should write at raw level smaller chunks of
> data than buffer's size unless it has to.
But how do you decide when it "has to"? Unless you want to constrain the
exact implemented algorithm, you can't do that in your tests.
When a direct or indirect (e.g. on close) flush is called for the file
object.
I feel that no matter what implementation algorithm BufferedWriter uses
it shouldn't write smaller chunks of data than buffer's size or else the
buffer is useless.
If you rewrite the above sentence using the word "statistically", then I
can agree :)
But if I look at e.g. the fwrite() manpage, I see no guarantee that the
stdio layer will never make a call to write() with a size smaller than
the buffer size. The buffered layer should be free to manage its buffer
in what it believes is the most efficient way. The only guarantee is
that it won't buffer more than max_buffer_size.
Anyway :) Practically, the test does work on both py3k and another
implementation, so I don't see any urgency to remove anything from it.
severb commented on Jan 7, 2009
severbmannequin
MannequinAuthor
More actions
Anyway :) Practically, the test does work on both py3k and another
implementation, so I don't see any urgency to remove anything from it.
Indeed, it doesn't hurt keeping it.
For completeness' sake I've updated your tests to cover the pre-flush
condition existent in the python implementation. Theoretically this
should be the desired behavior in any other implementation.
Activity
severb commentedon Nov 5, 2008
In some corner cases io.BufferedWriter raises an io.BlockingIOError
"lying" about the number of characters written.
I've added some tests and a small change to fix this issue.
amauryfa commentedon Nov 6, 2008
The patch is good.
I was first surprised by the fact that e.characters_written is not used
in the write() method; but _flush_unlocked() already adjusts the
_write_buf according to the original e.characters_written raised by the
underlying raw file. Everything is fine.
I suggest however to add some tests around the first "except
BlockingIOError". This would answer the question:
# XXX Why not just let the exception pass through?
For example, I modified a function in your patch:
severb commentedon Nov 6, 2008
Thanks for your review, here's a new patch.
I've added a new test for the pre-flush condition and made the comments
less cryptic.
tiran commentedon Nov 6, 2008
We have discussed this bug in the python developer chat yesterday. I
decided to wait until after the 3.0.0 release. The problem is not
critical enough for 3.0.0. I like to keep the amount of changes during
the RC phase to a minimum.
amauryfa commentedon Nov 6, 2008
I do concur with the desire to restrict changes during RC phase. Do this
also mean that merges from trunk will be reduced to the strict minimum?
No global merge, only on a revision basis after review.
In this case we could apply the patch to the trunk, and let a future
global merge propagate the change to py3k.
severb commentedon Dec 7, 2008
Christian, if the patch looks good to you I think now it's a good time
to commit it.
pitrou commentedon Dec 30, 2008
The tests should be written so as not to rely on internal implementation
details (the _write_buf attribute).
pitrou commentedon Dec 30, 2008
Here is a patch which replaces testWriteNonBlocking with a reasonable
implementation-independent test (it works with e.g. the io-c sandbox).
The new test also checks for the current problem, i.e. it passes with
the fix to io.py and fails without.
severb commentedon Dec 31, 2008
Thanks for the new implementation of MockNonBlockWriterIO class. It
makes tests so much easier to read.
There are some minor things in your patch that I would change. For example:
The comment is misleading because in fact no byte is written at raw
level. That's because the data size is smaller than the buffer size and
the buffer is empty (was emptied by the last write call). All this
renders raw.block_on(b"k") call useless. I also think this is the
correct behavior regardless of implementation language of BufferedWriter
class i.e. no write call should write at raw level smaller chunks of
data than buffer's size unless it has to.
Your tests can't cover the pre-flush condition because max_buffer_size
equals buffer_size.
Unless you'll beat me to it or prove me wrong, I'll update your patch
next year.
pitrou commentedon Dec 31, 2008
It depends on the implementation. A different implementation may use a
different algorithm.
But how do you decide when it "has to"? Unless you want to constrain the
exact implemented algorithm, you can't do that in your tests.
severb commentedon Dec 31, 2008
I feel that no matter what implementation algorithm BufferedWriter uses
it shouldn't write smaller chunks of data than buffer's size or else the
buffer is useless.
When a direct or indirect (e.g. on close) flush is called for the file
object.
pitrou commentedon Dec 31, 2008
If you rewrite the above sentence using the word "statistically", then I
can agree :)
But if I look at e.g. the fwrite() manpage, I see no guarantee that the
stdio layer will never make a call to write() with a size smaller than
the buffer size. The buffered layer should be free to manage its buffer
in what it believes is the most efficient way. The only guarantee is
that it won't buffer more than max_buffer_size.
Anyway :) Practically, the test does work on both py3k and another
implementation, so I don't see any urgency to remove anything from it.
severb commentedon Jan 7, 2009
For completeness' sake I've updated your tests to cover the pre-flush
condition existent in the python implementation. Theoretically this
should be the desired behavior in any other implementation.
benjaminp commentedon Feb 28, 2009
This has been cured in the io-c branch.
benjaminp commentedon Mar 4, 2009
This has been fixed in io-c branch. (r70152)
severb commentedon Mar 5, 2009
Looks like the test covering the pre-flush condition is missing.
benjaminp commentedon Mar 5, 2009
2009/3/5 Sever Băneșiu <report@bugs.python.org>:
That test is no longer applicable because max_buffer_size is
deprecated and unused.