Skip to content

BufferedWriter non-blocking overage #48513

Closed

Description

@severb
mannequin
BPO 4263
Nosy @amauryfa, @pitrou, @tiran, @benjaminp
Files
  • bw_overage.diff
  • bw_overage2.diff
  • nonblocking.patch
  • nonblocking2.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2009-03-04.21:31:12.964>
    created_at = <Date 2008-11-05.18:22:29.164>
    labels = ['type-bug', 'library']
    title = 'BufferedWriter non-blocking overage'
    updated_at = <Date 2009-03-05.22:32:04.973>
    user = 'https://bugs.python.org/severb'

    bugs.python.org fields:

    activity = <Date 2009-03-05.22:32:04.973>
    actor = 'benjamin.peterson'
    assignee = 'none'
    closed = True
    closed_date = <Date 2009-03-04.21:31:12.964>
    closer = 'benjamin.peterson'
    components = ['Library (Lib)']
    creation = <Date 2008-11-05.18:22:29.164>
    creator = 'severb'
    dependencies = []
    files = ['11945', '11953', '12498', '12636']
    hgrepos = []
    issue_num = 4263
    keywords = ['patch']
    message_count = 17.0
    messages = ['75523', '75563', '75564', '75565', '75566', '77218', '78546', '78548', '78599', '78601', '78606', '78608', '79324', '82922', '83137', '83185', '83224']
    nosy_count = 5.0
    nosy_names = ['amaury.forgeotdarc', 'pitrou', 'christian.heimes', 'benjamin.peterson', 'severb']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'patch review'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue4263'
    versions = ['Python 2.6', 'Python 3.0', 'Python 2.7']

    Activity

    1. severb commented on Nov 5, 2008

      severbmannequin
      MannequinAuthor

      In some corner cases io.BufferedWriter raises an io.BlockingIOError
      "lying" about the number of characters written.
      I've added some tests and a small change to fix this issue.

    2. added
      stdlibStandard Library Python modules in the Lib/ directory
      on Nov 5, 2008
    3. amauryfa commented on Nov 6, 2008

      @amauryfa
      Contributor

      The patch is good.

      I was first surprised by the fact that e.characters_written is not used
      in the write() method; but _flush_unlocked() already adjusts the
      _write_buf according to the original e.characters_written raised by the
      underlying raw file. Everything is fine.

      I suggest however to add some tests around the first "except
      BlockingIOError". This would answer the question:
      # XXX Why not just let the exception pass through?
      For example, I modified a function in your patch:

          def testWriteNonBlockingOverage(self):
              raw = MockNonBlockWriterIO((-1, -2))
              [...]
          # Subsequent calls to write() try to flush the raw file.
          try:
              bufio.write(b"x")
          except io.BlockingIOError as e:
              # Two more chars were written at the raw level
              self.assertEqual(bufio._write_buf, write_buf[2:])
              # But write() did not accept anything.
              self.assertEqual(e.characters_written, 0)
          else:
              self.fail("BlockingIOError not raised")
      
    4. severb commented on Nov 6, 2008

      severbmannequin
      MannequinAuthor

      Thanks for your review, here's a new patch.
      I've added a new test for the pre-flush condition and made the comments
      less cryptic.

    5. tiran commented on Nov 6, 2008

      @tiran
      Member

      We have discussed this bug in the python developer chat yesterday. I
      decided to wait until after the 3.0.0 release. The problem is not
      critical enough for 3.0.0. I like to keep the amount of changes during
      the RC phase to a minimum.

    6. added
      type-bugAn unexpected behavior, bug, or error
      on Nov 6, 2008
    7. amauryfa commented on Nov 6, 2008

      @amauryfa
      Contributor

      I do concur with the desire to restrict changes during RC phase. Do this
      also mean that merges from trunk will be reduced to the strict minimum?
      No global merge, only on a revision basis after review.

      In this case we could apply the patch to the trunk, and let a future
      global merge propagate the change to py3k.

    8. severb commented on Dec 7, 2008

      severbmannequin
      MannequinAuthor

      Christian, if the patch looks good to you I think now it's a good time
      to commit it.

    9. pitrou commented on Dec 30, 2008

      @pitrou
      Member

      The tests should be written so as not to rely on internal implementation
      details (the _write_buf attribute).

    10. pitrou commented on Dec 30, 2008

      @pitrou
      Member

      Here is a patch which replaces testWriteNonBlocking with a reasonable
      implementation-independent test (it works with e.g. the io-c sandbox).
      The new test also checks for the current problem, i.e. it passes with
      the fix to io.py and fails without.

    11. severb commented on Dec 31, 2008

      severbmannequin
      MannequinAuthor

      Thanks for the new implementation of MockNonBlockWriterIO class. It
      makes tests so much easier to read.

      There are some minor things in your patch that I would change. For example:

          # 1 byte will be written, the rest will be buffered
          raw.block_on(b"k")
          self.assertEquals(bufio.write(b"jklmn"), 5)
          # ...
          raw.block_on(b"0")

      The comment is misleading because in fact no byte is written at raw
      level. That's because the data size is smaller than the buffer size and
      the buffer is empty (was emptied by the last write call). All this
      renders raw.block_on(b"k") call useless. I also think this is the
      correct behavior regardless of implementation language of BufferedWriter
      class i.e. no write call should write at raw level smaller chunks of
      data than buffer's size unless it has to.

      Your tests can't cover the pre-flush condition because max_buffer_size
      equals buffer_size.

      Unless you'll beat me to it or prove me wrong, I'll update your patch
      next year.

    12. pitrou commented on Dec 31, 2008

      @pitrou
      Member

      The comment is misleading because in fact no byte is written at raw
      level. That's because the data size is smaller than the buffer size and
      the buffer is empty (was emptied by the last write call).

      It depends on the implementation. A different implementation may use a
      different algorithm.

      I also think this is the
      correct behavior regardless of implementation language of BufferedWriter
      class i.e. no write call should write at raw level smaller chunks of
      data than buffer's size unless it has to.

      But how do you decide when it "has to"? Unless you want to constrain the
      exact implemented algorithm, you can't do that in your tests.

    13. severb commented on Dec 31, 2008

      severbmannequin
      MannequinAuthor

      > The comment is misleading because in fact no byte is written at raw
      > level. That's because the data size is smaller than the buffer size and
      > the buffer is empty (was emptied by the last write call).

      It depends on the implementation. A different implementation may use a
      different algorithm.

      I feel that no matter what implementation algorithm BufferedWriter uses
      it shouldn't write smaller chunks of data than buffer's size or else the
      buffer is useless.

      > I also think this is the
      > correct behavior regardless of implementation language of BufferedWriter
      > class i.e. no write call should write at raw level smaller chunks of
      > data than buffer's size unless it has to.

      But how do you decide when it "has to"? Unless you want to constrain the
      exact implemented algorithm, you can't do that in your tests.

      When a direct or indirect (e.g. on close) flush is called for the file
      object.

    14. pitrou commented on Dec 31, 2008

      @pitrou
      Member

      I feel that no matter what implementation algorithm BufferedWriter uses
      it shouldn't write smaller chunks of data than buffer's size or else the
      buffer is useless.

      If you rewrite the above sentence using the word "statistically", then I
      can agree :)
      But if I look at e.g. the fwrite() manpage, I see no guarantee that the
      stdio layer will never make a call to write() with a size smaller than
      the buffer size. The buffered layer should be free to manage its buffer
      in what it believes is the most efficient way. The only guarantee is
      that it won't buffer more than max_buffer_size.

      Anyway :) Practically, the test does work on both py3k and another
      implementation, so I don't see any urgency to remove anything from it.

    15. severb commented on Jan 7, 2009

      severbmannequin
      MannequinAuthor

      Anyway :) Practically, the test does work on both py3k and another
      implementation, so I don't see any urgency to remove anything from it.
      Indeed, it doesn't hurt keeping it.

      For completeness' sake I've updated your tests to cover the pre-flush
      condition existent in the python implementation. Theoretically this
      should be the desired behavior in any other implementation.

    16. benjaminp commented on Feb 28, 2009

      @benjaminp
      Contributor

      This has been cured in the io-c branch.

    17. benjaminp commented on Mar 4, 2009

      @benjaminp
      Contributor

      This has been fixed in io-c branch. (r70152)

    18. severb commented on Mar 5, 2009

      severbmannequin
      MannequinAuthor

      Looks like the test covering the pre-flush condition is missing.

    19. benjaminp commented on Mar 5, 2009

      @benjaminp
      Contributor

      2009/3/5 Sever Băneșiu <report@bugs.python.org>:

      Sever Băneșiu <banesiu.sever@gmail.com> added the comment:

      Looks like the test covering the pre-flush condition is missing.

      That test is no longer applicable because max_buffer_size is
      deprecated and unused.

    20. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions