Skip to content

StreamReader does not return reamaing and ready data buffer before raise the Exeption #75044

Open

Description

@pfreixes
mannequin
BPO 30861
Nosy @gvanrossum, @dimaqq, @1st1, @pfreixes
PRs
  • bpo-30861: Return StreamReeader remaining buffer before raise excepiton #2593
  • gh-75044: Return StreamReeader remaining buffer before raise an exception #7473
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2017-07-05.20:55:08.161>
    labels = ['3.7', 'expert-asyncio']
    title = 'StreamReader does not return reamaing and ready data buffer before raise the Exeption'
    updated_at = <Date 2018-06-07.09:15:04.407>
    user = 'https://github.com/pfreixes'

    bugs.python.org fields:

    activity = <Date 2018-06-07.09:15:04.407>
    actor = 'pfreixes'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['asyncio']
    creation = <Date 2017-07-05.20:55:08.161>
    creator = 'pfreixes'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 30861
    keywords = ['patch']
    message_count = 20.0
    messages = ['297781', '297783', '297784', '297785', '297787', '297789', '297808', '297842', '297846', '297866', '297873', '297890', '297894', '297898', '297900', '297907', '298532', '298541', '298580', '309261']
    nosy_count = 4.0
    nosy_names = ['gvanrossum', 'Dima.Tisnek', 'yselivanov', 'pfreixes']
    pr_nums = ['2593', '7473']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue30861'
    versions = ['Python 3.4', 'Python 3.5', 'Python 3.6', 'Python 3.7']

    Activity

    1. pfreixes commented on Jul 5, 2017

      pfreixesmannequin
      MannequinAuthor

      Current implementation of StreamReader does not take care of the status of the buffer, once an exception has been set via set_exception any call to the read methods won't be able to get the missing data still pending to be processed.

      From the point of view of the developer, if there is no scheduled task for waiting data into the Streamreader between a network data gets into the buffer socket and a closing connection by the other peer arrives, the developer won't be able to gather the previous data.

    2. gvanrossum commented on Jul 5, 2017

      @gvanrossum
      Member

      Do you have a specific use case where it's important to access the remaining data? ISTM that this is happening when the connection is lost and then it shouldn't matter how much of the data you read or not -- the connection was broken before the remote side closed it, and you should not trust the data.

    3. pfreixes commented on Jul 5, 2017

      pfreixesmannequin
      MannequinAuthor

      Yeps,

      I have an example related to Redis [1], the server warns that the
      connection will be closed because you reached out the maximum number
      of connections.

      But IMHO the example it's just that an example, if the connection at
      some point was not broken/closed/... and some data came into the
      buffer, this data should be put available for the user.

      [1] https://github.com/antirez/redis/blob/09dd7b5ff02de4a311032939c27fd6fc62fbd4a3/src/networking.c#L615

      On Wed, Jul 5, 2017 at 11:19 PM, Guido van Rossum
      <report@bugs.python.org> wrote:

      Guido van Rossum added the comment:

      Do you have a specific use case where it's important to access the remaining data? ISTM that this is happening when the connection is lost and then it shouldn't matter how much of the data you read or not -- the connection was broken before the remote side closed it, and you should not trust the data.

      ----------
      nosy: +gvanrossum


      Python tracker <report@bugs.python.org>
      <http://bugs.python.org/issue30861\>


    4. 1st1 commented on Jul 5, 2017

      @1st1
      Member

      But IMHO the example it's just that an example, if the connection at
      some point was not broken/closed/... and some data came into the
      buffer, this data should be put available for the user.

      In my experience, you never want to recover any data from broken connections. In cases where it's safe, StreamReader attaches the buffered data to the exception (see IncompleteReadError).

    5. pfreixes commented on Jul 5, 2017

      pfreixesmannequin
      MannequinAuthor

      One of the disadvantages with the Exception is that you are relying on
      how the events are being succeeded, and this is something that you
      can't predict. With just a different delay between the data and the
      RST packet and you might get the data in two different ways, either
      via a regular read or via one adhoc Exception.

      This might be confusing for the developer and will end up with the
      second thing that IMHO is also important, the code that has to be
      produced by the developer gets complicated.

      On Wed, Jul 5, 2017 at 11:29 PM, Yury Selivanov <report@bugs.python.org> wrote:

      Yury Selivanov added the comment:

      > But IMHO the example it's just that an example, if the connection at
      some point was not broken/closed/... and some data came into the
      buffer, this data should be put available for the user.

      In my experience, you never want to recover any data from broken connections. In cases where it's safe, StreamReader attaches the buffered data to the exception (see IncompleteReadError).

      ----------


      Python tracker <report@bugs.python.org>
      <http://bugs.python.org/issue30861\>


    6. gvanrossum commented on Jul 5, 2017

      @gvanrossum
      Member

      The Redis example doesn't sound valid to me, as (IIUC) it closes the connection immediately?

    7. dimaqq commented on Jul 6, 2017

      dimaqqmannequin
      Mannequin

      My 2c:

      Pau's concern seems valid, in a sense that stream should work like TCP.
      That's what most users would assume -- read out data until the end, only then you can see what the actual error was (socket closed, or timeout or hard error)

      However, I suspect the devil may be in details -- the patch appears to assume that exception is only set by the underlying source of data (in which case user wants to read out everything up to the exception).

      I can imagine a couple of use-case where an exception is set "out of band":

      • to terminate bad reader (e.g. against Slowris attack)
      • there's a matching StreamWriter (e.g. shared socket)

      P.S.
      If I'm wrong here, then bug report and patch must be clearer :)

    8. pfreixes commented on Jul 6, 2017

      pfreixesmannequin
      MannequinAuthor

      As was said, the assumption here is the data that came to the buffer must be available.

      For example, the next snippet shows a Redis client that expects the data message plus the RST packet, where the redis-server was configured to accept max N connections, the connection used by this snippet was the N+1

      import socket
      import time
      
      s = socket.socket(
          socket.AF_INET, socket.SOCK_STREAM)
      s.connect(("localhost", 6379))
      
      # give enought time to get the
      # data plus the RST Package
      time.sleep(1)
      
      # read the data
      print(s.recv(100))

      It works like a charm.

      IMHO this is not a matter of RST vs close graceful, its a matter of giving always the chance to the developer to read the data that is still remaining and ready into the buffer.

      The current implementation is in somehow not predictable. Different network latencies with the same scenario can produce different outputs and the same with how the read of the Streamer is handled by the developer.

    9. gvanrossum commented on Jul 6, 2017

      @gvanrossum
      Member

      We seem to have a failure to communicate. I'm sure your example code "works", but you're not showing what's in the data it receives that is important for the app to read (your example just prints it).

      And surely your app should be robust even if the connection is closed *without* ever receiving that data (since it may be lost in transit).

      If you need an application-level error your design should include it in the protocol, not use the connection closing.

    10. pfreixes commented on Jul 7, 2017

      pfreixesmannequin
      MannequinAuthor

      I think that we are not in the same page, the following snippet is a nodejs client that emulates the same:

      var net = require('net');
      var sleep = require('sleep');

      var client = new net.Socket();
      client.connect(6379, '127.0.0.1', function() {
      console.log('Connected');
      sleep.sleep(10); // wait for data and RST package
      });

      client.on('data', function(data) {
      console.log('Received: ' + data);
      });

      client.on('close', function() {
      console.log('Connection closed');
      });

      The client will be able to get the data, even taking into account that the RST package has been processed by the Operating System. Meanwhile, the same behavior with asyncio is unpredictable.

    11. dimaqq commented on Jul 7, 2017

      dimaqqmannequin
      Mannequin

      It seems Guido sets a higher bar on the proposed change.

      @pfreixes, if you can show that this change is needed to implement "TCP half-closed" state (i.e. when remote calls shutdown(SHUT_WR) after it's done sending data but continues to recv(), then local is expected to read out the data, and then confirm reception and issue it's own shutdown(SHUT_WR) so that remote gets the acknowledgement), then there would be no question.

    12. gvanrossum commented on Jul 7, 2017

      @gvanrossum
      Member

      But half closed state is already supported.

      On Jul 7, 2017 2:37 AM, "Dima Tisnek" <report@bugs.python.org> wrote:

      Dima Tisnek added the comment:

      It seems Guido sets a higher bar on the proposed change.

      @pfreixes, if you can show that this change is needed to implement "TCP
      half-closed" state (i.e. when remote calls shutdown(SHUT_WR) after it's
      done sending data but continues to recv(), then local is expected to read
      out the data, and then confirm reception and issue it's own
      shutdown(SHUT_WR) so that remote gets the acknowledgement), then there
      would be no question.

      ----------


      Python tracker <report@bugs.python.org>
      <http://bugs.python.org/issue30861\>


    13. pfreixes commented on Jul 7, 2017

      pfreixesmannequin
      MannequinAuthor

      I would like to focus the issue as was initially described, Im still
      convinced that this is a buggy behaviour. As has been seen other systems
      such as python sync or nodejs behaves as is expected.

      This last one is IMHO something that can be skipped.

      Im wondering how curio or trio will handle this scenario. My bet is on as
      nodejs.

      El 07/07/2017 17:06, "Guido van Rossum" <report@bugs.python.org> escribió:

      Guido van Rossum added the comment:

      But half closed state is already supported.

      On Jul 7, 2017 2:37 AM, "Dima Tisnek" <report@bugs.python.org> wrote:

      >
      > Dima Tisnek added the comment:
      >
      > It seems Guido sets a higher bar on the proposed change.
      >
      > @pfreixes, if you can show that this change is needed to implement "TCP
      > half-closed" state (i.e. when remote calls shutdown(SHUT_WR) after it's
      > done sending data but continues to recv(), then local is expected to read
      > out the data, and then confirm reception and issue it's own
      > shutdown(SHUT_WR) so that remote gets the acknowledgement), then there
      > would be no question.
      >
      > ----------
      >
      > _______________________________________
      > Python tracker <report@bugs.python.org>
      > <http://bugs.python.org/issue30861\>
      > _______________________________________
      >

      ----------


      Python tracker <report@bugs.python.org>
      <http://bugs.python.org/issue30861\>


    14. 2 remaining items

    15. pfreixes commented on Jul 7, 2017

      pfreixesmannequin
      MannequinAuthor

      fair enough. I'm out with few chances to grab a computer. I'll be back in
      a week with more information.

      El 07/07/2017 20:14, "Yury Selivanov" <report@bugs.python.org> escribió:

      Yury Selivanov added the comment:

      I looked at the PR and it looks good. The intent is also clear. But the
      idea of delaying exceptions in StreamReader read methods still feels
      finicky.

      To go forward with this we need more examples than just nodejs. And it
      doesn't really matter what trio and curio do, as they aren't mainstream. We
      need a (set of) *concrete* examples where this is needed.

      An analysis of how this is approached in another mainstream
      libraries/ecosystems like C#, Twisted, etc would also help.

      ----------


      Python tracker <report@bugs.python.org>
      <http://bugs.python.org/issue30861\>


    16. pfreixes commented on Jul 17, 2017

      pfreixesmannequin
      MannequinAuthor

      The following links point to three different implementations of the same scenario using a Twisted, Node and Python blocking clients that try to reproduce the same scenario.

      All of them are able to give you the buffer when the RST package was already processed by the OS.

      I'm still trying to get a Java client, but TBH I won't expect a different behavior.

    17. gvanrossum commented on Jul 17, 2017

      @gvanrossum
      Member

      There's no need. You seem to have accidentally shown a use case for getting the buffered data -- it can contain an error message explaining why the connection was closed.

    18. pfreixes commented on Jul 18, 2017

      pfreixesmannequin
      MannequinAuthor

      The Java client https://gist.github.com/pfreixes/13fedf2a589c260e6c7c64ae73653bb1

      Works as is expected the buffer can be read till it gets empty, no matter when the RST was sent.

    19. pfreixes commented on Dec 30, 2017

      pfreixesmannequin
      MannequinAuthor

      Yeps, any update on this bug and the fix proposed?

    20. transferred this issue fromon Apr 10, 2022
    21. ImBearChild commented on Oct 19, 2025

      @ImBearChild

      Sorry to bring up this old issue. I encountered the same problem.

      If the client closes the connection for some reason while the server is sending data, the server receives a RST. In this case, the client can send the reason for closing the connection to the server (for example, if the data format is incorrect, making it possible to determine without receiving all the data, or if the client itself encountered an issue), but the server is unable to read this due to the limitations of StreamReader (since the exception is raised during write). This behavior is different from traditional blocking APIs.

      Is there a possibility that the PR related to this issue could be merged? If I want to work around this problem, is it safe to use StreamWriter.set_exception(None)?

    22. added
      stdlibStandard Library Python modules in the Lib/ directory
      type-featureA feature request or enhancement
      on Oct 19, 2025
    23. picnixz commented on Oct 19, 2025

      @picnixz
      Member

      Considering the time this issue was opened, changing it now would require a re-discussion to see how it evolved since then (are the examples that were shown still relevant?) If Java, node and others haven't changes since then, we could consider revisiting the PR and eventually merge it.

    24. ImBearChild commented on Oct 23, 2025

      @ImBearChild

      I don't know Java and NodeJS, but I have some knowledge of Rust and Go. So, I did some tests using these two languages along with Python, and I'll go straight to the conclusion:

      Python is the only language that gatekeeps asynchronous reading results. Both Rust (using the Tokio library) and Go return buffered results normally. Python's blocking socket API can also do this.


      Here are the details. I implemented a simple TCP server using different language and API:

      • Rust (Tokio asynchronous runtime)
      • Go (Native goroutine)
      • Python
        • blocking-style socket
        • normal asyncio module
        • asyncio with reader._exception set to None before reading

      The server does the following:

      1. Listens on 127.0.0.1:12345
      2. For each client connection:
        - Sends 1MB of zeros
        - Waits 1 second (simulating latency)
        - Reads one message from the client
        - Closes the connection

      I also implemented a client in Python that sends an error message and disconnects after reading 1 KB. This simulates a scenario in TCP connection data transmission where one party unexpectedly terminates the transfer.

      Both Rust and Go simply returned the result and exhibited the same behavior as reading to EOF normally during subsequent reads. The Python socket library returns the result but throws an error (Connection Reset) on the second read. However, after setting reader._exception to None (a dirty workaround), the behavior of asyncio aligns with that of Rust and Go. Of course, normal asyncio module will just throw the exception.

      I suggest that Python's behavior should align with other languages (it is very counterintuitive). If we can't achieve that (after all, counterintuitiveness isn't a serious issue), there should at least be a way to retrieve the data instead of discarding it.

      Here is the code. https://github.com/ImBearChild/asyncio-test

      Note: I conducted the tests on Fedora 42, and I'm not sure if these test cases behave the same way on Windows or other Unix platforms.

    25. ImBearChild commented on Nov 14, 2025

      @ImBearChild

      If Java, node and others haven't changes since then, we could consider revisiting the PR and eventually merge it.

      Is the example of Rust and Go sufficient to illustrate this point? Do we still need to test NodeJS or Java?

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      stdlibStandard Library Python modules in the Lib/ directorytopic-asynciotype-featureA feature request or enhancement

      Projects

      • Status
        Todo

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions