You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
StreamReader does not return reamaing and ready data buffer before raise the Exeption #75044
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee=Noneclosed_at=Nonecreated_at=<Date2017-07-05.20:55:08.161>labels= ['3.7', 'expert-asyncio']
title='StreamReader does not return reamaing and ready data buffer before raise the Exeption'updated_at=<Date2018-06-07.09:15:04.407>user='https://github.com/pfreixes'
Current implementation of StreamReader does not take care of the status of the buffer, once an exception has been set via set_exception any call to the read methods won't be able to get the missing data still pending to be processed.
From the point of view of the developer, if there is no scheduled task for waiting data into the Streamreader between a network data gets into the buffer socket and a closing connection by the other peer arrives, the developer won't be able to gather the previous data.
Do you have a specific use case where it's important to access the remaining data? ISTM that this is happening when the connection is lost and then it shouldn't matter how much of the data you read or not -- the connection was broken before the remote side closed it, and you should not trust the data.
pfreixes commented on Jul 5, 2017
pfreixesmannequin
MannequinAuthor
More actions
Yeps,
I have an example related to Redis [1], the server warns that the
connection will be closed because you reached out the maximum number
of connections.
But IMHO the example it's just that an example, if the connection at
some point was not broken/closed/... and some data came into the
buffer, this data should be put available for the user.
Do you have a specific use case where it's important to access the remaining data? ISTM that this is happening when the connection is lost and then it shouldn't matter how much of the data you read or not -- the connection was broken before the remote side closed it, and you should not trust the data.
But IMHO the example it's just that an example, if the connection at
some point was not broken/closed/... and some data came into the
buffer, this data should be put available for the user.
In my experience, you never want to recover any data from broken connections. In cases where it's safe, StreamReader attaches the buffered data to the exception (see IncompleteReadError).
pfreixes commented on Jul 5, 2017
pfreixesmannequin
MannequinAuthor
More actions
One of the disadvantages with the Exception is that you are relying on
how the events are being succeeded, and this is something that you
can't predict. With just a different delay between the data and the
RST packet and you might get the data in two different ways, either
via a regular read or via one adhoc Exception.
This might be confusing for the developer and will end up with the
second thing that IMHO is also important, the code that has to be
produced by the developer gets complicated.
> But IMHO the example it's just that an example, if the connection at
some point was not broken/closed/... and some data came into the
buffer, this data should be put available for the user.
In my experience, you never want to recover any data from broken connections. In cases where it's safe, StreamReader attaches the buffered data to the exception (see IncompleteReadError).
The Redis example doesn't sound valid to me, as (IIUC) it closes the connection immediately?
dimaqq commented on Jul 6, 2017
dimaqqmannequin
Mannequin
More actions
My 2c:
Pau's concern seems valid, in a sense that stream should work like TCP.
That's what most users would assume -- read out data until the end, only then you can see what the actual error was (socket closed, or timeout or hard error)
However, I suspect the devil may be in details -- the patch appears to assume that exception is only set by the underlying source of data (in which case user wants to read out everything up to the exception).
I can imagine a couple of use-case where an exception is set "out of band":
to terminate bad reader (e.g. against Slowris attack)
there's a matching StreamWriter (e.g. shared socket)
P.S.
If I'm wrong here, then bug report and patch must be clearer :)
pfreixes commented on Jul 6, 2017
pfreixesmannequin
MannequinAuthor
More actions
As was said, the assumption here is the data that came to the buffer must be available.
For example, the next snippet shows a Redis client that expects the data message plus the RST packet, where the redis-server was configured to accept max N connections, the connection used by this snippet was the N+1
importsocketimporttimes=socket.socket(
socket.AF_INET, socket.SOCK_STREAM)
s.connect(("localhost", 6379))
# give enought time to get the# data plus the RST Packagetime.sleep(1)
# read the dataprint(s.recv(100))
It works like a charm.
IMHO this is not a matter of RST vs close graceful, its a matter of giving always the chance to the developer to read the data that is still remaining and ready into the buffer.
The current implementation is in somehow not predictable. Different network latencies with the same scenario can produce different outputs and the same with how the read of the Streamer is handled by the developer.
We seem to have a failure to communicate. I'm sure your example code "works", but you're not showing what's in the data it receives that is important for the app to read (your example just prints it).
And surely your app should be robust even if the connection is closed *without* ever receiving that data (since it may be lost in transit).
If you need an application-level error your design should include it in the protocol, not use the connection closing.
pfreixes commented on Jul 7, 2017
pfreixesmannequin
MannequinAuthor
More actions
I think that we are not in the same page, the following snippet is a nodejs client that emulates the same:
var net = require('net');
var sleep = require('sleep');
var client = new net.Socket();
client.connect(6379, '127.0.0.1', function() {
console.log('Connected');
sleep.sleep(10); // wait for data and RST package
});
The client will be able to get the data, even taking into account that the RST package has been processed by the Operating System. Meanwhile, the same behavior with asyncio is unpredictable.
dimaqq commented on Jul 7, 2017
dimaqqmannequin
Mannequin
More actions
It seems Guido sets a higher bar on the proposed change.
@pfreixes, if you can show that this change is needed to implement "TCP half-closed" state (i.e. when remote calls shutdown(SHUT_WR) after it's done sending data but continues to recv(), then local is expected to read out the data, and then confirm reception and issue it's own shutdown(SHUT_WR) so that remote gets the acknowledgement), then there would be no question.
It seems Guido sets a higher bar on the proposed change.
@pfreixes, if you can show that this change is needed to implement "TCP
half-closed" state (i.e. when remote calls shutdown(SHUT_WR) after it's
done sending data but continues to recv(), then local is expected to read
out the data, and then confirm reception and issue it's own
shutdown(SHUT_WR) so that remote gets the acknowledgement), then there
would be no question.
I would like to focus the issue as was initially described, Im still
convinced that this is a buggy behaviour. As has been seen other systems
such as python sync or nodejs behaves as is expected.
This last one is IMHO something that can be skipped.
Im wondering how curio or trio will handle this scenario. My bet is on as
nodejs.
>
> Dima Tisnek added the comment:
>
> It seems Guido sets a higher bar on the proposed change.
>
> @pfreixes, if you can show that this change is needed to implement "TCP
> half-closed" state (i.e. when remote calls shutdown(SHUT_WR) after it's
> done sending data but continues to recv(), then local is expected to read
> out the data, and then confirm reception and issue it's own
> shutdown(SHUT_WR) so that remote gets the acknowledgement), then there
> would be no question.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue30861\>
> _______________________________________
>
I looked at the PR and it looks good. The intent is also clear. But the
idea of delaying exceptions in StreamReader read methods still feels
finicky.
To go forward with this we need more examples than just nodejs. And it
doesn't really matter what trio and curio do, as they aren't mainstream. We
need a (set of) *concrete* examples where this is needed.
An analysis of how this is approached in another mainstream
libraries/ecosystems like C#, Twisted, etc would also help.
The following links point to three different implementations of the same scenario using a Twisted, Node and Python blocking clients that try to reproduce the same scenario.
There's no need. You seem to have accidentally shown a use case for getting the buffered data -- it can contain an error message explaining why the connection was closed.
Sorry to bring up this old issue. I encountered the same problem.
If the client closes the connection for some reason while the server is sending data, the server receives a RST. In this case, the client can send the reason for closing the connection to the server (for example, if the data format is incorrect, making it possible to determine without receiving all the data, or if the client itself encountered an issue), but the server is unable to read this due to the limitations of StreamReader (since the exception is raised during write). This behavior is different from traditional blocking APIs.
Is there a possibility that the PR related to this issue could be merged? If I want to work around this problem, is it safe to use StreamWriter.set_exception(None)?
Considering the time this issue was opened, changing it now would require a re-discussion to see how it evolved since then (are the examples that were shown still relevant?) If Java, node and others haven't changes since then, we could consider revisiting the PR and eventually merge it.
I don't know Java and NodeJS, but I have some knowledge of Rust and Go. So, I did some tests using these two languages along with Python, and I'll go straight to the conclusion:
Python is the only language that gatekeeps asynchronous reading results. Both Rust (using the Tokio library) and Go return buffered results normally. Python's blocking socket API can also do this.
Here are the details. I implemented a simple TCP server using different language and API:
Rust (Tokio asynchronous runtime)
Go (Native goroutine)
Python
blocking-style socket
normal asyncio module
asyncio with reader._exception set to None before reading
The server does the following:
Listens on 127.0.0.1:12345
For each client connection:
- Sends 1MB of zeros
- Waits 1 second (simulating latency)
- Reads one message from the client
- Closes the connection
I also implemented a client in Python that sends an error message and disconnects after reading 1 KB. This simulates a scenario in TCP connection data transmission where one party unexpectedly terminates the transfer.
Both Rust and Go simply returned the result and exhibited the same behavior as reading to EOF normally during subsequent reads. The Python socket library returns the result but throws an error (Connection Reset) on the second read. However, after setting reader._exception to None (a dirty workaround), the behavior of asyncio aligns with that of Rust and Go. Of course, normal asyncio module will just throw the exception.
I suggest that Python's behavior should align with other languages (it is very counterintuitive). If we can't achieve that (after all, counterintuitiveness isn't a serious issue), there should at least be a way to retrieve the data instead of discarding it.
Activity
pfreixes commentedon Jul 5, 2017
Current implementation of StreamReader does not take care of the status of the buffer, once an exception has been set via
set_exceptionany call to the read methods won't be able to get the missing data still pending to be processed.From the point of view of the developer, if there is no scheduled task for waiting data into the Streamreader between a network data gets into the buffer socket and a closing connection by the other peer arrives, the developer won't be able to gather the previous data.
gvanrossum commentedon Jul 5, 2017
Do you have a specific use case where it's important to access the remaining data? ISTM that this is happening when the connection is lost and then it shouldn't matter how much of the data you read or not -- the connection was broken before the remote side closed it, and you should not trust the data.
pfreixes commentedon Jul 5, 2017
Yeps,
I have an example related to Redis [1], the server warns that the
connection will be closed because you reached out the maximum number
of connections.
But IMHO the example it's just that an example, if the connection at
some point was not broken/closed/... and some data came into the
buffer, this data should be put available for the user.
[1] https://github.com/antirez/redis/blob/09dd7b5ff02de4a311032939c27fd6fc62fbd4a3/src/networking.c#L615
On Wed, Jul 5, 2017 at 11:19 PM, Guido van Rossum
<report@bugs.python.org> wrote:
1st1 commentedon Jul 5, 2017
In my experience, you never want to recover any data from broken connections. In cases where it's safe, StreamReader attaches the buffered data to the exception (see IncompleteReadError).
pfreixes commentedon Jul 5, 2017
One of the disadvantages with the Exception is that you are relying on
how the events are being succeeded, and this is something that you
can't predict. With just a different delay between the data and the
RST packet and you might get the data in two different ways, either
via a regular read or via one adhoc Exception.
This might be confusing for the developer and will end up with the
second thing that IMHO is also important, the code that has to be
produced by the developer gets complicated.
On Wed, Jul 5, 2017 at 11:29 PM, Yury Selivanov <report@bugs.python.org> wrote:
gvanrossum commentedon Jul 5, 2017
The Redis example doesn't sound valid to me, as (IIUC) it closes the connection immediately?
dimaqq commentedon Jul 6, 2017
My 2c:
Pau's concern seems valid, in a sense that stream should work like TCP.
That's what most users would assume -- read out data until the end, only then you can see what the actual error was (socket closed, or timeout or hard error)
However, I suspect the devil may be in details -- the patch appears to assume that exception is only set by the underlying source of data (in which case user wants to read out everything up to the exception).
I can imagine a couple of use-case where an exception is set "out of band":
P.S.
If I'm wrong here, then bug report and patch must be clearer :)
pfreixes commentedon Jul 6, 2017
As was said, the assumption here is the data that came to the buffer must be available.
For example, the next snippet shows a Redis client that expects the data message plus the RST packet, where the redis-server was configured to accept max N connections, the connection used by this snippet was the N+1
It works like a charm.
IMHO this is not a matter of RST vs close graceful, its a matter of giving always the chance to the developer to read the data that is still remaining and ready into the buffer.
The current implementation is in somehow not predictable. Different network latencies with the same scenario can produce different outputs and the same with how the read of the Streamer is handled by the developer.
gvanrossum commentedon Jul 6, 2017
We seem to have a failure to communicate. I'm sure your example code "works", but you're not showing what's in the data it receives that is important for the app to read (your example just prints it).
And surely your app should be robust even if the connection is closed *without* ever receiving that data (since it may be lost in transit).
If you need an application-level error your design should include it in the protocol, not use the connection closing.
pfreixes commentedon Jul 7, 2017
I think that we are not in the same page, the following snippet is a nodejs client that emulates the same:
var net = require('net');
var sleep = require('sleep');
var client = new net.Socket();
client.connect(6379, '127.0.0.1', function() {
console.log('Connected');
sleep.sleep(10); // wait for data and RST package
});
client.on('data', function(data) {
console.log('Received: ' + data);
});
client.on('close', function() {
console.log('Connection closed');
});
The client will be able to get the data, even taking into account that the RST package has been processed by the Operating System. Meanwhile, the same behavior with asyncio is unpredictable.
dimaqq commentedon Jul 7, 2017
It seems Guido sets a higher bar on the proposed change.
@pfreixes, if you can show that this change is needed to implement "TCP half-closed" state (i.e. when remote calls shutdown(SHUT_WR) after it's done sending data but continues to recv(), then local is expected to read out the data, and then confirm reception and issue it's own shutdown(SHUT_WR) so that remote gets the acknowledgement), then there would be no question.
gvanrossum commentedon Jul 7, 2017
But half closed state is already supported.
On Jul 7, 2017 2:37 AM, "Dima Tisnek" <report@bugs.python.org> wrote:
pfreixes commentedon Jul 7, 2017
I would like to focus the issue as was initially described, Im still
convinced that this is a buggy behaviour. As has been seen other systems
such as python sync or nodejs behaves as is expected.
This last one is IMHO something that can be skipped.
Im wondering how curio or trio will handle this scenario. My bet is on as
nodejs.
El 07/07/2017 17:06, "Guido van Rossum" <report@bugs.python.org> escribió:
2 remaining items
pfreixes commentedon Jul 7, 2017
fair enough. I'm out with few chances to grab a computer. I'll be back in
a week with more information.
El 07/07/2017 20:14, "Yury Selivanov" <report@bugs.python.org> escribió:
pfreixes commentedon Jul 17, 2017
The following links point to three different implementations of the same scenario using a Twisted, Node and Python blocking clients that try to reproduce the same scenario.
All of them are able to give you the buffer when the RST package was already processed by the OS.
I'm still trying to get a Java client, but TBH I won't expect a different behavior.
gvanrossum commentedon Jul 17, 2017
There's no need. You seem to have accidentally shown a use case for getting the buffered data -- it can contain an error message explaining why the connection was closed.
pfreixes commentedon Jul 18, 2017
The Java client https://gist.github.com/pfreixes/13fedf2a589c260e6c7c64ae73653bb1
Works as is expected the buffer can be read till it gets empty, no matter when the RST was sent.
pfreixes commentedon Dec 30, 2017
Yeps, any update on this bug and the fix proposed?
ImBearChild commentedon Oct 19, 2025
Sorry to bring up this old issue. I encountered the same problem.
If the client closes the connection for some reason while the server is sending data, the server receives a
RST. In this case, the client can send the reason for closing the connection to the server (for example, if the data format is incorrect, making it possible to determine without receiving all the data, or if the client itself encountered an issue), but the server is unable to read this due to the limitations ofStreamReader(since the exception is raised duringwrite). This behavior is different from traditional blocking APIs.Is there a possibility that the PR related to this issue could be merged? If I want to work around this problem, is it safe to use
StreamWriter.set_exception(None)?picnixz commentedon Oct 19, 2025
Considering the time this issue was opened, changing it now would require a re-discussion to see how it evolved since then (are the examples that were shown still relevant?) If Java, node and others haven't changes since then, we could consider revisiting the PR and eventually merge it.
ImBearChild commentedon Oct 23, 2025
I don't know Java and NodeJS, but I have some knowledge of Rust and Go. So, I did some tests using these two languages along with Python, and I'll go straight to the conclusion:
Python is the only language that gatekeeps asynchronous reading results. Both Rust (using the Tokio library) and Go return buffered results normally. Python's blocking socket API can also do this.
Here are the details. I implemented a simple TCP server using different language and API:
reader._exceptionset to None before readingThe server does the following:
- Sends 1MB of zeros
- Waits 1 second (simulating latency)
- Reads one message from the client
- Closes the connection
I also implemented a client in Python that sends an error message and disconnects after reading 1 KB. This simulates a scenario in TCP connection data transmission where one party unexpectedly terminates the transfer.
Both Rust and Go simply returned the result and exhibited the same behavior as reading to EOF normally during subsequent reads. The Python socket library returns the result but throws an error (Connection Reset) on the second read. However, after setting
reader._exceptionto None (a dirty workaround), the behavior of asyncio aligns with that of Rust and Go. Of course, normal asyncio module will just throw the exception.I suggest that Python's behavior should align with other languages (it is very counterintuitive). If we can't achieve that (after all, counterintuitiveness isn't a serious issue), there should at least be a way to retrieve the data instead of discarding it.
Here is the code. https://github.com/ImBearChild/asyncio-test
Note: I conducted the tests on Fedora 42, and I'm not sure if these test cases behave the same way on Windows or other Unix platforms.
ImBearChild commentedon Nov 14, 2025
Is the example of Rust and Go sufficient to illustrate this point? Do we still need to test NodeJS or Java?