You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
While working on bpo-1859, I found that textwrap.wrap() returns an empty list when passed the empty string:
>>> from textwrap import wrap
>>> wrap('')
[]
as opposed to a list containing the empty string which is what I expected--
['']
I originally accepted this as intended behavior, but for the following reasons I now think it is a bug that we should consider fixing:
The textwrap documentation says that wrap(), "Wraps the single paragraph in text (a string) so every line is at most width characters long. Returns a list of output lines, without final newlines." The empty string is less than width characters long, so following the documentation, wrapping should not change it.
It is known that wrap() is broken for strings containing newlines (i.e. strings containing more than one paragraph). Indeed, this is the issue that bpo-1859 is meant to fix.
The commonly recommended work-around is to call splitlines() on the incoming string, pass the pieces to wrap(), and then join the return values on newlines.
However, the behavior described in this issue causes this workaround not to behave sensibly when the original string contains breaks between paragraphs. See this message, for example:
Currently, this work-around would return "a\nb" for both "a\nb" and "a\n\nb" for example when common sense says it should preserve the newlines and return "a\nb" and "a\n\nb", respectively.
In addition, the behavior causes the following inconsistent behavior:
The documentation says this about drop_whitespace: "If true, whitespace that, after wrapping, happens to end up at the beginning or end of a line is dropped." If the first case is correct, then, the second case should at least return a non-empty list because that is what would result from dropping whitespace from the string ' ' in the first list. This is how drop_whitespace behaves when the string contains non-whitespace characters:
There is no unit test for the case of the empty string, and existing tests still pass after fixing the issue.
If we cannot correct this behavior, then I feel we should at least document the inconsistent behavior, and then work around it in the fix for bpo-1859.
Marking for this issue to be resolved either way before fixing bpo-1859. I am happy to prepare the patch.
Could you possibly write a patch for 2.7, 3.2 and 3.3?. With a test :-)
I would be happy to write a patch with tests (I think there may be a few edge cases we would want to test here). Should be ready within the next one or two days.
The starred ones fail with the current code (the test cases are similar to the snippets I included in my first comment). The others fail under a certain incorrect implementation of this issue. (I'm adding them to increase coverage of edge cases and to protect against future regressions.)
Actually, here is a slightly more benign version of the patch.
This patch makes the change in the wrap() method and leaves _wrap_chunks() alone.
This is less intrusive because it doesn't change the behavior of _wrap_chunks(), which some people might be calling if they are subclassing TextWrapper and accessing the private API directly.
Also you will note that the return of the empty list for an empty line is exactly what you want for wrapping multiple line-break-delimited paragraphs. Consider:
>>> doc = "a para\nanother para\n\na third, but with an extra blank line between\n"
>>> for line in doc.splitlines():
... print('\n'.join(textwrap.wrap(line, width=5)))
... if line:
... print()
a
para
anoth
er
para
a thi
rd,
but
with
an
extra
blank
line
betwe
en
In other words, we need to add a blank line after our formatted paragraph, unless it is empty, in which case we don't want to add one or we'll have an extra. This assumes that single-line-paragraphs do not have blank lines between them...if they do, then the algorithm is even simpler, as you don't need the if.
Here is a patch that documents and adds tests for the existing behavior (i.e. keeping the current behavior the same).
I also expanded the patch slightly to cover related edge cases that involve the interplay between whitespace, empty lines, and indenting (some of which came out of the discussions above).
Thanks a lot for the very thorough review, David. I should be able to update the patch and respond to a couple of your points later today or tomorrow at the latest.
I responded to David's comments on the review tool. Later today I will update the patch in response to his comments (accommodating all of his suggestions) along with a couple other changes.
FWIW, I agree that the existing behavior should not be changed. Most likely, a change would break some code that is currently working, and there would be little to no gain.
Attaching an updated patch as promised in my previous comment.
Note that I removed two edge test cases pertaining to leading whitespace. I would rather discuss those cases as part of a different issue to avoid making this thread even longer (and it is off the topic of the original issue).
Activity
cjerdonek commentedon Jul 31, 2012
While working on bpo-1859, I found that textwrap.wrap() returns an empty list when passed the empty string:
as opposed to a list containing the empty string which is what I expected--
['']
I originally accepted this as intended behavior, but for the following reasons I now think it is a bug that we should consider fixing:
The textwrap documentation says that wrap(), "Wraps the single paragraph in text (a string) so every line is at most width characters long. Returns a list of output lines, without final newlines." The empty string is less than width characters long, so following the documentation, wrapping should not change it.
It is known that wrap() is broken for strings containing newlines (i.e. strings containing more than one paragraph). Indeed, this is the issue that bpo-1859 is meant to fix.
The commonly recommended work-around is to call splitlines() on the incoming string, pass the pieces to wrap(), and then join the return values on newlines.
However, the behavior described in this issue causes this workaround not to behave sensibly when the original string contains breaks between paragraphs. See this message, for example:
http://bugs.python.org/issue1859#msg166627
Currently, this work-around would return "a\nb" for both "a\nb" and "a\n\nb" for example when common sense says it should preserve the newlines and return "a\nb" and "a\n\nb", respectively.
The documentation says this about drop_whitespace: "If true, whitespace that, after wrapping, happens to end up at the beginning or end of a line is dropped." If the first case is correct, then, the second case should at least return a non-empty list because that is what would result from dropping whitespace from the string ' ' in the first list. This is how drop_whitespace behaves when the string contains non-whitespace characters:
If we cannot correct this behavior, then I feel we should at least document the inconsistent behavior, and then work around it in the fix for bpo-1859.
Marking for this issue to be resolved either way before fixing bpo-1859. I am happy to prepare the patch.
jcea commentedon Jul 31, 2012
I think this is a bug.
Could you possibly write a patch for 2.7, 3.2 and 3.3?. With a test :-)
I am worried about changing a behaviour that somebody is depending of. Opinions?
Adding a couple of persons I think that could have an opinion about this.
cjerdonek commentedon Jul 31, 2012
I would be happy to write a patch with tests (I think there may be a few edge cases we would want to test here). Should be ready within the next one or two days.
cjerdonek commentedon Jul 31, 2012
Attaching patch with test cases.
I added 6 new tests:
*test_empty_string
test_whitespace_trailing
*test_drop_whitespace__all_whitespace
*test_initial_indent__empty_string
test_subsequent_indent__trailing_whitespace
test_subsequent_indent__long_indent
The starred ones fail with the current code (the test cases are similar to the snippets I included in my first comment). The others fail under a certain incorrect implementation of this issue. (I'm adding them to increase coverage of edge cases and to protect against future regressions.)
cjerdonek commentedon Jul 31, 2012
Actually, here is a slightly more benign version of the patch.
This patch makes the change in the wrap() method and leaves _wrap_chunks() alone.
This is less intrusive because it doesn't change the behavior of _wrap_chunks(), which some people might be calling if they are subclassing TextWrapper and accessing the private API directly.
cjerdonek commentedon Aug 1, 2012
I verified that this patch can be applied to 2.7 and that those tests pass as well.
pitrou commentedon Aug 1, 2012
Uh, how is this a bug? An empty text doesn't contain lines at all, so returning an empty list of lines sounds right.
Furthermore, by "fixing" this, you may break existing software.
cjerdonek commentedon Aug 1, 2012
Here are additional test cases impacted by this issue:
jcea commentedon Aug 3, 2012
I think this is a bug. The question to ponder is backwards compatibility, specially if this is going to be backported to 2.7/3.2.
Chris, could you possibly ask for opinions in python-dev and/or python-list?
pitrou commentedon Aug 3, 2012
Do you have any argument?
ethanfurman commentedon Aug 3, 2012
The documentation says, "Returns a list of output lines"; an empty list is not a list of lines.
ethanfurman commentedon Aug 3, 2012
Not sure I would worry about fixing it in 2.7, although I don't have strong feelings about that.
pitrou commentedon Aug 3, 2012
Really?
>>> "".splitlines() []ethanfurman commentedon Aug 3, 2012
Really.
--> ''.split('\n')
['']
14 remaining items
ethanfurman commentedon Aug 8, 2012
Chris Jerdonek wrote:
s/paragraph/text/
Having now more carefully read the docs (which, admittedly, I should
have done before responding the first time) I found this:
In other words, it is not designed to work on multiple paragraphs at
once. And the documentation is fine.
Move along, no bug to see here, move along...
bitdancer commentedon Aug 8, 2012
Also you will note that the return of the empty list for an empty line is exactly what you want for wrapping multiple line-break-delimited paragraphs. Consider:
>>> doc = "a para\nanother para\n\na third, but with an extra blank line between\n" >>> for line in doc.splitlines(): ... print('\n'.join(textwrap.wrap(line, width=5))) ... if line: ... print() a paraanoth
er
para
a thi
rd,
but
with
an
extra
blank
line
betwe
en
In other words, we need to add a blank line after our formatted paragraph, unless it is empty, in which case we don't want to add one or we'll have an extra. This assumes that single-line-paragraphs do not have blank lines between them...if they do, then the algorithm is even simpler, as you don't need the if.
cjerdonek commentedon Aug 26, 2012
Here is a patch that documents and adds tests for the existing behavior (i.e. keeping the current behavior the same).
I also expanded the patch slightly to cover related edge cases that involve the interplay between whitespace, empty lines, and indenting (some of which came out of the discussions above).
cjerdonek commentedon Aug 29, 2012
Attaching an updated patch that improves the organization of the new test cases (test ordering, test names, and test comments, etc).
bitdancer commentedon Sep 1, 2012
Added some review comments on issue-15510-4.patch.
cjerdonek commentedon Sep 3, 2012
Thanks a lot for the very thorough review, David. I should be able to update the patch and respond to a couple of your points later today or tomorrow at the latest.
cjerdonek commentedon Sep 5, 2012
I responded to David's comments on the review tool. Later today I will update the patch in response to his comments (accommodating all of his suggestions) along with a couple other changes.
rhettinger commentedon Sep 5, 2012
FWIW, I agree that the existing behavior should not be changed. Most likely, a change would break some code that is currently working, and there would be little to no gain.
cjerdonek commentedon Sep 5, 2012
Attaching an updated patch as promised in my previous comment.
Note that I removed two edge test cases pertaining to leading whitespace. I would rather discuss those cases as part of a different issue to avoid making this thread even longer (and it is off the topic of the original issue).
python-dev commentedon Sep 8, 2012
New changeset 65e5f28801e1 by R David Murray in branch '3.2':
bpo-15510: clarify textwrap's handling of whitespace, and add confirming tests.
http://hg.python.org/cpython/rev/65e5f28801e1
New changeset 0e9ad455355b by R David Murray in branch 'default':
Merge bpo-15510: clarify textwrap's handling of whitespace, and add confirming tests.
http://hg.python.org/cpython/rev/0e9ad455355b
New changeset fca48971ac35 by R David Murray in branch '2.7':
bpo-15510: clarify textwrap's handling of whitespace, and add confirming tests.
http://hg.python.org/cpython/rev/fca48971ac35
bitdancer commentedon Sep 8, 2012
Thanks, Chris.