Skip to content

rfc822 long header continuation broken #35918

Closed

Description

@richard
mannequin
BPO 504152
Nosy @loewis, @warsaw, @bitdancer
Superseder
  • bpo-12586: Provisional new email API: new policy implementing custom header objects
  • Files
  • rfc822.diff
  • email_test.diff
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/bitdancer'
    closed_at = <Date 2012-05-16.01:55:58.075>
    created_at = <Date 2002-01-16.01:31:17.000>
    labels = ['type-bug', 'library']
    title = 'rfc822 long header continuation broken'
    updated_at = <Date 2012-05-16.01:55:58.074>
    user = 'https://bugs.python.org/richard'

    bugs.python.org fields:

    activity = <Date 2012-05-16.01:55:58.074>
    actor = 'r.david.murray'
    assignee = 'r.david.murray'
    closed = True
    closed_date = <Date 2012-05-16.01:55:58.075>
    closer = 'r.david.murray'
    components = ['Library (Lib)']
    creation = <Date 2002-01-16.01:31:17.000>
    creator = 'richard'
    dependencies = []
    files = ['8357', '18555']
    hgrepos = []
    issue_num = 504152
    keywords = ['patch']
    message_count = 15.0
    messages = ['8766', '8767', '8768', '8769', '8770', '8771', '8772', '8773', '55454', '55457', '55524', '71573', '85676', '114158', '160794']
    nosy_count = 5.0
    nosy_names = ['loewis', 'barry', 'richard', 'r.david.murray', 'BreamoreBoy']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'needs patch'
    status = 'closed'
    superseder = '12586'
    type = 'behavior'
    url = 'https://bugs.python.org/issue504152'
    versions = ['Python 3.1', 'Python 2.7', 'Python 3.2', 'Python 3.3']

    Activity

    1. richard commented on Jan 16, 2002

      richardmannequin
      MannequinAuthor

      I don't believe this is fixed in 2.1.2 or 2.2, but
      haven't checked.

      The code in rfc822.Message.readheaders incorrectly
      unfolds long message headers. The relevant information
      from rfc2822 is in section 2.2.3. In short:

      """
      The process of moving from this folded multiple-line
      representation of a header field to its single line
      representation is called "unfolding". Unfolding is
      accomplished by simply removing any CRLF that is
      immediately followed by WSP. Each header field should
      be treated in its unfolded form for further syntactic
      and semantic evaluation.
      """

      This means that the code in readheaders:

                  if headerseen and line[0] in ' \t':
                      # It's a continuation line.
                      list.append(line)
                      x = (self.dict[headerseen] + "\n " +
      line.strip())
                      self.dict[headerseen] = x.strip()
                      continue

      should be:

                  if headerseen and line[0] in ' \t':
                      # It's a continuation line.
                      list.append(line)
                      x = self.dict[headerseen] + line
                      self.dict[headerseen] = x.strip()
                      continue

      ie. no stripping of the leading whitespace and no
      adding the newline.

    2. added
      stdlibStandard Library Python modules in the Lib/ directory
      on Jan 16, 2002
    3. added
      stdlibStandard Library Python modules in the Lib/ directory
      on Jan 16, 2002
    4. gvanrossum commented on Jan 16, 2002

      @gvanrossum
      Member

      Logged In: YES
      user_id=6380

      Richard, have you found a situation where it matters? I
      thought that usually the next phase calls for normalizing
      whitespace by squashing repeated spaces/tabs and removing
      them from front and back.

    5. richard commented on Jan 16, 2002

      richardmannequin
      MannequinAuthor

      Logged In: YES
      user_id=6405

      Yes, we had someone submit a bug report on the roundup
      users mailing list because someone had sent a message to
      the roundup mail gateway which was split. The client was
      extra-specially broken, since it split in the middle of a
      word (which is not to spec), but the more general case of
      folding on whitespace will cause roundup problems since I
      hadn't expected there to be any newlines in the header.

      I can modify roundup to strip out the newline, but it'd be
      nice to have rfc822.Message not put it in there...

    6. loewis commented on Jan 16, 2002

      loewismannequin
      Mannequin

      Logged In: YES
      user_id=21627

      Even though it might not matter, I don't think changing it
      would hurt, either, and the change brings it definitely
      closer to following the word of RFC 2822.

      If no case is brought forward where it matters, fixing it
      for 2.3 alone should be sufficient.

    7. warsaw commented on Apr 15, 2002

      @warsaw
      Member

      Logged In: YES
      user_id=12800

      There is some value in not unfolding long lines by default.
      FWIW, the email package also retains the line breaks for
      such multi-line headers. The advantage to retaining this is
      that message input/output can be idempotent (i.e. you get
      the same thing in as you get out). This can be useful when
      using the message to generate a hash value, and for other
      user-friendly reasons.

      That being said, there is also some use in providing a way
      to return the unfolded line. I don't see a lot of benefit
      in adding such a feature to the rfc822 module, but I could
      see adding it to the email package. Specifically, I would
      propose to add it to the email.Header.Header class, either
      as a separate method (e.g. Header.unfold()) or as a default
      argument to the Header.encode() method (e.g.
      Header.encode(self, unfold=0)).

      If we did the latter, then I'd change Header.__str__() to
      call .encode(unfold=1).

      Assigning to Ben to get his feedback. Ben, feel free to
      comment and re-assign this bug to me.

    8. richard commented on Nov 10, 2003

      richardmannequin
      MannequinAuthor

      Logged In: YES
      user_id=6405

      Hurm. This issue has been lost to the void, but it's as valid today as it
      ever was. I've just had another user of Roundup run into the same thing:

      RE: [issue51] Mails being delayed
      [assignedto=stuartm;priority=medium]

      (that should be one long line) became

      RE: [issue51] Mails being delayed [assignedto=stuartm;priority=me
      dium]

      when sent by Outlook. Note that the current code reconstructs that line
      as "me\ndium" which is about as wrong as it can get, as there's no way
      for my code to determine whether that *should* be "me dium" or
      "medium" since the other whitespace has been stripped (so just stripping
      out the newline, as my code currently does, doesn't help).

      I stand by my original post, requesting that the code be fixed as stated.

    9. warsaw commented on Nov 10, 2003

      @warsaw
      Member

      Logged In: YES
      user_id=12800

      Since this was never addressed in the email package either,
      perhaps you'd like to bring it up in the email-sig?

    10. richard commented on Nov 10, 2003

      richardmannequin
      MannequinAuthor

      Logged In: YES
      user_id=6405

      OK, I've sent a message, but I don't have the time to sign up to the list.

    11. smontanaro commented on Aug 30, 2007

      @smontanaro
      Contributor

      Is this still an issue? No activity since 2003-11.

    12. gvanrossum commented on Aug 30, 2007

      @gvanrossum
      Member

      How about this patch? It basically does

      self.dict[headerseen] += line.rstrip()

      which should be what the RFC prescribes, maintaining the invariant that
      the dict values don't end in whitespace.

      I haven't written a test for this behavior.

    13. gvanrossum commented on Aug 31, 2007

      @gvanrossum
      Member

      Barry, what do you think of this patch? How does the email package
      handle this case?

    14. kcarnold commented on Aug 20, 2008

      kcarnoldmannequin
      Mannequin

      This issue still seems to be present in Python 2.5's email module.

      feedparser.py line 444-445 says:

      # XXX reconsider the joining of folded lines
      lhdr = EMPTYSTRING.join(lastvalue)[:-1].rstrip('\r\n')

      I think that should be something like:
      lhdr = EMPTYSTRING.join(ln.rstrip('\r\n') for ln in lastvalue)[:-1])

      The resulting headers still need a fair amount of massaging, though; why
      not just use Header instances for the headers?

    15. bitdancer commented on Apr 6, 2009

      @bitdancer
      Member

      The source lines mentioned in this issue have not been changed in trunk,
      and the feedparser line has not been changed in py3k as of r71355
      (rfc822 no longer exists, so I'm not sure if the replacement code in
      py3k has the issue or not).

    16. assigned and unassigned on May 5, 2010
    17. BreamoreBoy commented on Aug 17, 2010

      BreamoreBoymannequin
      Mannequin

      Confirmed still an issue in py3k by using the email subject line from msg8771 and adding two extra test cases to TestParsers got two failures. Tried several variations of the patch from msg71573 (in the original the parantheses are unbalanced) and pushed the number of failures to over 80. I've attached a patch against the unit test file, note that the comments will need changing.

    18. bitdancer commented on May 16, 2012

      @bitdancer
      Member

      The email package no longer strips the leading whitespace. It doesn't unfold the headers, but changing that at this stage has untenable consequences.

      However, the email package in 3.3 will have a provisional API that will provide the correct unfolding of headers in an easy to use way. So I'm closing this in favor of the issue where I will add that API.

    19. transferred this issue fromon Apr 9, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions