Skip to content

cElementTree.iterparse & ElementTree.iterparse return differently encoded strings #50515

Closed

Description

@NeilMuller
BPO 6266
Nosy @florentx
Superseder
  • bpo-6472: Update ElementTree with upstream changes
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2010-03-11.15:57:40.682>
    created_at = <Date 2009-06-11.10:53:51.753>
    labels = ['expert-XML', 'type-bug']
    title = 'cElementTree.iterparse & ElementTree.iterparse return differently encoded strings'
    updated_at = <Date 2010-03-11.15:57:40.680>
    user = 'https://bugs.python.org/NeilMuller'

    bugs.python.org fields:

    activity = <Date 2010-03-11.15:57:40.680>
    actor = 'flox'
    assignee = 'none'
    closed = True
    closed_date = <Date 2010-03-11.15:57:40.682>
    closer = 'flox'
    components = ['XML']
    creation = <Date 2009-06-11.10:53:51.753>
    creator = 'Neil Muller'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 6266
    keywords = ['patch']
    message_count = 9.0
    messages = ['89248', '89550', '89551', '89552', '89560', '89564', '89568', '99442', '100871']
    nosy_count = 5.0
    nosy_names = ['effbot', 'Neil Muller', 'jerith', 'nlopes', 'flox']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = '6472'
    type = 'behavior'
    url = 'https://bugs.python.org/issue6266'
    versions = ['Python 2.7']

    Activity

    1. NeilMuller commented on Jun 11, 2009

      NeilMullermannequin
      MannequinAuthor

      Consider:

      >>> from StringIO import StringIO
      >>> source = StringIO('<body xmlns="http://&#233;ffbot.org/ns">text</body>')
      >>> import xml.etree.ElementTree as ET
      >>> events = ("start-ns",)
      >>> context = ET.iterparse(source, events)
      >>> for action, elem in context:
      ...    print action, elem
      ... 
      start-ns ('', u'http://\xe9ffbot.org/ns')
      >>> source.seek(0)
      >>> import xml.etree.cElementTree as cET
      >>> context = cET.iterparse(source, events)
      >>> for action, elem in context:
      ...    print action, elem
      ... 
      start-ns ('', 'http://\xc3\xa9ffbot.org/ns')

      I'm not sure which is more correct here, but unsing different encodings
      in the result is somewhat unexpected.

    2. nlopes commented on Jun 20, 2009

      nlopesmannequin
      Mannequin

      This is a pretty dumb patch, but it does it's job.
      Basically it decodes the utf-8 encoded prefix and uri. Then, encodes it
      into Latin1. Probably there are better ways of doing this and those
      ideas are welcome. Patch attached.

    3. effbot commented on Jun 21, 2009

      effbotmannequin
      Mannequin

      Converting from UTF-8 to Unicode is the right thing to do, but
      converting back to Latin-1 is not correct -- note that ET returns a
      Unicode string, not an 8-bit string. There's a "makestring" helper that
      does the right thing in the library; just changing:

      parcel = Py_BuildValue("ss", (prefix) ? prefix : "", uri);

      to

      parcel = Py_BuildValue("sN", (prefix) ? prefix : "", makestring(uri));

      should work (even if you should probably do that in two steps, and look
      for errors from makestring before proceeding).

    4. nlopes commented on Jun 21, 2009

      nlopesmannequin
      Mannequin

      You're right about the conversion to Latin1.
      I actually played a bit with makestring before going in another
      direction (although not very good) because makestring alone wasn't
      giving what is intended.

      I'll try to check tomorrow a good approach for this (already had that in
      mind).

    5. effbot commented on Jun 21, 2009

      effbotmannequin
      Mannequin

      It should definitely give what's intended (either a Unicode string, or, if
      the content is plain ASCII, an 8-bit string). What did you get instead?

    6. nlopes commented on Jun 21, 2009

      nlopesmannequin
      Mannequin

      I got pure gibberish output, but I know why. It was a compilation gone
      wrong.

      To get the output as ElementTree, I think instead of

      parcel = Py_BuildValue("sN", (prefix) ? prefix : "", makestring(uri));

      it should be

      parcel = Py_BuildValue("sN", (prefix) ? prefix : "",
      PyUnicode_AsUnicode(makestring(uri), strlen(uri)));

      Else it will not be the expected result.

      Or am I overseeing something?

    7. nlopes commented on Jun 21, 2009

      nlopesmannequin
      Mannequin

      Don't mind what I just said. I overlooked the N. I couldn't figure out
      what was going wrong with your solution.

      That works. Mine is a ... aham.

      :)

    8. florentx commented on Feb 16, 2010

      florentxmannequin
      Mannequin

      Merged with the upstream patch proposed on bpo-6472 (with test case).

    9. florentx commented on Mar 11, 2010

      florentxmannequin
      Mannequin

      Fixed with bpo-6472.

    10. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      topic-XMLtype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions