Skip to content

zipfile.extractall fails in Posix shell with utf-8 filename #64528

Closed as not planned

Description

@LaurentMazuel
BPO 20329
Nosy @ncoghlan, @vstinner, @bitdancer, @serhiy-storchaka, @csabella
Files
  • test_ut8.zip: Zip where filenames are in UTF-8
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2014-01-21.15:05:53.908>
    labels = ['extension-modules', 'type-bug']
    title = 'zipfile.extractall fails in Posix shell with utf-8 filename'
    updated_at = <Date 2017-12-18.14:39:09.490>
    user = 'https://bugs.python.org/LaurentMazuel'

    bugs.python.org fields:

    activity = <Date 2017-12-18.14:39:09.490>
    actor = 'vstinner'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Extension Modules']
    creation = <Date 2014-01-21.15:05:53.908>
    creator = 'Laurent.Mazuel'
    dependencies = []
    files = ['33589']
    hgrepos = []
    issue_num = 20329
    keywords = []
    message_count = 10.0
    messages = ['208648', '208655', '208755', '208817', '208858', '208859', '212349', '308345', '308350', '308568']
    nosy_count = 6.0
    nosy_names = ['ncoghlan', 'vstinner', 'r.david.murray', 'Laurent.Mazuel', 'serhiy.storchaka', 'cheryl.sabella']
    pr_nums = []
    priority = 'normal'
    resolution = None
    stage = None
    status = 'open'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue20329'
    versions = ['Python 3.3']

    Activity

    1. LaurentMazuel commented on Jan 21, 2014

      LaurentMazuelmannequin
      MannequinAuthor

      Hello,

      Considering a zip file which contains utf-8 filenames (as uploaded zip file), the following code fails if launched in a Posix shell.

      >>> with zipfile.ZipFile("test_ut8.zip") as fd:
      ...     fd.extractall()
      ... 
      Traceback (most recent call last):
        File "<stdin>", line 2, in <module>
        File "/opt/python/3.3/lib/python3.3/zipfile.py", line 1225, in extractall
          self.extract(zipinfo, path, pwd)
        File "/opt/python/3.3/lib/python3.3/zipfile.py", line 1213, in extract
          return self._extract_member(member, path, pwd)
        File "/opt/python/3.3/lib/python3.3/zipfile.py", line 1276, in _extract_member
          open(targetpath, "wb") as target:
      UnicodeEncodeError: 'ascii' codec can't encode characters in position 10-14: ordinal not in range(128)

      With shell:
      $ locale
      LANG=POSIX
      ...

      But filesystem is not encoding dependant. On a Unix system, filename are only bytes, there is no reason to refuse to unzip a zip file (in fact, "unzip" command line don't fail to unzip the file in a Posix shell).

      Since "open" can take "bytes" filename, changing the line 1276 from

      open(targetpath)
      to:
      open(targetpath.encode("utf-8"))

      fixes the problem.

      zipfile should not care about the encoding of the filename and should use the bytes sequence filename extracted directly from the bytes sequence of the zipfile. Having "ZipInfo.filename" as a string (and not bytes) is great for an API, but is not needed to open/write a file on the disk. Then, ZipInfo should store the direct bytes sequences of filename as a "bytes_filename" field and use it in the "open" of "extract".

      In addition, considering the patch of bug 10614, the right patch could use the new "ZipInfo.encoding" field:

      open(targetpath.encode(member.encoding))

    2. bitdancer commented on Jan 21, 2014

      @bitdancer
      Member

      If you live in a current-posix world, this might make sense. However, one can also argue that the filename should be *transcoded* from the tarfile encoding to the local FS filename encoding, which I believe is what we are currently doing. Which, if you are using POSIX as the locale, will fail a lot. If you use a sensible modern locale that includes utf-8, you wouldn't have a problem.

      Unfortunately, the reality is probably that sometimes you want one behavior and sometimes you want the other :(

      Encoding using member.encoding is probably wrong, though. If you are trying to preserve the original bytes, is is probably best do so, and not assume that the tarfile encoding field is valid.

      I'm adding Victor Stinner to nosy: he's thought about these issues much more deeply than I have. The answer may be that we will only support transcoding filenames in our tarfile module...and certainly it looks like doing anything else, even if we want to, would be a new feature.

    3. LaurentMazuel commented on Jan 22, 2014

      LaurentMazuelmannequin
      MannequinAuthor

      Thanks for your answer.

      I think you can't transcode internal zip filenames to FS encoding. Actually, in Unix the FS only stores bytes for filename, there is no "FS encoding". Then, if you change your locale, the filename printed will change too in your console. If you transcode filename using the current locale, unzipping twice the same file with two different locales will lead to two different files, which is not (I think) you are intending for.
      The problem will not arise in Windows (NTFS is UTF-16) nor MAC OSX (UTF-8)

      Moreover, a simple "unzip" works like a charm. It doesn't care about encoding or current locale and extract the file using the initial bytes in the zip. Unzipping twice with the two different locales creates only one file.

      An interesting link (even if it is not an official reference):
      http://unix.stackexchange.com/questions/2089/what-charset-encoding-is-used-for-filenames-and-paths-on-linux

    4. bitdancer commented on Jan 22, 2014

      @bitdancer
      Member

      Believe me, we are *well* aware of the issue that linux stores filenames as bytes.

      I agree that the inability to always transcode is an issue. That's why I'd like the opinion of someone who has studied this problem in more depth.

    5. ncoghlan commented on Jan 22, 2014

      @ncoghlan
      Contributor

      The POSIX locale tells Python 3 to use ASCII for all operating system interfaces, including the standard streams. This is an antiquated behaviour in the POSIX spec that Python 3 doesn't currently work around.

      bpo-19977 is a proposal to work around this limitation by default.

      As an immediate workaround, it's possible to either set PYTHONIOENCODING explicitly so Python ignores the incorrect encoding claims from the OS, or else to do your own encoding and write directly to the sys.stdout.buffer binary interface.

      Python 3.4 also allows setting *just* the default error handler for the streams, while still getting the encoding from the OS.

    6. ncoghlan commented on Jan 22, 2014

      @ncoghlan
      Contributor

      My apologies, I completely misread the issue and thought it was related to displaying file names, rather than opening them.

      I believe Python 3.4 includes some changes in this area - are you in a position to retry this on the latest 3.4 beta release?

    7. LaurentMazuel commented on Feb 27, 2014

      LaurentMazuelmannequin
      MannequinAuthor

      Thank for your answer.

      Unfortunately, I cannot test easily python 3.4 for now. But I have downloaded the source code and "diff" from 3.3 to 3.4 the "zipfile" module and see no difference relating to this problem. I can be wrong, maybe if some core improvement of Python may change something?

    8. csabella commented on Dec 14, 2017

      @csabella
      Contributor

      I created an environment under 3.3.1 in which this error was still occurring, but within that same environment, it is not occurring for 3.7. I believe this can be closed.

    9. vstinner commented on Dec 15, 2017

      @vstinner
      Member

      I created an environment under 3.3.1 in which this error was still occurring, but within that same environment, it is not occurring for 3.7. I believe this can be closed.

      Python 3.7 now uses the UTF-8 encoding when the LC_CTYPE locale is POSIX (PEP-538, PEP-540). You should still be able to reproduce the bug with a locale with an encoding different than UTF-8.

      Moreover, I understand that Python 3.6 is still affected by the bug.

      I don't think that we can fix this bug, sadly. But I'm happy to see that the PEP-538 and PEP-540 are already useful!

    10. vstinner commented on Dec 18, 2017

      @vstinner
      Member

      I don't think that we can fix this bug, sadly. But I'm happy to see that the PEP-538 and PEP-540 are already useful!

      Oops, I mean "we cannot *close* this bug" (right now). Sorry.

      I mean that IMHO we still have to fix the bug.

    11. transferred this issue fromon Apr 10, 2022
    12. added
      stdlibStandard Library Python modules in the Lib/ directory
      on Nov 24, 2023
    13. serhiy-storchaka commented on May 18, 2026

      @serhiy-storchaka
      Member

      I think we can finally close it now.

      Note that while the original issue was about the POSIX locale, you can still get the same error on non-UTF-8 locale. You perhaps can get similar error on macOS or other platforms depending on the filesystem when using non-normalized filenames, but we can little do here, if not provide a general hook for transforming filenames. But this is a different complex issue.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      extension-modulesC modules in the Modules dirstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

      Projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions