You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
zipfile.extractall fails in Posix shell with utf-8 filename #64528
Considering a zip file which contains utf-8 filenames (as uploaded zip file), the following code fails if launched in a Posix shell.
>>> with zipfile.ZipFile("test_ut8.zip") as fd:
... fd.extractall()
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/opt/python/3.3/lib/python3.3/zipfile.py", line 1225, in extractall
self.extract(zipinfo, path, pwd)
File "/opt/python/3.3/lib/python3.3/zipfile.py", line 1213, in extract
return self._extract_member(member, path, pwd)
File "/opt/python/3.3/lib/python3.3/zipfile.py", line 1276, in _extract_member
open(targetpath, "wb") as target:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 10-14: ordinal not in range(128)
With shell:
$ locale
LANG=POSIX
...
But filesystem is not encoding dependant. On a Unix system, filename are only bytes, there is no reason to refuse to unzip a zip file (in fact, "unzip" command line don't fail to unzip the file in a Posix shell).
Since "open" can take "bytes" filename, changing the line 1276 from
open(targetpath)
to:
open(targetpath.encode("utf-8"))
fixes the problem.
zipfile should not care about the encoding of the filename and should use the bytes sequence filename extracted directly from the bytes sequence of the zipfile. Having "ZipInfo.filename" as a string (and not bytes) is great for an API, but is not needed to open/write a file on the disk. Then, ZipInfo should store the direct bytes sequences of filename as a "bytes_filename" field and use it in the "open" of "extract".
In addition, considering the patch of bug 10614, the right patch could use the new "ZipInfo.encoding" field:
If you live in a current-posix world, this might make sense. However, one can also argue that the filename should be *transcoded* from the tarfile encoding to the local FS filename encoding, which I believe is what we are currently doing. Which, if you are using POSIX as the locale, will fail a lot. If you use a sensible modern locale that includes utf-8, you wouldn't have a problem.
Unfortunately, the reality is probably that sometimes you want one behavior and sometimes you want the other :(
Encoding using member.encoding is probably wrong, though. If you are trying to preserve the original bytes, is is probably best do so, and not assume that the tarfile encoding field is valid.
I'm adding Victor Stinner to nosy: he's thought about these issues much more deeply than I have. The answer may be that we will only support transcoding filenames in our tarfile module...and certainly it looks like doing anything else, even if we want to, would be a new feature.
LaurentMazuel commented on Jan 22, 2014
LaurentMazuelmannequin
MannequinAuthor
More actions
Thanks for your answer.
I think you can't transcode internal zip filenames to FS encoding. Actually, in Unix the FS only stores bytes for filename, there is no "FS encoding". Then, if you change your locale, the filename printed will change too in your console. If you transcode filename using the current locale, unzipping twice the same file with two different locales will lead to two different files, which is not (I think) you are intending for.
The problem will not arise in Windows (NTFS is UTF-16) nor MAC OSX (UTF-8)
Moreover, a simple "unzip" works like a charm. It doesn't care about encoding or current locale and extract the file using the initial bytes in the zip. Unzipping twice with the two different locales creates only one file.
The POSIX locale tells Python 3 to use ASCII for all operating system interfaces, including the standard streams. This is an antiquated behaviour in the POSIX spec that Python 3 doesn't currently work around.
bpo-19977 is a proposal to work around this limitation by default.
As an immediate workaround, it's possible to either set PYTHONIOENCODING explicitly so Python ignores the incorrect encoding claims from the OS, or else to do your own encoding and write directly to the sys.stdout.buffer binary interface.
Python 3.4 also allows setting *just* the default error handler for the streams, while still getting the encoding from the OS.
Unfortunately, I cannot test easily python 3.4 for now. But I have downloaded the source code and "diff" from 3.3 to 3.4 the "zipfile" module and see no difference relating to this problem. I can be wrong, maybe if some core improvement of Python may change something?
I created an environment under 3.3.1 in which this error was still occurring, but within that same environment, it is not occurring for 3.7. I believe this can be closed.
I created an environment under 3.3.1 in which this error was still occurring, but within that same environment, it is not occurring for 3.7. I believe this can be closed.
Python 3.7 now uses the UTF-8 encoding when the LC_CTYPE locale is POSIX (PEP-538, PEP-540). You should still be able to reproduce the bug with a locale with an encoding different than UTF-8.
Moreover, I understand that Python 3.6 is still affected by the bug.
I don't think that we can fix this bug, sadly. But I'm happy to see that the PEP-538 and PEP-540 are already useful!
Note that while the original issue was about the POSIX locale, you can still get the same error on non-UTF-8 locale. You perhaps can get similar error on macOS or other platforms depending on the filesystem when using non-normalized filenames, but we can little do here, if not provide a general hook for transforming filenames. But this is a different complex issue.
Activity
LaurentMazuel commentedon Jan 21, 2014
Hello,
Considering a zip file which contains utf-8 filenames (as uploaded zip file), the following code fails if launched in a Posix shell.
With shell:
$ locale
LANG=POSIX
...
But filesystem is not encoding dependant. On a Unix system, filename are only bytes, there is no reason to refuse to unzip a zip file (in fact, "unzip" command line don't fail to unzip the file in a Posix shell).
Since "open" can take "bytes" filename, changing the line 1276 from
fixes the problem.
zipfile should not care about the encoding of the filename and should use the bytes sequence filename extracted directly from the bytes sequence of the zipfile. Having "ZipInfo.filename" as a string (and not bytes) is great for an API, but is not needed to open/write a file on the disk. Then, ZipInfo should store the direct bytes sequences of filename as a "bytes_filename" field and use it in the "open" of "extract".
In addition, considering the patch of bug 10614, the right patch could use the new "ZipInfo.encoding" field:
bitdancer commentedon Jan 21, 2014
If you live in a current-posix world, this might make sense. However, one can also argue that the filename should be *transcoded* from the tarfile encoding to the local FS filename encoding, which I believe is what we are currently doing. Which, if you are using POSIX as the locale, will fail a lot. If you use a sensible modern locale that includes utf-8, you wouldn't have a problem.
Unfortunately, the reality is probably that sometimes you want one behavior and sometimes you want the other :(
Encoding using member.encoding is probably wrong, though. If you are trying to preserve the original bytes, is is probably best do so, and not assume that the tarfile encoding field is valid.
I'm adding Victor Stinner to nosy: he's thought about these issues much more deeply than I have. The answer may be that we will only support transcoding filenames in our tarfile module...and certainly it looks like doing anything else, even if we want to, would be a new feature.
LaurentMazuel commentedon Jan 22, 2014
Thanks for your answer.
I think you can't transcode internal zip filenames to FS encoding. Actually, in Unix the FS only stores bytes for filename, there is no "FS encoding". Then, if you change your locale, the filename printed will change too in your console. If you transcode filename using the current locale, unzipping twice the same file with two different locales will lead to two different files, which is not (I think) you are intending for.
The problem will not arise in Windows (NTFS is UTF-16) nor MAC OSX (UTF-8)
Moreover, a simple "unzip" works like a charm. It doesn't care about encoding or current locale and extract the file using the initial bytes in the zip. Unzipping twice with the two different locales creates only one file.
An interesting link (even if it is not an official reference):
http://unix.stackexchange.com/questions/2089/what-charset-encoding-is-used-for-filenames-and-paths-on-linux
bitdancer commentedon Jan 22, 2014
Believe me, we are *well* aware of the issue that linux stores filenames as bytes.
I agree that the inability to always transcode is an issue. That's why I'd like the opinion of someone who has studied this problem in more depth.
ncoghlan commentedon Jan 22, 2014
The POSIX locale tells Python 3 to use ASCII for all operating system interfaces, including the standard streams. This is an antiquated behaviour in the POSIX spec that Python 3 doesn't currently work around.
bpo-19977 is a proposal to work around this limitation by default.
As an immediate workaround, it's possible to either set PYTHONIOENCODING explicitly so Python ignores the incorrect encoding claims from the OS, or else to do your own encoding and write directly to the sys.stdout.buffer binary interface.
Python 3.4 also allows setting *just* the default error handler for the streams, while still getting the encoding from the OS.
ncoghlan commentedon Jan 22, 2014
My apologies, I completely misread the issue and thought it was related to displaying file names, rather than opening them.
I believe Python 3.4 includes some changes in this area - are you in a position to retry this on the latest 3.4 beta release?
LaurentMazuel commentedon Feb 27, 2014
Thank for your answer.
Unfortunately, I cannot test easily python 3.4 for now. But I have downloaded the source code and "diff" from 3.3 to 3.4 the "zipfile" module and see no difference relating to this problem. I can be wrong, maybe if some core improvement of Python may change something?
csabella commentedon Dec 14, 2017
I created an environment under 3.3.1 in which this error was still occurring, but within that same environment, it is not occurring for 3.7. I believe this can be closed.
vstinner commentedon Dec 15, 2017
Python 3.7 now uses the UTF-8 encoding when the LC_CTYPE locale is POSIX (PEP-538, PEP-540). You should still be able to reproduce the bug with a locale with an encoding different than UTF-8.
Moreover, I understand that Python 3.6 is still affected by the bug.
I don't think that we can fix this bug, sadly. But I'm happy to see that the PEP-538 and PEP-540 are already useful!
vstinner commentedon Dec 18, 2017
Oops, I mean "we cannot *close* this bug" (right now). Sorry.
I mean that IMHO we still have to fix the bug.
serhiy-storchaka commentedon May 18, 2026
I think we can finally close it now.
Note that while the original issue was about the POSIX locale, you can still get the same error on non-UTF-8 locale. You perhaps can get similar error on macOS or other platforms depending on the filesystem when using non-normalized filenames, but we can little do here, if not provide a general hook for transforming filenames. But this is a different complex issue.