Skip to content

UnicodeDecodeError when retrieving binary data from cgi.FieldStorage() #51103

Closed

Description

@loveminix
mannequin
BPO 6854
Nosy @terryjreedy, @ezio-melotti, @bitdancer
Superseder
  • bpo-4953: cgi module cannot handle POST with multipart/form-data in 3.x
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2010-01-10.17:16:40.113>
    created_at = <Date 2009-09-07.05:13:32.083>
    labels = ['type-bug', 'library']
    title = 'UnicodeDecodeError when retrieving binary data from cgi.FieldStorage()'
    updated_at = <Date 2010-01-10.17:16:40.099>
    user = 'https://bugs.python.org/loveminix'

    bugs.python.org fields:

    activity = <Date 2010-01-10.17:16:40.099>
    actor = 'r.david.murray'
    assignee = 'none'
    closed = True
    closed_date = <Date 2010-01-10.17:16:40.113>
    closer = 'r.david.murray'
    components = ['Library (Lib)']
    creation = <Date 2009-09-07.05:13:32.083>
    creator = 'loveminix'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 6854
    keywords = []
    message_count = 6.0
    messages = ['92343', '92360', '92385', '92470', '92520', '97523']
    nosy_count = 4.0
    nosy_names = ['terry.reedy', 'ezio.melotti', 'r.david.murray', 'loveminix']
    pr_nums = []
    priority = 'normal'
    resolution = 'duplicate'
    stage = 'resolved'
    status = 'closed'
    superseder = '4953'
    type = 'behavior'
    url = 'https://bugs.python.org/issue6854'
    versions = ['Python 3.1']

    Activity

    1. loveminix commented on Sep 7, 2009

      loveminixmannequin
      MannequinAuthor

      The following cgi applet uploads a binary file to the server. It gets a
      "UnicodeDecodeError" inside cgi.FieldStorage(). The same code works in
      python 2.6.

      #! /usr/bin/python3

      import os, cgi;
      import cgitb; cgitb.enable();
      
      pathInfo = os.environ.get("PATH_INFO", "");
      serverName = os.environ.get("SERVER_NAME", "");
      scriptName = os.environ.get("SCRIPT_NAME", "");
      
      if pathInfo == "":
          print(
      """\
      Content-type: text/html
      """
      );
          print(
      """\
      <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
      "http://www.w3.org/TR/html4/strict.dtd">
      <html>
      <head>
          <meta http-equiv="Content-type" content="text/html;charset=UTF-8">
      </head>
      <body>
          <div>
              <form action="/?url=http%3A%2F%2F%7B0%7D%7B1%7D%2Fupload"
      enctype="multipart/form-data" method="post">
              <p>
                  <input type="file" name="file"><br>
                  <input type="submit" value="Upload">
              </p>
              </form>
          </div>
      </body>
      </html>\
      """.format(serverName, scriptName)
      );
      elif pathInfo == "/upload":
          fieldStorage = cgi.FieldStorage();
          fileItem = fieldStorage["file"];
          if fileItem.filename != "":
              file = open("/tmp/test.txt", mode="wb");
              file.write(fileItem.file.read());
              file.close();
              print(
      """\
      Content-type: text/html
      """
      );
              print(
      """\
      <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
      "http://www.w3.org/TR/html4/strict.dtd">
      <html>
      <head>
          <meta http-equiv="Content-type" content="text/html;charset=UTF-8">
      </head>
      <body>
          <div>
              <p>Success</p>
          </div>
      </body>
      </html>\
      """
      );
    2. added
      stdlibStandard Library Python modules in the Lib/ directory
      type-bugAn unexpected behavior, bug, or error
      on Sep 7, 2009
    3. ezio-melotti commented on Sep 7, 2009

      @ezio-melotti
      Member

      Can you paste the traceback of the error?

    4. loveminix commented on Sep 7, 2009

      loveminixmannequin
      MannequinAuthor

      Here is the trackback (the uploaded file is a PDF file):

      UnicodeDecodeError Python 3.1.1: /home/chu7/software/bin/python3
      Mon Sep 7 12:31:07 2009

      A problem occurred in a Python script. Here is the sequence of function
      calls leading up to the error, in the order they occurred.

      /home/chu7/web/cgi-bin/upload.py in ()
      35 );

       36 elif pathInfo == "/upload":
      

      => 37 fieldStorage = cgi.FieldStorage();

       38     fileItem = fieldStorage["file"];
      
       39     if fileItem.filename != "":
      

      fieldStorage undefined, cgi = <module 'cgi'
      from '/home/chu7/software/lib/python3.1/cgi.py'>, cgi.FieldStorage =
      <class 'cgi.FieldStorage'>
      /home/chu7/software/lib/python3.1/cgi.py in __init__(self=FieldStorage
      (None, None, []), fp=None, headers={'content-length': '76784', 'content-
      type': 'multipart/form-data; boundary=---------------------------
      7d95563062a'}, outerboundary='', environ=<os._Environ object at
      0x8ee040c>, keep_blank_values=0, strict_parsing=0)
      489 self.read_urlencoded()

      490         elif ctype[:10] == 'multipart/':
      

      => 491 self.read_multi(environ, keep_blank_values,
      strict_parsing)

      492         else:
      
      493             self.read_single()
      

      self = FieldStorage(None, None, []), self.read_multi = <bound method
      FieldStorage.read_multi of FieldStorage(None, None, [])>, environ =
      <os._Environ object at 0x8ee040c>, keep_blank_values = 0,
      strict_parsing = 0
      /home/chu7/software/lib/python3.1/cgi.py in read_multi
      (self=FieldStorage(None, None, []), environ=<os._Environ object at
      0x8ee040c>, keep_blank_values=0, strict_parsing=0)
      609 # Create bogus content-type header for proper multipart
      parsing

      610         parser.feed('Content-Type: %s; boundary=%s\\r\\n\\r\\n' % 
      

      (self.type, ib))

      => 611 parser.feed(self.fp.read())

      612         full_msg = parser.close()
      
      613         # Get subparts
      

      parser = <email.feedparser.FeedParser object at 0x910daac>, parser.feed
      = <bound method FeedParser.feed of <email.feedparser.FeedParser object
      at 0x910daac>>, self = FieldStorage(None, None, []), self.fp =
      <_io.TextIOWrapper name='<stdin>' encoding='ANSI_X3.4-1968'>,
      self.fp.read = <built-in method read of _io.TextIOWrapper object at
      0x8ee833c>
      /home/chu7/software/lib/python3.1/encodings/ascii.py in decode
      (self=<encodings.ascii.IncrementalDecoder object at 0x8f6b74c>,
      input=b'-----------------------------7d95563062a\r\nCo...\n-------------
      ----------------7d95563062a--\r\n', final=True)
      24 class IncrementalDecoder(codecs.IncrementalDecoder):

       25     def decode(self, input, final=False):
      

      => 26 return codecs.ascii_decode(input, self.errors)[0]

       27 
      
       28 class StreamWriter(Codec,codecs.StreamWriter):
      

      global codecs = <module 'codecs'
      from '/home/chu7/software/lib/python3.1/codecs.py'>,
      codecs.ascii_decode = <built-in function ascii_decode>, input = b'------
      -----------------------7d95563062a\r\nCo...\n---------------------------
      --7d95563062a--\r\n', self = <encodings.ascii.IncrementalDecoder object
      at 0x8f6b74c>, self.errors = 'strict'

      UnicodeDecodeError: 'ascii' codec can't decode byte 0xc7 in position
      158: ordinal not in range(128)
      args = ('ascii', b'-----------------------------
      7d95563062a\r\nCo...\n-----------------------------7d95563062a--\r\n',
      158, 159, 'ordinal not in range(128)')
      encoding = 'ascii'
      end = 159
      object = b'-----------------------------7d95563062a\r\nCo...\n----
      -------------------------7d95563062a--\r\n'
      reason = 'ordinal not in range(128)'
      start = 158
      with_traceback = <built-in method with_traceback of
      UnicodeDecodeError object at 0x905bd2c>

    5. loveminix commented on Sep 9, 2009

      loveminixmannequin
      MannequinAuthor

      Is there an update on this? Let me know if more information is needed.

    6. terryjreedy commented on Sep 11, 2009

      @terryjreedy
      Member
      1. When you have a problem, and you are not sure it is an error in the
        interpreter or stdlib, try posting to python-list first. If you do not
        soon get a definitive determination here, do that. This is not a
        question-answering or user-code debugging list.

      2. When posting code so others can read it, please leave off trailing
        semi-colons. Python is not C. The unnecessary ';'s are distracting noise.

      3. When posting problematic code, please strip it down to the minimum
        necessary to show the problem. From what I understand, the minimum here is
        "import cgi; d = cgi.FieldStorage()". The rest is distracting noise.

      4. When you get a error traceback, copy and paste *the whole traceback*
        (especially when requested).

      5. Post the exact interpreter version used. Is this with 3.1.1, with
        all the latest bug fixes? Even better, try with the lastest version and
        say so.

      6. When the problem involved interacting with the external system (such
        as os.environ), specify the os. It often makes a difference.

      In this case, your problem seems to be that the relevant field of
      os.environ has a non-ascii char. My impression is that this is not
      intented to be allowed, at least for posix systems, but I am not sure.
      It is possible that something in the module has not been updated
      properly, but I do not know enough of the specs to say whether the
      problem is your data or the library code.

      You should determine the 'offending' string and print out its
      representation (with repr(s)) and include that with any further post
      here or on python-list. In general, include offending data (also
      minimized) along with offending code.

    7. bitdancer commented on Jan 10, 2010

      @bitdancer
      Member

      This appears to be a duplicate of bpo-4953.

    8. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions