Skip to content

decoding functions in _codecs module accept str arguments #49124

Closed

Description

@pitrou
BPO 4874
Nosy @malemburg, @amauryfa, @pitrou, @vstinner
Files
  • _codecs_bytes-2.patch
  • mbdecode-unicode.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/pitrou'
    closed_at = <Date 2009-01-22.12:25:56.020>
    created_at = <Date 2009-01-07.23:48:15.674>
    labels = ['extension-modules', 'type-bug', 'release-blocker']
    title = 'decoding functions in _codecs module accept str arguments'
    updated_at = <Date 2009-01-22.12:25:56.019>
    user = 'https://github.com/pitrou'

    bugs.python.org fields:

    activity = <Date 2009-01-22.12:25:56.019>
    actor = 'pitrou'
    assignee = 'pitrou'
    closed = True
    closed_date = <Date 2009-01-22.12:25:56.020>
    closer = 'pitrou'
    components = ['Extension Modules']
    creation = <Date 2009-01-07.23:48:15.674>
    creator = 'pitrou'
    dependencies = []
    files = ['12770', '12831']
    hgrepos = []
    issue_num = 4874
    keywords = ['patch']
    message_count = 12.0
    messages = ['79384', '79387', '79402', '79741', '79843', '79991', '80362', '80363', '80364', '80365', '80366', '80367']
    nosy_count = 4.0
    nosy_names = ['lemburg', 'amaury.forgeotdarc', 'pitrou', 'vstinner']
    pr_nums = []
    priority = 'release blocker'
    resolution = 'fixed'
    stage = None
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue4874'
    versions = ['Python 3.0', 'Python 3.1']

    Activity

    1. pitrou commented on Jan 7, 2009

      @pitrou
      MemberAuthor

      The following function calls should raise a TypeError instead. Encoding
      functions are fine (they only accept str).

      >>> import codecs
      >>> codecs.utf_8_decode('aa')
      ('aa', 2)
      >>> codecs.utf_8_decode('éé')
      ('éé', 4)
      >>> codecs.latin_1_decode('éé')
      ('éé', 4)
    2. vstinner commented on Jan 8, 2009

      @vstinner
      Member

      Patch replacing "s*" parsing format by "y*" for:

      • utf_7_decode()
      • utf_8_decode()
      • utf_16_decode()
      • utf_16_le_decode()
      • utf_16_be_decode()
      • utf_16_ex_decode()
      • utf_32_decode()
      • utf_32_le_decode()
      • utf_32_be_decode()
      • utf_32_ex_decode()
      • unicode_escape_decode()
      • raw_unicode_escape_decode()
      • latin_1_decode()
      • ascii_decode()
      • charmap_decode()
      • mbcs_decode()

      Using run_tests.sh, all tests are ok (with 19 skipped tests). I guess
      that there is not tests for all these functions :-/

      Note: codecs documentation was already correct:

      .. method:: Codec.decode(input[, errors])
      (...)
      *input* must be a bytes object or one which provides the read-only
      character
      buffer interface -- for example, buffer objects and memory mapped
      files.

    3. malemburg commented on Jan 8, 2009

      @malemburg
      Member

      On 2009-01-08 01:59, STINNER Victor wrote:

      STINNER Victor <victor.stinner@haypocalc.com> added the comment:

      Patch replacing "s*" parsing format by "y*" for:

      • utf_7_decode()
      • utf_8_decode()
      • utf_16_decode()
      • utf_16_le_decode()
      • utf_16_be_decode()
      • utf_16_ex_decode()
      • utf_32_decode()
      • utf_32_le_decode()
      • utf_32_be_decode()
      • utf_32_ex_decode()
      • latin_1_decode()
      • ascii_decode()
      • charmap_decode()
      • mbcs_decode()

      These are fine.

      • unicode_escape_decode()
      • raw_unicode_escape_decode()

      These changes are in line with their C API codec interfaces as well,
      but those particular codecs could well also be made to work on Unicode
      input, since unescaping can well be applied to Unicode as well.

      I'll probably open a new item for this.

      Using run_tests.sh, all tests are ok (with 19 skipped tests). I guess
      that there is not tests for all these functions :-/

      The mbcs codec is only available on Windows.

      All others are tested by test_codecs.py.

      Which ones are skipped in your case ?

      Note: codecs documentation was already correct:

      .. method:: Codec.decode(input[, errors])
      (...)
      *input* must be a bytes object or one which provides the read-only
      character
      buffer interface -- for example, buffer objects and memory mapped
      files.

      That's not entirely correct: codecs are allowed to accept any
      object type and can also return any object type. It up to them
      to decide, e.g. a codec may accept both bytes and Unicode input
      and always generate Unicode output when decoding.

      I guess I have to review those doc changes...

    4. pitrou commented on Jan 13, 2009

      @pitrou
      MemberAuthor

      The patch is probably fine, but it would be nice to add some unit tests
      for the new behaviour.

    5. malemburg commented on Jan 14, 2009

      @malemburg
      Member

      On 2009-01-13 14:14, Antoine Pitrou wrote:

      Antoine Pitrou <pitrou@free.fr> added the comment:

      The patch is probably fine, but it would be nice to add some unit tests
      for the new behaviour.

      +1 from my side.

      Thanks for the patch, Viktor.

    6. vstinner commented on Jan 17, 2009

      @vstinner
      Member

      New patch:

      • Leave unicode_escape_decode() and raw_unicode_escape_decode()
        unchanged (still accept unicode string)
      • Test changes (reject unicode for most codecs decode functions)
      • Write tests for unicode_escape_decode() and
        raw_unicode_escape_decode() (there was no test for these functions)
    7. self-assigned this
      on Jan 22, 2009
    8. pitrou commented on Jan 22, 2009

      @pitrou
      MemberAuthor

      Committed in r68855, r68856. Thanks!

    9. amauryfa commented on Jan 22, 2009

      @amauryfa
      Contributor

      IMO, Modules/cjkcodecs/multibytecodec.c should be changed as well.

    10. pitrou commented on Jan 22, 2009

      @pitrou
      MemberAuthor

      Right, I hadn't thought of that.

    11. reopened this on Jan 22, 2009
    12. pitrou commented on Jan 22, 2009

      @pitrou
      MemberAuthor

      "Fixing" multibytecodec.c produces a TypeError in the following test:

          def test_errorcallback_longindex(self):
              dec = codecs.getdecoder('euc-kr')
              myreplace  = lambda exc: ('', sys.maxsize+1)
              codecs.register_error('test.cjktest', myreplace)
              self.assertRaises(IndexError, dec,
                                'apple\x92ham\x93spam', 'test.cjktest')

      TypeError: decode() argument 1 must be bytes or buffer, not str

      Since the test is meant to test recovery from a misbehaving callback, I
      guess the type of the input string is not really important and can be
      changed to a bytes string instead. What do you think?

      (in any case, here is a patch)

    13. amauryfa commented on Jan 22, 2009

      @amauryfa
      Contributor

      The patch looks good.
      I think the missing b in test_errorcallback_longindex is an overlook
      when the tests were updated for py3k. You are right to change the test.

    14. pitrou commented on Jan 22, 2009

      @pitrou
      MemberAuthor

      Committed in r68857, r68858.

    15. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions